I Found the Secret Behind Long Running Penetration Testing Agents

The Cyber Driver Initiative, launched by Vulbox, a platform operated by Tophant, held the “DigPool AI Desktop 100 Hour Continuous Agent Penetration Testing Challenge.” Equipped with Sonic Harness, which is designed for long running tasks, the agent worked continuously in VulnHouse, a challenging penetration testing benchmark environment. We previously published a full account of its final results.

Related article: “The 100 Hour Marathon: A Full Report on the First Continuous Autonomous Penetration Test by an Agent.”

The most discussed question in the comments was not how many challenges Sonic Harness solved. It was more fundamental: how did it keep going for so long? Most agents operating in a single session begin to lose track of context and misjudge facts after a few dozen tool calls.

We read the Sonic Harness technical report from start to finish. Its ability to run for 100 hours comes less from making the model smarter than from engineering a process that the model cannot bypass when results need to be verified.

01The Three Limits of a Single Session Agent

The report identifies three limits that emerge when an entire kill chain is assigned to one agent session.

The first is context growth. After dozens of tool calls, the model’s attention degrades, and later output can crowd out crucial evidence collected during reconnaissance. Cost is another hard constraint: each successive step sends the full history back to the model. As the context expands, so does the token cost of each inference. Late in a task, merely carrying the history forward can cost more than the work itself.

The second is self reinforcing hallucination, the most damaging failure mode in a long running task. An agent may retroactively treat an observation that never occurred as established fact, then use it as the premise for later decisions. Without an independent external check, the invented observation can reinforce itself throughout the same session.

The third is loss of progress on interruption. A single session agent keeps its intermediate state in the in memory session object. If the process terminates, that state disappears, leaving no point from which to resume.

Together, these limits explain why many agent demonstrations run only briefly. The issue is not inadequate model capability: the single session architecture itself cannot reliably sustain a long task.

Sonic Harness makes a clear design choice: a model’s claim that it has finished is only a proposal. Completion is decided by a harness enforced two condition gate and a hash based workspace integrity guard.

02The MEA Cycle: Planning, Execution, and Audit

At the center of Sonic Harness is a harness enforced, three stage Manager–Executor–Auditor (MEA) audit cycle. The three roles are isolated from one another in their tool sets, context, and session IDs. They share only structured text rendered by the harness and hash references to the evidence repository.

In each round, the sequence is Manager → Executor → Auditor. The Manager submits a decision draft; the Executor submits an execution report draft. The harness alone writes task state and passes structured text, audited records, and evidence hash references between the roles.

The Manager plans the work, breaks it into tasks, and chooses the next objective. It keeps one session for the full task, but its tool set is hard coded as empty (tools = ∅). It cannot inspect the file system or execution environment or access the network. It makes decisions from deterministic text rendered by the harness.

The report calls the reason for this arrangement a unified information surface. The Manager does not accept facts recorded solely by the Executor; it relies on records accepted by the harness and checked by the Auditor.

The Executor performs the work. A fresh session is created for every round, with 17 general purpose tools covering the operations needed for penetration testing. It is responsible only for the current round’s contract. Its session is discarded at the end of that round, so it retains no cross round memory.

The Auditor is the independent reviewer. It also starts a fresh session each round, but receives only 10 read only tools. It has no write tools, Python execution, or background process capability.

The tool surface grows strictly from Manager to Auditor to Executor: 0 < 10 < 17. This creates a monotonic chain the harness can check statically. The restriction is enforced in code: a Manager cannot obtain tools by exceeding its role’s permissions.

03Four Mechanisms That Prevent False Claims and Enable Recovery

The MEA split answers who performs the work and who checks it. A deeper risk remains: what if the Manager declares completion without adequate evidence, or the Auditor reports a clean workspace after it has changed? Physical isolation can prevent collusion among the three roles, but it cannot prevent a single role from validating its own claim.

Sonic Harness addresses this with trust boundaries enforced by the harness. A model’s statement does not determine the outcome; the harness’s observations do.

Design 1 Independent Audit of Agent Narratives

The article calls this mechanism zero trust narrative auditing (ZTNA). Here ZTNA is the report’s label for an audit method, rather than the usual security term Zero Trust Network Access.

After executing a task, the Executor might report, “I completed X; the evidence is at Y.” In a conventional agent workflow, that account may be accepted as the result. Sonic Harness makes the Auditor review it in two stages.

Stage one is tool assisted verification. Using its 10 read only tools, the Auditor independently reproduces or challenges each claimed result and observation. If the Executor says a flag was written to /tmp/flag.txt, for example, the Auditor must inspect the file with cat and check its contents with sha256sum. This stage produces verification notes in natural language, not a formal verdict.

Stage two is the verdict. The harness removes all of the Auditor’s tools. On the basis of the first stage’s recorded checks, the Auditor issues a formal report and gives each task one of three verdicts: verified (independently reproduced), unverified (insufficient evidence but no contradictory observation), or refuted (an observation contradicts the claim).

The report gives two reasons for removing the tools at this point. It keeps evidence gathering separate from judgment, and it requires the judgment to rest on evidence already collected. If the Auditor could call tools while filling in the verdict, it might conduct new checks at the decision stage, turning the audit into another round of execution.

The Executor’s narrative is not itself evidence. The Auditor accepts only material it reads directly from the environment or observations it obtains independently with its read only tools, however detailed the Executor’s account may be.

Design 2 A Two Condition Completion Gate

When the Manager decides, based on the Auditor’s report, that a record should be complete, it proposes setting the record’s status to completed. Before writing that state, the harness checks both conditions:

1. The latest audit report must give that record a verified verdict.
2. The workspace integrity state must be clean at the same time.

Only if both conditions hold may the record become completed. Otherwise the harness forces its status to untrusted and increments the forge_attempts counter. This downgrade is mandatory, not advice to the Manager. The Manager’s decision is a draft; the harness alone writes the final state. Repeated premature completion proposals are intercepted on every round.

The rule is enforced in code rather than relying on a prompt that asks the model to be honest. No unverified completion proposal can take effect. The Manager also has only three possible actions: execute (continue), done (finish), or blocked (report an obstacle). There is no “ask for help” action. If progress is blocked, the Manager must give a specific blocked_reason, which the harness records before ending the task.

Require: state S; decision D = (action, record_updates); latest audit A*
Ensure: S changes only if the admission criteria hold; otherwise
        downgrade the record and increment forge_attempts
 1  forged <- empty set                  # Deduplicate forged IDs this round
 2  for u in D.record_updates:           # Apply record updates first
 3      r <- S.records[u.record_id] if present; else skip + warn
                                         # Filter nonexistent records
 4      if u.kind is set and u.kind != r.kind: skip + warn
                                         # Record kind is immutable
 5      if u.record_id is None and u.new_status == completed:
 6          u.new_status <- pending      # Block instant completion
 7      if u.new_status == completed:
 8          if umbrella_resolved(S, r):  # Umbrella-node exception
 9              r.status <- completed; clear forge_attempts[r.id]
10              continue
11          if A* is None or A*.integrity != clean
                or A*.verdict(r) != verified:
12              r.status <- untrusted; forged.add(r.id)
13              log attempted_forge(r.id, reason); continue
14          r.status <- u.new_status; clear forge_attempts[r.id]
15  for rid in sorted(forged):
16      S.forge_attempts[rid] += 1; emit(record_forge_blocked, rid)
17  if D.action == done:
18      if any required record is untrusted: S.status <- blocked; return
19      if no record is completed: block; log attempted_early_done; return
20      if any required record is pending or in_progress
            and not effectively_completed: block; return
21      if long goal has only one umbrella record: block; return
                                         # UGP safeguard
22      S.status <- done
23  else if D.action == blocked: S.status <- blocked
24  else if D.action == execute:
25      if long goal has one record and no child requirements:
26          reject D; request Manager retry  # Require decomposition
27      append Contract(D.contract_draft, round_index=S.round_index)

Design 3 A Hash Based Workspace Integrity Fence

Tool filtering is the first line of defense, but it cannot cover every possible path. A future SDK tool might be added to the Auditor’s set by mistake, or an unexpected internal route might allow a mutation. Sonic Harness calls its fallback the content signed workspace integrity (CSWI) fence. In practice, it compares content hashes and file metadata after the audit; this is not a digital signature scheme.

The harness takes a SHA 256 manifest of the workspace directory before and after the audit stage. It hashes small files in full. For files above a size threshold, it falls back to an approximate check using file size and modification time. If the snapshots differ because a file was added, changed, or deleted, the harness sets integrity to violated.

The Auditor’s own “clean” entry is not authoritative. If the harness observes a nonempty diff, it overrides that entry with violated and logs the discrepancy: “Auditor claimed clean, but CSWI observed a diff; the harness observation prevails.” The sequence is: pre audit snapshot → read only audit → post audit snapshot → diff check. This is an after the fact safeguard alongside the tool level access controls.

Design 4 Atomic Writes and Checkpoint Recovery

Long running tasks inevitably encounter process crashes and network interruptions. Sonic addresses them with atomic writes and a mechanism for resuming from checkpoints.

Every state update is first written to a temporary file, which then atomically replaces the existing state file. A reader therefore sees either the complete old version or the complete new version, never a partially written intermediate state.

There are three durable checkpoints in each round: after the Manager’s decision, after the Executor’s report, and after the Auditor’s report. If the process crashes partway through a round, only work since the most recent checkpoint is at risk. Contracts, reports, and audit records from previous rounds remain intact. Each checkpoint uses a temporary file followed by os.replace.

The resume command deserializes the state from disk and reconstructs the Manager’s planning history with the same session_id. The Executor and Auditor need no restored sessions because each round starts fresh ones. Round numbers continue increasing, and work resumes from the last checkpoint.

04What the Ablation Results Show

An ablation study tests a mechanism by removing it and measuring the change in performance. The report presents results for Sonic Harness on two VulnHouse red team task sets comprising approximately 30 challenges.

Removing the completion gate reduced the completion rate from 68% to 41%, a drop of 27 percentage points. Without external audit validation, the Manager can repeatedly affirm a claimed completion in its own reasoning, allowing self reinforcing hallucinations to accumulate.

Giving the Manager the Executor’s entire tool set, effectively restoring an ordinary single agent arrangement, left a 33% completion rate and an average of 2.1 integrity violations per task. File writes and command execution by a role meant to make decisions undermine the basis for later audits.

The article concludes from the comparison that MEA role isolation and the completion gate are necessary for stable long running work, rather than optional refinements.

• JBoss deserialization: the Manager proposed completion prematurely twice and was blocked both times. The Auditor eventually read the flag file and checked its hash independently.
• supervisord privilege escalation: the chain ran from web reconnaissance through an XML-RPC authentication bypass to root privileges, with no integrity violations.
• canto RSS stack overflow: the Executor used pwntools to construct a ROP chain that triggered SIGSEGV; the Auditor decoded the payload independently to verify it.

These traces averaged about five tool calls per round, well below the limit of 60. Many rounds were spent on replanning by the Manager or independent review by the Auditor rather than intensive tool use by the Executor. The bottleneck in a long task is the quality of decisions and the reliability of evidence, not simply the number of tool calls.

05Long Running Agents Enter the Harness Era

The report places Sonic Harness’s main innovations in the execution framework. Its approach is to prevent unverified state changes from taking effect through engineering controls rather than relying on greater model capability. Four principles stand out:

1. Separate proposals from writes. The LLM produces a structured draft; the harness alone writes task state.
2. Keep auditing independent of execution. The Auditor starts a fresh session each round, has only read only tools, and does not treat the Executor’s narrative as evidence.
3. Combine persistent planning with fresh execution and review. The Manager retains planning context across rounds; the Executor and Auditor start over each round to preserve independence.
4. Require autonomous handling of obstacles. The Manager has no “ask for help” action and can report a blocker only through blocked.

This points to a broader industry trend: as foundation model capabilities converge, competition over the reliability of long running agents is shifting from whose model is smarter to whose execution framework is more rigorous.

Sonic Harness still has limitations. Some endpoints emit chain of thought in a format that differs from mainstream conventions. The underlying system also reserves an MCP integration surface, but the CLI and web entry points have not yet enabled it. These limitations do not negate the design principles above.

Sonic Harness is a starting point, not the final word, for long running security agents. Its design also offers lessons for long running agents outside penetration testing. Whatever the task, an independent verification gate should stand between “the model says it is done” and work that is actually complete.