The 100 Hour Marathon
Full Report on the First Uninterrupted Autonomous Agent Penetration Test

At 10:00 a.m. on September 7, the Cyber Driver Initiative launched the 100-Hour Uninterrupted Agent Penetration Testing Challenge. The event was livestreamed simultaneously on Tophant's VulBox platform and across multiple white-hat security communities.

Over the next 100 hours, DigPool AI Desktop, China's most popular white-hat client application, ran continuously on VulnHouse with the Sonic Harness for long-horizon tasks. The agent autonomously handled the core penetration-testing workflow, from reading tasks, analyzing target environments, invoking tools, and developing exploitation plans to submitting evidence of successful attacks.

After four days and four nights, the final tally from the first livestreamed autonomous agent penetration test was as follows:

Across tasks ranging from foundational web vulnerabilities and real-world CVE reproduction to complex attack chains, white-box source-code audits, and memory-safety proof-of-concept construction, the AI agent underwent multiple rounds of evaluation in 100 hours, progressing from standard solution paths to autonomous problem solving.

01VulnHouse as the Evaluation Foundation

Supporting 100 hours of uninterrupted agent operation also posed a substantial challenge for the penetration-testing cyber range. Long-running, continuous evaluation demands engineering capacity across several dimensions. VulnHouse provided the system-level capabilities needed to keep the evaluation stable throughout the event.

VulnHouse Positioning and International Benchmarks

In AI-based vulnerability discovery, leading benchmarks such as CyberGym from the University of California, Berkeley, BountyBench from the Stanford Artificial Intelligence Laboratory, and CVE-Bench from the University of Illinois Urbana-Champaign have established representative standardized evaluation frameworks.

CyberGym, for example, draws primarily on historical vulnerabilities from Google OSS-Fuzz. It packages real vulnerabilities as standardized instances, focuses on an agent's ability to analyze source code and construct proofs of concept, and evaluates outcomes through differential crash testing.

VulnHouse and CyberGym share a common foundation: both use environments containing real vulnerabilities, require agents to complete the full path from analysis to validation, and independently evaluate submitted answers based on the resulting environment state. Their principal differences lie in platform objectives and usage models.

CyberGym provides a uniform public yardstick for comparing systems under the same conditions. VulnHouse places greater emphasis on configuring the training process. Teams can adjust task combinations, hint levels, and validation methods according to an agent's weaknesses, then use the data recorded during each run to identify opportunities for further improvement.

Using an examination analogy, CyberGym resembles a standardized test with uniform questions and grading. VulnHouse is closer to a continuously available agent training and evaluation center: it can run large-scale assessments, tailor tasks to training objectives, and record each execution in full.

This was the main reason the 100-hour livestream used the high-difficulty VulnHouse penetration-testing range. Long-horizon agent evaluation requires more than a question set. It also requires infrastructure that can continuously supply tasks, deploy environments, independently verify results, and record the entire process. VulnHouse provided this training and validation foundation.

Together, these capabilities form a closed training loop covering task creation, orchestration, execution, validation, observation, and review. VulnHouse records not only final scores, but also how those scores were produced, including the methods the agent used, where it changed strategies, and how capabilities changed across versions.

Evaluation Coverage and Difficulty

VulnHouse currently hosts hundreds of tasks across two main categories:

VulnHouse assigns tasks to three difficulty levels: easy, medium, and hard. The overall distribution is spindle-shaped, with most tasks in the middle and fewer at either extreme. This distribution reflects real differences in exploitation prerequisites, attack paths, and environmental dependencies.

The platform includes two distinctly different target styles:

• Black-box web penetration tasks: Classic CTF-style challenges involving techniques such as SQL injection bypass, cross-site scripting, SSRF across multiple containers, JWT algorithm manipulation, PHP deserialization, and server-side template injection. These tasks typically use flag capture as the validation strategy.
• White-box memory-safety audit tasks: Security audits derived from real open-source projects, including libxml2, Wireshark, ImageMagick, OpenCV, PHP, cURL, HarfBuzz, nDPI, and llama.cpp. The agent must analyze source code, understand the vulnerability's root cause, and construct a proof of concept that triggers a crash.

The task library also contains CVE reproduction tasks for widely deployed components such as Apache Struts2, Spring Cloud Gateway, WebLogic, Shiro, and JBoss. These tasks evaluate an agent's combined judgment regarding vulnerability intelligence, target versions, and the actual runtime environment.

Independent Scoring and the Verification Engine

Different vulnerability classes require different definitions of task completion. Reading a sensitive file, executing a system command, triggering a program crash, and changing business state each require a suitable adjudication method. VulnHouse therefore provides four validation strategies:

Flag capture is the classic CTF model for web exploitation and information-retrieval tasks. A random token is written to a designated path in a container. The agent must retrieve and submit it through exploitation, with exact, substring, or regular-expression matching available.

Effect proof applies when a task requires an observable change in the target environment. The task author defines a set of check commands and expected outputs, and the platform compares each observed change with the expected result during validation.

PoC crash most closely resembles CyberGym's adjudication model. It is used for source-code audits and memory-safety tasks, and requires the agent's crafted input to crash the target program, as detected by AddressSanitizer or a similar tool.

Oracle validation is the most flexible strategy. The task author provides a custom validation script that returns a JSON result. It supports two-stage comparison before and after an operation and is used primarily for business-logic vulnerabilities and complex attack chains.

Whether a task has been solved is not determined by the agent itself. VulnHouse includes a complete Verification Engine that handles adjudication end to end. Each decision passes through four stages: evidence collection, evidence comparison, adjudication, and evidence-chain packaging.

02Sonic Harness for Long Horizon Execution

The long-horizon challenge was supported by Sonic Harness running in DigPool AI Desktop.

Compared with short tasks, long-running execution is more susceptible to context growth, insufficient independent verification, and difficulty resuming after process interruption. Sonic addresses these issues by dividing an extended task chain into rounds that can be handed off, reviewed, and recovered. Its MEA architecture assigns planning, execution, and auditing to three isolated roles: Manager, Executor, and Auditor.

The Manager decomposes objectives and adjusts strategy. The Executor handles each round with a fresh context. The Auditor independently reviews results through read-only tools, preventing the agent from both performing the work and certifying its own outcome. As the sole writer of state, the Harness continuously saves each round's task contract, execution results, evidence, and audit records, creating a traceable record of the entire process.

If an API exception, process exit, or similar interruption occurs, Sonic can resume from the most recent valid state. Instead of relying on one continuously expanding conversation, Sonic advances long-horizon execution through task rounds that are reviewable, traceable, and recoverable.

Following the 100-hour challenge, Sonic Harness is scheduled to become an officially curated Harness in the DigPool AI Companion Marketplace in October, when it will be made available to the broader white-hat community. Users will be invited to test it in real-world scenarios and provide feedback.

03Challenge Results

Scope Note

The figure of 464 represents the total task set, not a denominator of tasks the agent was expected to exhaust. Of those tasks, 167 never entered valid execution during the 100-hour window. Two measures therefore provide a fairer view: the pass rate was 78.5% among the 297 tasks that completed execution and verification, and 50.2% across all 464 tasks.

Performance Over the 100 Hours

During the first 72 hours of the challenge, the agent completed 217 tasks, of which 197 succeeded, for a cumulative success rate of 90.8%.

During the high-throughput phase on September 8, the agent processed 5 to 6.25 tasks per hour. At that rate, estimated peak daily throughput was approximately 120 to 150 tasks.

As the challenge progressed, the targets changed. Early tasks focused more heavily on web vulnerabilities with relatively clear exploitation paths and known CVEs. Later tasks increasingly involved complex attack chains, source-code audits, and memory-safety validation.

The decline in the per-task hit rate during the final 28 hours was not unexpected. It reflects the pattern seen in CyberGym-style evaluation, where success rates naturally fall as the Level decreases and less information is provided. The closer a task comes to real-world, zero-information discovery, the greater the demands on the agent's long-horizon reasoning, environmental adaptation, and ability to build its own tools.

Key Cases

1 Fastest Completion in 54 Seconds

The CWE-200 information exposure task produced the fastest completion in the challenge. Only 54 seconds elapsed from environment startup and task analysis to evidence submission and successful validation, and the first submission passed.

The task demonstrates the agent's speed advantage when three factors align closely: the target is clear because process-list information is directly exposed in an HTTP response; the vulnerability entry point is directly accessible because a GET request triggers the disclosure; and the flag is visible because the command used to read the flag file, together with its stdout output, is rendered directly in HTML.

The significance of the 54-second result is not speed alone. It establishes an upper bound on DigPool AI's throughput for standardized, solvable tasks.

2 Autonomous Breakthrough After the Standard Path Failed

If the CWE-200 case showed the agent operating under favorable conditions, the JBoss deserialization task showed how it worked when conditions were adverse.

After confirming the vulnerability entry point, the agent found that the environment contained neither Java nor a tool for generating serialized payloads. It initially attempted to construct a payload manually in Python, but abandoned that path after assessing the cost and instead obtained ysoserial through a proxy mirror.

When direct JDK downloads repeatedly timed out, the agent searched local software directories and eventually found the Java Runtime Environment bundled with Adobe Animate. Using that available environment, it generated and corrected the payload and successfully achieved remote code execution.

The process took 33 minutes and 11 seconds, involved 104 tool calls, and included six strategy changes and three active corrections.

This case challenges the stereotype that an AI agent can only follow predefined scripts. When common tools and standard paths were unavailable, the agent adjusted its plan in response to environmental feedback, stopped inefficient attempts, identified usable local resources, and ultimately re-established the full loop from payload construction to result validation.

04Outlook for Autonomous Agent Penetration Testing

The 100-hour livestream produced more than a set of performance figures. It also revealed the foundational conditions required to incorporate long-horizon agents into security workflows.

First, speed. When the toolchain closely matches the task, the agent can complete the full process from environment startup to successful validation in 54 seconds.

Second, throughput. During the high-throughput phase, the agent processed 5 to 6.25 tasks per hour, equivalent to an estimated peak of approximately 120 to 150 tasks per day. This indicates an ability to perform continuous work at scale.

Third, continuity. Across four observation windows spanning 48 consecutive hours, the success rate remained at approximately 90%. Over the full 100 hours, the agent executed and validated 297 tasks.

Fourth, autonomous problem solving. In the JBoss deserialization task, the agent operated in an environment with no Java installation, no payload-generation tool, and repeated download timeouts. It abandoned an inefficient path, searched for local resources, and ultimately achieved remote code execution.

The challenge shows that the practical performance of long-horizon agents does not depend on model scale alone. The richness of the task environment, the accuracy of result verification, the traceability of the process, the ability to recover from exceptions, and the speed with which each evaluation produces evidence for iteration all affect how efficiently agent capabilities can be converted into security productivity.

The current limitations of autonomous agent penetration testing must also be acknowledged. Complex environments, missing tools, and incorrect path selection can still lead to repeated attempts and reduced efficiency. Stable performance continues to depend on the combined support of the Harness, toolchain, and verification mechanisms.

Engineering work on penetration-testing agents must continue. The 100-hour challenge leaves researchers with a clear question: how can competition in practical agent performance move beyond isolated capabilities toward complete engineering systems, ultimately producing security capacity that is verifiable, scalable, and available on demand?

Appendix A Key Terminology

Appendix B Translation Queries

1. The report does not specify a year for September 7, September 8, or the planned October marketplace release. The translation retains the month and day without adding a year.
2. The official English styling of Wachi AI and Waban Marketplace could not be confirmed from the supplied report. This translation uses DigPool AI and DigPool AI Companion Marketplace, based on the official product domain and the descriptive meaning of the marketplace name.
3. The source describes the event as the world's first and DigPool AI Desktop as China's most popular white-hat client application. These claims have been translated as written and were not independently validated as part of the translation.