Hive: human-led audits with adversarial AI

A convincing vulnerability description is easy to produce. Establishing whether it is correct requires a deeper understanding of the code, the system’s assumptions, and the path an attacker could actually take. That distinction shapes how we use AI at Oak Security.

Hive is our continuously evolving proprietary multi-model AI harness. We use it in all audits. At least one senior human security researcher always manually audits every line of in-scope code, then verifies all findings before they enter the final report.

Hive adds independent perspectives and structured challenges to that work. Our researchers direct the audit and remain responsible for its conclusions. The methodology follows five stages: scope, discover, deduplicate, validate, and review.

Establish scope and account for coverage

The audit starts with an agreed target revision and explicit scope. Researchers map components, entry points, privileged operations, dependencies, and the invariants the system must preserve.

Hive tracks file-read coverage for each model as well as across the run. That distinction matters: all files may have been read somewhere in the run while one model missed a component entirely. Missing work is retried where feasible, and remaining gaps are made visible to the reviewer.

Reading a file does not prove that a model understood it or examined every interaction with other components. Coverage records help us account for the work completed and interpret its limitations. A completed analysis with no candidates, unread code, and a model that failed to run are different outcomes.

Start with independent perspectives

A model can miss a vulnerability, make an incorrect assumption, or become attached to its first explanation. Asking it to expand that explanation does not necessarily expose the mistake.

Multiple models examine the in-scope code in separate contexts. They can share the agreed scope and architectural context, but they do not share candidate findings or discovery reasoning while discovery runs. Each produces its own candidates, locations, and threat assumptions for later comparison.

For each model, discovery has two separate lanes. In the blind pass, the auditor-selected specialist vulnerability catalogue is withheld, while broad areas such as authorisation, arithmetic, parsing, and state transitions guide the investigation. In the informed pass, the model receives specialist knowledge for the stack or domain. The passes use separate contexts and outputs; they do not depend on a particular order in time.

Blind refers to the supplied catalogue, not the model’s prior knowledge. Models can share training material, assumptions, and blind spots. Keeping their work separate preserves different routes into the problem without implying statistically independent errors.

Make candidates traceable and merge by root cause

A candidate needs a testable claim: the detecting model and pass, target revision, affected files and lines, attacker, required control, reachable preconditions, violated invariant or boundary, and resulting harm. A familiar vulnerability pattern is a reason to investigate; the target code still has to support the claim.

Deduplication follows the underlying defect and its correction. If two candidates describe the same missing authorisation check and the same targeted fix resolves both, they can be merged while preserving both detection records. A rounding flaw in the same function stays separate because it requires a different correction. Similar titles and overlapping locations alone do not establish duplication.

Make the finding survive a challenge

Discovery produces candidates. Each still needs to withstand scrutiny.

Every configured model validates every surviving finding, including its own, in a fresh context without access to discovery reasoning or other validators’ votes. It receives the claim, location, threat model, and agreed scope. The task is to challenge the claim: re-read the cited code, inspect surrounding functions and callers, check the attacker’s access, and look for constraints that make the proposed attack fail.

The burden of evidence rests on the finding. The validator must be able to state the attacker, reachable preconditions, ordered steps, and permitted path through the unmodified source. A timeout or unusable verdict is missing evidence. It is neither approval nor a substantive refutation.

Consider an illustrative example from our DSS presentation. A model notices an internal reserve-release function without a local role check and claims that any caller can drain funds. The validator traces every caller and finds that the only reachable external wrapper enforces an operator role that the alleged attacker cannot obtain. Under those assumptions, the arbitrary-caller claim fails. The reviewer retains the guard and caller evidence. Another unguarded wrapper would change the analysis; rejecting this particular claim does not establish that the whole contract is safe.

Model agreement is useful evidence, but it does not establish correctness by itself. Senior human researchers review the code and verify the findings, including their impact and severity, before deciding what belongs in the client report.

Keep confidence beside the evidence

Independent discovery and later approval are different signals. A third model approving a finding discovered by two others does not become a third independent discovery. A detector later refuting its own claim is also useful evidence that deserves inspection.

The reviewer can inspect detection history, validation outcomes, written reasons, and disagreements alongside confidence. Confidence expresses support for the claim; severity concerns the harm if it holds. A high confidence signal is not a measured probability of correctness, and several approvals cannot override a concrete guard that makes an attack unreachable.

Bring the right threat knowledge

Different stacks expose different failure modes. A smart contract, a cloud deployment, and an off-chain service need analysis grounded in their own execution models and trust boundaries.

Hive’s threat corpora organise vulnerability classes and detection guidance across several technology stacks:

Smart contracts and protocols: Solidity, CosmWasm, Cosmos SDK, and Noir.
Applications and infrastructure: Rust, Python, C/C++ memory safety, and cloud infrastructure, including infrastructure as code.
Cross-cutting and economic risks: general software vulnerability classes and DeFi economic attacks.

These corpora guide the informed pass. They complement the blind pass and the researcher’s understanding of the particular system. Specialist knowledge guides where to look; code evidence establishes whether a candidate holds. We continue to evolve both the corpora and the harness as our research develops.

Optimise for findings that stand up to review

We have tested this methodology against many other approaches. In our internal comparisons, its key advantage is substantially less noise and fewer false positives.

That matters in an audit. Every unsupported claim takes time to investigate and can distract from a weakness that needs attention. Independent discovery broadens the search; adversarial validation challenges weak candidates; human verification establishes which findings are valid and what they mean for the client.

These are qualitative observations from our internal testing. Results depend on the codebase, audit scope, and model configuration. We keep evaluating and refining the methodology as those conditions change.

Human review remains central

Hive is part of every Oak Security audit. Our commitment to manual review applies in every engagement: at least one senior human security researcher manually audits every line of in-scope code and then verifies all findings before reporting.

The reviewer receives the target revision, scope, coverage and run-health record, canonical findings with detection history, and validators’ reasons and disagreements. Reasoned validation, a successful proof of concept against unmodified source, and human acceptance are separate evidence and decision states. The researcher evaluates what each claim requires before accepting it.

Understanding intent, recognising when assumptions are wrong, and judging the practical consequences of a flaw require experienced researchers. Hive supports that work with additional discovery and scrutiny, while our team retains responsibility for the final assessment.