2026-11-02 5 min read Measurements Scan

Our cyber security scan for smart contracts found 14 of 20 known bugs. Anyone can recount it.

On five public audits our cyber security scan for smart contracts found 14 of 20 known High-severity bugs, one run each. Our research runs together reach 77.4 percent on seven. Every verdict is open and one command recounts it.

Until today we could not tell you how good our cyber security scan for smart contracts really is. It reads the Solidity code of a project and reports the bugs it thinks are there. We had a number for our research runs, 77 percent when six of them were put together, and we had a number for an older, simpler route, 8 of 20. We did not have a number for the thing you actually buy.

Today we do. We took the five audits of that older measurement, ran the current setup once on each, and let a copy of the benchmark's own judge grade the reports. It found 14 of the 20 known High findings.

That is 70 percent, on five audits, with one run each. It makes no promise about your contracts, and it is the first honest count of the scan we sell. Anyone can redo it: clone the folder, run one command, and the same 14 come out.

The scan is for people and for AI agents. A person pays by card or iDEAL, and an AI agent pays over x402 in USDC on Base.

Five columns of glowing teal cells on a dark blue background, one column per audit. Each cell is one known finding, lit when the scan found it and dashed when it did not. The headline reads 14 of 20.

Each cell is one known finding. A lit cell is a finding the scan found. Drawn from the table of the measurement, not a photograph.

And the 77 percent

You may have seen 77 percent from us. It is real and just as recountable: six of our research runs together found 24 of the 31 known High findings on seven audits, graded by the same judge copy. It is a research result, so it sits above the bar of the scan you can buy, which is the bottom one in the picture: 14 of 20 on five audits.

The leader's own report scores 80.6 percent under the same judge, so there is room above us, and the room is measured. Each bar has its folder, its verdicts and a script that recounts it.

Four horizontal bars. The leader's report 80.6 percent, six of our research runs together 77.4 percent and one research run 71.0 percent, all on seven audits, and the scan you can buy 70.0 percent on five audits.

Four numbers you can recount, each with its own set. Drawn from the tables of the two result folders, not a photograph.

How it went

We wrote our guess down before the second set of runs: 5 of 11 on three audits. The scan found 9. Our guess was too low, and we are glad to say so in public.

The older route had found 8 of 20. Four of the six extra findings come from one audit, forte, where the old run had been stopped by its cost cap and counted as zero. On the other four audits the count went from 8 of 15 to 10 of 15. We would rather tell you that ourselves than let you work it out.

The numbers

The first number is the scan you can buy: five audits, 20 known High findings, one run each. The others are there for comparison. Each has the place where you can check it, and each is graded by a copy of the benchmark's own judge.

14 of 20

Known findings found by the setup behind the deep scan. 70.0 percent, one run per audit.
the folder
17 of 90

Hypotheses the scan wrote that match a known finding when two open judges name the same one: 18.9 percent. It is a floor, because a real bug the audit did not list counts as a miss.
the precision section
11 of 14

Known findings found on six more audits, strictly, with two open judges. These audits are not fresh: earlier versions of our scan ran on them.
the six more audits
77.4 percent

Six of our research runs together on seven audits: 24 of 31 known findings. A research result, and not the scan you can buy.
the research folder
15 of 20

What the leader's own report finds on the same five audits under our judge copy. The benchmark's own grader would give 16, because it counts one more finding on phi.
the comparison
8 of 20

What our older, simpler route found on the same five audits. It is the number we corrected on 2 November.
the correction
5.88 dollars

What the five scans cost in model fees, without the judge.
table.json

Audit by audit

Found out of known, per audit. A green cell means all found, amber some, grey none. The leader column is the leader's report under our judge copy.

Who	Our scan, as sold now	Our older route	The leader
nextgen	yes: 2 of 2	yes: 2 of 2	yes: 2 of 2
size	partly: 1 of 4	partly: 1 of 4	partly: 3 of 4
phi	partly: 4 of 6	partly: 2 of 6	partly: 3 of 6
secondswap	yes: 3 of 3	yes: 3 of 3	yes: 3 of 3
forte	partly: 4 of 5	no: 0 of 5, stopped by its cost cap	partly: 4 of 5

The cells come from table.json of the two result folders. One run per audit, so a single finding moves a cell. Under the benchmark's own grader the leader finds 4 of 6 on phi.

Five towers of stacked glass cubes on an oak workbench at dawn, most cubes glowing teal and a few dark, a brass ruler and a magnifying glass in front, a window with morning mist behind.

Five audits, five towers: most cubes lit, a few dark. The picture does not show the real counts. Illustration generated with AI.

What it means

For us it gives the coming weeks a floor to build on. We know where the scan stands, so every change from here is measured against 14 of 20 and not against a feeling. We also know where it is weakest, size and phi, and where it needs to read more, because the runs did not open every file.

For you, if you build or buy security tooling for agents and contracts, it gives you something to ask every vendor for: the outputs, the judge and a script that recounts them. We publish ours. The settings of the scan stay ours, and the measurement stays open.

We think this is what the work should look like. A number you can recount is worth more than a bigger number you have to believe, and we are proud that ours can be checked by someone who has never met us. We would like to see more of it around us, from the engineers who build next to us and from the ones who build against us.

What we do not claim

Five audits and one run each. One finding moves the share by five points.
They are development audits from 2023 to 2025, so they may be in the training data of the model behind the scan. This is not a result on new contracts.
The hardest audit, noya, is not in the set.
Two runs, size and nextgen, hit our own cost limit of 1.50 dollars and were stopped. We have not measured what the shop's own limits would do on them.
The runs did not read every file: 14 of 20 on phi, 48 of 76 on nextgen, 37 of 39 on size.
The cheaper 1 and 5 USDC routes are older and are not measured here.
The precision figure comes from two open judges that are stricter than the grader copy: they find 9 of the 20 strictly where the copy found 14. Another vendor's precision cannot be set next to it.
The six more audits we measured afterwards are not fresh: earlier versions of our scan ran on all of them. They show the same setup on more audits, with 11 of 14 known findings strictly and 11 of 54 hypotheses matching, and they are not a result on new contracts.
The judge is a copy of the benchmark's grader, run through OpenRouter. It gave the same verdict as the benchmark's own answer on 30 of 31 findings when we checked it on the leader's report.

If a number in the folder does not match when you run the check, tell us in issue 41 of the repository. We will correct it and keep the old text visible.

Two pairs of hands at an old oak desk, one holding a brass magnifying glass over a handwritten ledger, the other on a laptop keyboard with a teal terminal on screen, a brass lamp and a cup of tea beside them.

Recounting, by hand and by script. Illustration generated with AI.

Recount it, or put the scan to work

The folder has every verdict, the exact reports the judge read, and a script that recounts them with no network and no key. The scan itself is for sale for agents over x402 and for people with a card or iDEAL.