3 min read Measurements Tools
The scan held its rate on nine newer contests
We ran the scan once on nine Code4rena contests from the last year. The first 15 hypotheses held 5 of 66 known findings, the same rate as before. What that says, and a second list in reports.
Earlier today we published a run of the scan on 8 Sherlock contests: 8 of 105 known High and Medium findings. Those contests were no longer secret, so the open question was whether the rate holds on contests the scan was not built on.
We ran the scan once on 9 Code4rena contests from September 2025 to March 2026 and had the same two judges compare the results with the known findings in the public final reports. The rule for reading the result was written before the run: 5 percent or more of the known findings in the first 15 hypotheses means it holds.
The first 15 hypotheses held 5 of the 66 known findings (7.6 percent). The first 30 held 8 (12.1 percent). That is the same rate as on Sherlock, so by our own rule it holds.
What this does not say
- 58 of the 66 known findings were not in the first 15. This is not a scan that finds what a human audit finds.
- Small numbers and one run per contest. A repeat could land somewhere else.
- Code4rena reports list the findings that were kept after merging duplicates and removing low ones, so the known list is a different kind of list than on Sherlock.
- Contests from early 2026 may be in the training data of a model. We do not know the cut-off dates.
- The Olas contest is bigger than the scan reads, so it read 38 percent of it. The report says so for every scan.
- The judges are language models. This is our own measurement, not an independent one.
A second list in reports
A report can now show up to 30 hypotheses, in two lists: the 15 the model is most confident about, and 15 more with lower confidence. On the stored lists the second list held 8 known findings that the first list did not, at a slightly lower precision: about 3 in 100 hypotheses matched a known finding in the second list, against about 4 in 100 in the first. We chose this after seeing the numbers, so it is a product choice and not a pre-registered test.
One thing we tried and dropped
A free language model that reads the code and picks the best 15 from the whole list did not beat the scan's own ordering. On four held-out sets it matched 5, 6, 7 and 4 known findings against 7, 8, 8 and 7 (some calls failed and count as zero, as the plan said, and we stopped the run before the last few finished). We do not use it.
What you can check
The judgements of both judges, the whole lists, the plan file with its hash and the limits are in the results folder, and the row is on the scoreboard. How the scan finds things stays private.