public
Scoreboard
How many of the findings that human auditors already found in closed contests does our smart contract scan find? Rows link to their results and, where one exists, to the pre-registration written before the run. A row without that link has no public rule from before the run: for version 3.1 the rule is only in our own notes, and the row says so. A row without a result is empty on purpose.
Bug Bounty Intelligence · open scoreboard
Last updated 2026-10-30. Newest and still open rows first.
Every run of our scan on our fixed sets of known findings goes on this page within 7 days, the misses included, so you see the whole record and not only the good runs. If a version scores lower than the run before it on the same set, its row says so, with both numbers side by side. The rule is public and has an OpenTimestamps stamp next to it: The Publication Rule
- pending decides the question
Contests that appear after today
Every system that takes part, ours included
Not measured yet
New contests as their final reports are published
The cleanest test, because the answers do not exist yet when the report is committed. The hash of a system's report is committed before the contest results are published, and scored afterwards with the same two judges. Takes weeks, since results arrive per contest.
- measured partly tuned on this set
Version 3.1 on the 40 audits of EVMbench
Version 3.1 of the scan, as in production, unchanged
23 of 117
The 40 audits of the public EVMbench detect set, 117 high severity findings. 4 of the 40 overlap earlier tuning sets. It reported 1537 hypotheses in total.
Counts the first 15 hypotheses per audit, 600 of the 1,537 the scan reported, like the rows below. The first 30 hold 29 of 117, and so does the whole list. Only High findings are counted here, so the number cannot be set next to the Code4rena and Sherlock rows, which count High and Medium. Over the whole list 2.9 percent of the hypotheses match a known finding, and in 18 of the 40 audits nothing passed both judges. Without the 4 audits that overlap our earlier tuning sets it is 21 of 110. On the 7 newest audits, none used before, it is 3 of 14. The 29 older audits are public and may be in the training data of the models behind the scan and the judges. One run, no audit run twice. The rules for reading the result were written in a working note before the run, but we anchored that note only after it, so the order is something you take on our word. Three people outside our team recomputed the totals from the verdict files and got the same numbers; that covers the arithmetic, not whether the judges' verdicts are right.
- measured decides the question
Slither 0.11.6 on the same contests
Slither 0.11.6 with default settings, a free static analysis tool
0 of 34
Code4rena contests where it ran in our setup: 4 of 9 newer and 4 of 6 older, 34 known findings. It reported 167 hypotheses in total.
Strict hits: none. Two hypotheses on one contest matched a known finding for one judge only. Static tools are not built for the business-logic flaws that contests reward, they ran on only part of the contests in our setup, and the judges are language models. On the same contests our scan found 6 of 34 in its first 15 hypotheses; the older contests may be in the training data of the model behind our scan.
- measured decides the question
Aderyn 0.6.8 on the same contests
Aderyn 0.6.8 with default settings, a free static analysis tool
0 of 70
Code4rena contests where it ran in our setup: 4 of 9 newer and 6 of 6 older, 70 known findings. It reported 202 hypotheses in total.
Strict hits: none. Static tools are not built for the business-logic flaws that contests reward, they ran on only part of the contests in our setup, and the judges are language models. On the same contests our scan found 10 of 70 in its first 15 hypotheses; the older contests may be in the training data of the model behind our scan.
- measured not tuned on this set
Version 3.1 on nine newer Code4rena contests
Version 3.1 of the scan, as in production
5 of 66
9 Code4rena contests from September 2025 to March 2026, 66 known findings. Not used to build the scan. It reported 135 hypotheses in total.
Counts the first 15 hypotheses per contest, which is what reports showed until 25 October; the first 30 held 8 of 66. The rule for reading the result was written before the run: 5 percent or more holds, and this is 7.6 percent. 3 of 35 easy findings, 0 of 8 middle and 2 of 23 hard were in the first 15; 61 of 66 known findings were not. Small numbers, one run per contest. The Code4rena known lists are shorter than Sherlock's, contests from early 2026 may be in the training data of a model, and the Olas contest was read for 38 percent of its code.
- measured not tuned on this set
Version 3.1 on the same eight contests
Version 3.1 of the scan, live since the evening of 25 October
8 of 105
The 8 contests of the sealed run, 105 known findings. No longer secret: their results were published on 24 October. It reported 120 hypotheses in total.
6 of 59 easy findings, 1 of 14 middle, 1 of 32 hard. Precision 7.5 percent, about 20 to 45 seconds and 0.02 to 0.15 dollar per scan. Two things changed at once, a newer model and a fix of which files count, so we cannot say which one did the work. One run, so a repeat could land between about 6 and 10. The models may have seen these contests. 97 of 105 known findings were still not found. The rule for reading the result was written in our own notes before the run, which are not public, so this row is less verifiable than the sealed run.
- measured not tuned on this set
The sealed run: the new version
Version 3 of the scan, its files committed by hash before the run
4 of 105
Sealed set: 8 contests, 105 known findings, committed by a hash before anyone looked at them. It reported 120 hypotheses in total.
2 of 59 easy findings, 0 of 14 middle, 2 of 32 hard. Precision 5.0 percent, 46 to 105 seconds and at most 0.064 dollar per scan. It found 3 known findings that the running scan did not. The rule written first needed 5, so it is not met and the running scan stayed in production. On the evening of 25 October version 3.1 replaced it, see the row above. Our own reading of two disagreements between the judges would have made it 5; the number we registered is the one both judges agree on.
- measured not tuned on this set
The sealed run: the scan as it runs
Version 2.1 of the scan, the one in production until the evening of 25 October, with the pattern block
1 of 105
Sealed set: 8 contests, 105 known findings. It reported 43 hypotheses in total.
1 of 59 easy findings, 0 of 14 middle, 0 of 32 hard. Precision 4.7 percent, 2.7 to 8.6 minutes per scan. The one it found, the new version found as well.
- measured tuned on this set, not evidence
The next version, first trial
In development. How it works stays private until after the sealed run; the hash of its files is published before that run
7 of 85
Development set: 6 contests, 85 known findings. It reported 88 hypotheses in total.
7 of 31 easy findings, 0 of 26 middle, 0 of 28 hard. We tune on this set, so this is a record of how we got here and not evidence.
- measured tuned on this set, not evidence
The scan as it runs, on the development set
Version 2 of the scan with the pattern block of production
1 of 85
Development set: 6 contests, 85 known findings. It reported 39 hypotheses in total.
Without the pattern block it found 0 of 85 with 36 hypotheses. Two more known findings were matched by only one of the two judges, which we do not count. We tune on this set, so this is not evidence.
- measured partly tuned on this set
Version 2 of the scan
The scan after the first round of fixes
0 of 40
7 contests: 4 it was tuned on (31 known findings) and 3 new ones (9 known). It reported 31 hypotheses in total.
0 of 31 on the four contests it was tuned on and 0 of 9 on the three new ones. It read 29 percent of the in-scope code. We found afterwards that the measured copy lacked a block of historical patterns that production carries, and the results page carries that correction above its tables.
- measured not tuned on this set
Version 1, as it ran
The scan as it ran on the morning of 24 October 2026
0 of 31
4 contests, 31 known findings. It reported 2 hypotheses in total.
Two hypotheses in total, neither matching a known finding. One job failed and two returned empty reports. The prediction we wrote before the run was at most 2 of 31.
How to read a number
What is counted, and by whom.
- A known finding is a unique valid High or Medium finding in the final report of a Sherlock or Code4rena contest. The number of wardens who found it is its difficulty: easy is found by 5 or more, middle by 3 or 4, hard by 1 or 2.
- A hit counts only when two judges of different model families both name the same known finding for the same root cause. A person reads every disagreement.
- The contests were split into a development set and a sealed set by sorting them on a hash that includes the hash of Base block 53022890, which nobody here could choose. The sealed set is committed with the hash de1a23a4aabc17def5145229c8ee4b4b216e25e8a5ec990e0c5bb37620b61ca1 and was run once, on 24 October 2026. The set is published now, so anyone can recompute that hash.
What would count as proof
Five steps, in the order we take them.
- The sealed run: contests nobody here has seen, with the decision rule fixed in advance. Done on 24 October 2026; the bar was not met.
- Contests that appear after today, with the hash of our report committed before the results are published.
- A test that makes each reported flaw actually happen, so a finding does not rest on anyone's word.
- A flaw that a real owner confirms.
- Someone else re-scoring our published outputs with the open judges and getting our numbers.
What is open and what is private
- Open: the contests, the two judges and their check, the rules we wrote before each run, every output and every judge statement. Anyone can re-score them.
- Private: how the scan finds things. Before the sealed run we published the SHA-256 of the files it uses, so any later change would show.
What this is not
- It is not an audit, an approval or a certification. A scan that finds nothing does not make code safe.
- These contests may be in the training data of the models, so absolute numbers can be too high. Comparisons between systems on the same contests are paired and less affected.
- The judges are language models and we have a stake in the result. That is why every table and every judge statement is published.
Cite this page
tokenizen, scoreboard of the Bug Bounty Intelligence scan, https://tokenizen.nl/en/scoreboard
Found a mistake in how we count? Open an issue in the repository. We correct it in public.