2026-11-03 5 min read Measurements Tools

A test for any AI smart contract security scan. Even ours.

An outside reviewer put a made-up finding id into a copy of our evidence, and our checker said all good. So we wrote the test down, with the numbers it gives and the limit of each.

A score is a number somebody else counted. Whether it deserves belief depends on the checker that did the counting, and on whether anyone has tried to fool that checker on purpose.

This week an outside reviewer tried it on ours. The reviewer put a made-up finding id into a copy of our evidence and ran our checker. It said all good. We ran the same trick against every checker of ours: five of them can be reached with a made-up id, and three of those five said all good. Two are fixed. The third is on our public to-do list. Every correction is in ERRATA.md, with who found it and how.

A checker nobody has tried to fool is a checker you cannot trust yet. So we wrote the test down. The film below runs for 84 seconds. Under it is what it says, with every number next to its limit.

Two tests, not one

There are two things to test, and each needs its own plant.

• Test the scan: plant faults. Copy a contract, add known faults at places given by a published seed, run the scan on the planted copy and on the untouched original, and count how many planted faults it names. The seed fixes where the faults go, so anyone can plant the same ones. It does not fix what the scan or the judge does on a given day, so one run can move a count by a finding or two. We say so next to every number.
• Test the checker: plant a made-up finding id, or a flipped verdict, into a copy of the evidence and run the check. A checker that works says CAUGHT. A blind one says all good.

The second test needs no scan and runs in seconds. The first works on any scan that can read a repository: the folder with our planted-faults run explains how to run your own tool on the same planted copies. Our own run of it is there too, with its limits.

Run it yourself

You need git and Node 22 or newer, and nothing else. The first command recounts one result folder from its verdict files and checks its hashes, offline. The second runs the checker test on our own folders: it copies the 7 result folders that have a check script, plants a made-up finding id and a flipped verdict in each copy, and prints CAUGHT, BLIND or N/A. Right now it prints one BLIND, the third checker on our public to-do list, so expect it to end with an error. We would rather show you that than hide it.

git clone https://github.com/holistis/bug-bounty-intelligence-mcp
cd bug-bounty-intelligence-mcp
node results/2026-11-02-shop-setup-five-audits/check.mjs
node tools/checker-selftest.mjs

These two commands do not test anybody else's scan. For that, use the planted-faults folder above.

What we do not claim

• Our scan is not the best. Under our judge, the leader we compare against on the public benchmark is ahead of our research runs by one finding on the combined list and by three on a single run. Under the benchmark's own grader the gap on the combined list may be two. And four in five of the findings of our deep scan match no listed High finding.
• The 77.4 percent does not describe the scan you can buy. It comes from six research runs combined, and one run alone found 22 of 31.
• An unmatched finding is not necessarily wrong. It may be a real bug that nobody has listed.
• The planted faults do not look like real ones. They are simple and synthetic.
• Our checkers can still be fooled. One still says all good to a made-up id, and it is on our public to-do list.
• Our judges may be wrong. Nobody has yet checked by hand whether they name the right finding, and doing so is one of the first issues below.
• You cannot rerun the scan. The engine is closed. What a stranger can recount is the evidence: the verdicts, the hashes and the totals.

Knowledge, attitude, behavior: what we hope changes

We want to change three things in the people who read security scores, in this order.

• What they know. A number means what it counts and nothing more. We print the count, the limit and the link together, so a reader sees what a score can say and where it stops.
• How they feel about a limit. A limit stated up front is information. We state ours, and we would like that to become ordinary, for us and for anyone who publishes a score.
• What they do. The next time you read a score anywhere, ask where the checker is and whether anyone tried to fool it. If you want to help, run the two commands, label a finding that no judge matched, or try to break a checker.

Why the ruler is open

By ruler we mean the evidence folders, the check scripts, the fault planter and the re-judge tool: everything that measures a scan, as opposed to the scan itself. We want a digital world that is safe, reliable and lasting, where machines do the heavy lifting and humans keep their hands on the wheel. A measurement that only its maker can run proves little, so the ruler is open source under MIT, and texts in it that come from others keep their own licences. The scan engine stays closed.

Three first issues are open, each scoped to one folder or one audit: write the check that catches a made-up finding id, check by hand whether our judges name the right finding, or label the findings no judge matched. Your commits keep your name, and your name goes into the README of the folder you worked on. It is unpaid, public and yours.