2 min read Measurements Tools
Two free tools found none of the known findings on the same contests
We ran Slither and Aderyn with default settings on the contests where our scan was measured. Neither matched a known High or Medium finding. What that says and what it does not.
We wanted to know how our scan compares with free tools people already use, on the same contests and with the same two judges. We ran Slither 0.11.6 and Aderyn 0.6.8 with default settings and turned each finding into a hypothesis. The plan was written down with a hash before the judges saw anything.
Neither tool matched a known High or Medium finding. Slither found 0 of 34 on the contests where it ran, and Aderyn 0 of 70. On the same contests our scan found 6 of 34 and 10 of 70 in its first 15 hypotheses. On the four newer contests, which our scan was not built on, it was 3 of 15 against 0 of 15 for each tool.
What this does not say
- Static tools are not built to find the business-logic flaws that audit contests reward. They are free and fast and find other things, such as patterns and missing checks. A finding of theirs can be a valid Low remark that is not on the list of High and Medium findings.
- The tools ran on only part of the contests in our setup. Slither ran on 8 of 15 and Aderyn on 10 of 15, and the reasons for the others are listed. A user with a working project setup would get results on more.
- The older contests may be in the training data of the model behind our scan; the tools do not have that. The judges are language models and this is our own measurement.
- Small numbers. Read it as a description, not a ranking.
What you can check
The hypotheses as the judges saw them, every judgement, the list of runs that did not work and the plan file are in the results folder, and both tools have a row on the scoreboard. The same procedure can be applied to the output of any other tool on the same contests.