KarmaPro
Benchmarks
Comparison of Karma Pro’s C/C++ scanner against published analysers on the full Juliet Test Suite, following the methodology from Dyremark & Elfstrand (2025).
Juliet Test Suite for C/C++
| Tool | Recall | Precision | F1 |
|---|---|---|---|
| Karma Pro | 18.4% | 68.6% | 29.1% |
| Clang-Tidy | 14.9% | 32.9% | 20.5% |
| Flawfinder | 6.4% | 38.3% | 11.0% |
| Cppcheck | 6.2% | 68.5% | 11.4% |
| Snyk Code | 5.7% | 62.6% | 10.4% |
| Semgrep | 1.6% | 67.1% | 3.1% |
Reference: Dyremark & Elfstrand, False Positives, Real Problems: Evaluating Static Analysis Tools (2025). https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1968169&dswid=-3943.
Comparison of Karma Pro's C/C++ scanner against published analysers on the MITRE Top-25 subset of the Juliet Test Suite, following the methodology from Dyremark & Elfstrand (2025).
Juliet Test Suite for C/C++ MITRE Top 25
| Tool | Recall | Precision | F1 |
|---|---|---|---|
| Karma Pro | 20.3% | 78.1% | 32.2% |
| Clang-Tidy | 17.4% | 50.0% | 25.8% |
| Cppcheck | 6.6% | 61.8% | 11.9% |
| Semgrep | 0.9% | 75.0% | 1.9% |
| Snyk | 4.7% | 62.5% | 8.7% |
| Flawfinder | 8.5% | 37.5% | 13.8% |
Reference: Dyremark & Elfstrand, False Positives, Real Problems: Evaluating Static Analysis Tools (2025). https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1968169&dswid=-3943.
Comparison of Karma Pro’s Java scanner against published analysers on the Juliet Test Suite v1.3 for Java, following the methodology from Alqaradaghi & Kozsik (IEEE Access 2024).
Juliet Test Suite v1.3 for Java
| Tool | Recall | Precision | F1 |
|---|---|---|---|
| Karma Pro | 78% | 97% | 85% |
| PMD | 82% | 92% | 83% |
| Find Security Bugs (FSB) | 86% | 73% | 75% |
| SonarQube (Sonar) | 68% | 78% | 61% |
| Infer | 53% | 88% | 58% |
| SpotBugs (SB) | 49% | 89% | 52% |
Reference: Alqaradaghi & Kozsik, Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code, IEEE Access 12 (2024), 55824–55842. https://doi.org/10.1109/ACCESS.2024.3389955.
Comparison of Karma Pro’s JavaScript scanner against published analysers on the VulcaN corpus, following the methodology from Brito et al. (2023).
VulcaN corpus for JavaScript
| Tool | Recall | Precision | F1 |
|---|---|---|---|
| Karma Pro | 45.7% | 48.9% | 47.2% |
| ESLint (security plugin) | 41.3% | 47.3% | 44.1% |
| CodeQL | 32.0% | 55.2% | 40.5% |
| graudit | 20.3% | 28.1% | 23.6% |
| odgen | 16.7% | 37.0% | 23.0% |
| msdevskim | 10.3% | 22.0% | 14.1% |
| njsscan | 9.7% | 23.2% | 13.6% |
| appinspector | 6.0% | 9.2% | 7.3% |
| mosca | 3.0% | 6.1% | 4.0% |
| drek | 1.7% | 6.9% | 2.7% |
| insidersec | 0.7% | 3.8% | 1.1% |
Reference: Brito et al., Study of JavaScript Static Analysis Tools for Vulnerability Detection in Node.js Packages, IEEE Transactions on Reliability 72(4), 1324–1339, 2023. https://doi.org/10.1109/TR.2023.3286301.
Comparison of Karma Pro’s Solidity scanner against published analysers on the 69-contract ICSE subset of the SmartBugs corpus, following the methodology from Durieux et al. (ICSE 2020), using the full rule set.
SmartBugs corpus for Solidity
| Tool | Recall | Precision | F1 |
|---|---|---|---|
| Karma Pro | 81% | 80% | 81% |
| Mythril | 43% | 38% | 40% |
| Manticore | 20% | 56% | 30% |
| Oyente | 28% | 33% | 30% |
| Slither | 20% | 50% | 29% |
| Osiris | 20% | 38% | 26% |
| SmartCheck | 13% | 39% | 20% |
| Maian | 10% | 88% | 18% |
| Securify | 12% | 18% | 14% |
Reference: Durieux et al., Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts, ICSE 2020. https://doi.org/10.1145/3377811.3380364.
* Recall = TP/(TP+FN) (true positive rate, how many actual vulnerabilities were found). Precision = TP/(TP+FP) (positive predictive value, how many reported findings are correct). F1 = 2·(Precision·Recall)/(Precision+Recall) (the harmonic mean of precision and recall). The benchmarks shown here compare AST/Heuristic scanning only and do not include AI scanning or ML.