How ykshdja/smart-contract-analyzer-benchmark works
1. This repository is best understood as two connected parts: a minimal live analysis app for Solidity code, and a larger offline evaluation workspace for comparing vulnerability scanners.
2. The runtime app boundary is very small. frontend/index.html is a static browser UI, and backend/main.py exposes the API over FastAPI/Uvicorn. The frontend sends Solidity source code to the backend’s /analyze endpoint and displays the returned findings.
3. The core analysis logic lives in backend/analyzer.py. The README describes it as pattern-based analysis, so the shipped product is not a compiler pipeline or symbolic executor; it is a lightweight rule matcher that returns structured vulnerability records such as type, severity, line, description, match, and context.
4. There is no obvious persistent application state, database, or message queue in the current tree. The main data flow is request/response: user code in, vulnerability list out.
5. The repository also contains a benchmark and comparison harness. Scripts like evaluate_slither_auto.py, evaluate_mythril_*.py, evaluate_semgrep_*.py, evaluate_solhint_*.py, evaluate_aderyn_*.py, and evaluate_scrubd_sd.py suggest the author runs multiple external analyzers against the same contract corpora and compares their outputs.
6. compute_metrics.py and read_report.py appear to post-process those runs into summary metrics and human-readable reports. The presence of report.md and writeup.txt indicates the repo is being used for research-style evaluation, not just an app demo.
7. The contract corpora are split across several folders: contracts/, test_contracts/, and test_sample/. These are likely the inputs used by both the manual analyzer and the evaluation scripts.
8. The directories SCRUBD, sailfish, conkas, and semgrep-rules look like tool-specific assets or vendored benchmark material. Architecturally, they act as external analysis sources/rule sets rather than part of the live backend.
9. The mcore_* directories with .state_id and state_00000000.pkl files are generated artifacts. They look like serialized intermediate state or cached results from one of the experiment pipelines, so they belong on the “offline outputs” side of the diagram.
10. The important architectural boundary is therefore: browser UI → FastAPI analysis service → pattern-based findings, plus a separate offline pipeline that feeds contracts into third-party analyzers, aggregates their results, and produces metrics/reports.
11. If you draw this repo, emphasize the simplicity of the production path and the breadth of the research/evaluation path. The repo is not a full platform; it is a prototype analyzer surrounded by experiment scripts, datasets, and result artifacts.
Main entry points: contracts Contracts · semgrep-rules Tool assets · mcore___mt3ta8 Run state · evaluate_mythril_cd.py Mythril eval
Components 24
Live analysis
Minimal browser-to-API path for analyzing Solidity source code.
- Browser UI · Single-page HTML interface for submitting Solidity code and viewing findings.
frontend/index.html - API service · Exposes the analysis endpoint and serves as the runtime boundary.
backend/main.py - Rule matcher · Core vulnerability detection logic that returns structured findings.
backend/analyzer.py
Evaluation pipeline
Offline benchmarking, comparison, and report generation.
- Run scripts · Folder-level orchestration for batch analysis runs over contract corpora.
analyze_folder.py - Direct runner · Alternate batch runner for direct execution against corpora.
analyze_folder_direct.py - Slither eval · Runs Slither against the benchmark corpus for comparison.
evaluate_slither_auto.py - Mythril eval · Runs Mythril in multiple modes to compare coverage and findings.
evaluate_mythril_cd.py - Mythril fast · Faster Mythril evaluation variant.
evaluate_mythril_cd_fast.py - Mythril sd · Secondary Mythril evaluation variant.
evaluate_mythril_sd.py - Semgrep eval · Runs Semgrep against the benchmark corpus for comparison.
evaluate_semgrep_cd.py - Semgrep sd · Secondary Semgrep evaluation variant.
evaluate_semgrep_sd.py - Solhint eval · Runs Solhint against the benchmark corpus for comparison.
evaluate_solhint_cd.py - Solhint sd · Secondary Solhint evaluation variant.
evaluate_solhint_sd.py - Aderyn eval · Runs Aderyn against the benchmark corpus for comparison.
evaluate_aderyn_cd.py - Aderyn sd · Secondary Aderyn evaluation variant.
evaluate_aderyn_sd.py - Scrubd eval · Runs Scrubd against the benchmark corpus for comparison.
evaluate_scrubd_sd.py - Metrics · Aggregates benchmark outputs into summary metrics.
compute_metrics.py - Report reader · Transforms run outputs into readable summaries and report material.
read_report.py - Report · Human-readable benchmark report.
report.md
Datasets and assets
Contract corpora, tool-specific resources, and generated experiment state.
- Contracts · Primary contract set used for analysis and benchmarking.
contracts - Test contracts · Additional contract set used in experiments and comparisons.
test_contracts - Samples · Small sample set for quick checks or demonstrations.
test_sample - Tool assets · Vendored rulesets and tool-specific resources used by offline evaluations.
semgrep-rules - Run state · Serialized intermediate states and cached outputs from experiment runs.
mcore___mt3ta8
Connections 18
- Browser UI → API service: submit code · Browser sends Solidity source to the analysis API.
- API service → Rule matcher: analyze · API delegates request handling to the rule-based analyzer.
- Rule matcher → Browser UI: return findings · Structured vulnerability records are returned to the UI.
- Run scripts → Contracts: batch input · Runs the benchmark pipeline over the main contract corpus.
- Direct runner → Test contracts: batch input · Direct runner consumes the experimental contract set.
- Slither eval → Contracts: benchmark · Uses the shared corpus for comparison runs.
- Mythril eval → Contracts: benchmark · Feeds the corpus into Mythril evaluation modes.
- Semgrep eval → Contracts: benchmark · Feeds the corpus into Semgrep evaluation modes.
- Solhint eval → Contracts: benchmark · Feeds the corpus into Solhint evaluation modes.
- Aderyn eval → Contracts: benchmark · Feeds the corpus into Aderyn evaluation modes.
- Scrubd eval → Contracts: benchmark · Feeds the corpus into Scrubd evaluation runs.
- Mythril eval → Tool assets: rules/data · Uses external tool assets and rules where applicable.
- Semgrep eval → Tool assets: rules/data · Uses external Semgrep rulesets.
- Metrics → Run state: aggregate · Consumes serialized run artifacts to compute summary metrics.
- Report reader → Run state: read results · Reads generated experiment state for report preparation.
- Report reader → Report: write report · Produces the human-readable benchmark report.
- Metrics → Report: summaries · Metrics feed into the final report narrative.
- Tool assets → Run state: produce · Tool runs and assets contribute to cached intermediate state.