How smartbugs/smartbugs-wild works
This repository is primarily a data publication and curation pipeline for a smart-contract dataset, not an application runtime.
1. Core purpose
- The repo packages the “SmartBugs Wild Dataset”: 47,398 unique Ethereum smart contracts plus associated metadata.
- The main outputs are contract source files, contract metadata, and derived analysis tables.
2. Upstream data source
- The dataset starts from the Ethereum blockchain.
- Contract addresses were selected using a Google BigQuery query over Ethereum tables, filtering for contracts with at least one transaction.
3. Acquisition boundary
- After candidate addresses are identified, the repository relies on Etherscan as the external service for fetching:
- contract source code
- contract balance information
- The README explicitly notes that the collection scripts live under script/.
4. Collection scripts
- script/get_contracts.py is the contract source downloader/collector.
- script/get_balance.py is the balance collector.
- These scripts are the active data-ingestion layer; they bridge blockchain-derived address lists to Etherscan-backed metadata.
5. Primary dataset storage
- contracts/ contains the actual contract source files, one Solidity file per contract address, named as <contract_address>.sol.
- This folder is the largest and most important artifact in the repo.
6. Metadata bundle
- contracts.csv.tar.gz holds the metadata for all contracts.
- This is the structured companion to the raw source files, suitable for indexing, filtering, and analysis.
7. Derived analysis artifacts
- balances.json stores contract balance results.
- duplicates.json records deduplicated/duplicate identification results.
- nb_lines.csv provides line-count metrics for contracts.
- script/all_contract.csv appears to be a consolidated contract list used by the collection pipeline.
8. Curation / filtering step
- A key transformation in the pipeline is duplicate removal.
- The final dataset is the result of:
- collecting contract addresses
- downloading source code
- filtering out duplicates
- So the repo represents a curated subset of all available Solidity contracts.
9. Repository structure as architecture
- Inputs: Ethereum blockchain data + Etherscan APIs
- Processing: SQL address selection + Python collection scripts + deduplication
- Outputs: source corpus in contracts/ + metadata/metrics files at repo root
- This makes the repository a data pipeline output repository with scripts for reproducibility.
10. Operational/tooling implications
- The only material runtime/tooling dependencies implied by the repo are:
- SQL/BigQuery for initial selection
- Python for collection scripts
- Etherscan for external metadata/source retrieval
- No application server, UI, or long-running service is present.
11. Governance/licensing boundary
- The root LICENSE applies to the repository generally.
- Files under contracts/ are treated separately because they are publicly sourced from Etherscan and may retain original licenses.
- This is an important boundary if you diagram data provenance and legal constraints.
12. How to diagram it
- Model the repo as a pipeline with three layers:
- source selection from Ethereum/BigQuery
- enrichment/download from Etherscan via Python scripts
- curated dataset artifacts stored locally as source files and CSV/JSON metadata
- The central asset is the contract corpus; everything else supports collection, filtering, and analysis.
Main entry points: contracts contracts · all_contract.csv all contracts · get_contracts.py get_contracts · get_balance.py get_balance
Components 12
Source selection
Ethereum data is queried and filtered into a candidate contract list.
- Ethereum · Upstream Ethereum contract history and transactions.
- BigQuery filter · Query over Ethereum tables to select contracts with transactions.
- all contracts · Consolidated contract address list used by the collection pipeline.
script/all_contract.csv
Collection
Python scripts fetch source code and metadata from Etherscan.
- get_contracts · Downloads contract source code from Etherscan.
script/get_contracts.py - get_balance · Fetches contract balance information from Etherscan.
script/get_balance.py - Etherscan · External service used for source code and balance retrieval.
Curated outputs
Final contract corpus and derived analysis artifacts stored in the repo.
- contracts · One Solidity file per contract address; the main dataset asset.
contracts - contracts csv · Structured metadata bundle for the contract corpus.
contracts.csv.tar.gz - balances · Contract balance results derived during collection.
balances.json - duplicates · Duplicate identification and deduplication results.
duplicates.json - line counts · Per-contract line-count metrics.
nb_lines.csv
Repo boundary
Repository-level documentation and licensing context.
- LICENSE · Repository-wide licensing boundary; contract files may retain source licenses.
LICENSE
Connections 13
- Ethereum → BigQuery filter: tables · Ethereum blockchain data is queried through BigQuery.
- BigQuery filter → all contracts: candidate addresses · Filtered contract addresses are consolidated for collection.
- all contracts → get_contracts: input list · Collector uses the address list to fetch source code.
- all contracts → get_balance: input list · Balance collector reuses the same contract list.
- get_contracts → Etherscan: source fetch · Downloads Solidity source and related metadata from Etherscan.
- get_balance → Etherscan: balance fetch · Retrieves balance information from Etherscan.
- Etherscan → contracts: contract sources · Fetched source files are stored as the contract corpus.
- get_contracts → contracts csv: metadata · Collection output is packaged into the metadata archive.
- get_balance → balances: results · Balance collection writes the balance table.
- contracts → duplicates: deduplication · Corpus is filtered to identify duplicates and unique contracts.
- contracts → line counts: metrics · Contract sources are measured for line counts.
- duplicates → contracts csv: curation · Duplicate filtering contributes to the curated dataset bundle.
- contracts → LICENSE: license boundary · Repository license applies at the package level; source files may differ.