How simaarasteh/binpool works
Project Type and Purpose
• BinPool is an open-source dataset-creation tool (not a web app or microservice).
• It automates the end-to-end process of collecting, building, extracting, and packaging vulnerable vs. patched Debian binaries into a structured dataset for security analysis.
File Structure Analysis
• Top-level scripts (e.g., crawl_debian_snapshot.py, extract_deb_tar.py, extract_statistics.py) form a linear ETL pipeline:
• Crawl/Fetch: crawl_debian_snapshot.py, create_debian_url.py
• Download/Build: binpool_automation/ (automation wrappers), full_script.sh
• Extraction: extract_deb_files.py, extract_deb_tar.py, extract_binpool_bins.py, extract_source.py, extract_metadata.py
• Post-processing: create_binpool_datasheet.py, extract_statistics.py, binpool_info.py
• metadata/ holds intermediate pickle files mapping CVEs, CWEs, functions, packages.
• images/ holds documentation assets only.
• tasks/ is per-team-member tracking (out of scope for architecture).
Key Architectural Insights from README
• External data sources: Debian Security Tracker JSON, Debian Snapshot archive.
• Local Debian chroot (via debootstrap + quilt) to build binaries at optimization levels 0–3.
• Final dataset layout: per-CVE directories with vulnerable/opt*/debfiles/bins and patch/opt*/debfiles/bins.
Detailed Capture of Architectural Elements
• Debian chroot environment as a distinct execution context for building.