# simaarasteh/binpool architecture

> Architecture diagram of the GitHub repository simaarasteh/binpool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/simaarasteh/binpool
- Repository: https://github.com/simaarasteh/binpool

## Overview

##### Project Type and Purpose
- BinPool is an open-source dataset-creation tool (not a web app or microservice).
- It automates the end-to-end process of collecting, building, extracting, and packaging vulnerable vs. patched Debian binaries into a structured dataset for security analysis.

##### File Structure Analysis
- Top-level scripts (e.g., crawl_debian_snapshot.py, extract_deb_tar.py, extract_statistics.py) form a linear ETL pipeline:
- **Crawl/Fetch**: crawl_debian_snapshot.py, create_debian_url.py
- **Download/Build**: binpool_automation/ (automation wrappers), full_script.sh
- **Extraction**: extract_deb_files.py, extract_deb_tar.py, extract_binpool_bins.py, extract_source.py, extract_metadata.py
- **Post-processing**: create_binpool_datasheet.py, extract_statistics.py, binpool_info.py
- `metadata/` holds intermediate pickle files mapping CVEs, CWEs, functions, packages.
- `images/` holds documentation assets only.
- `tasks/` is per-team-member tracking (out of scope for architecture).

##### Key Architectural Insights from README
- External data sources: Debian Security Tracker JSON, Debian Snapshot archive.
- Local Debian chroot (via debootstrap + quilt) to build binaries at optimization levels 0–3.
- Final dataset layout: per-CVE directories with `vulnerable/opt*/debfiles/bins` and `patch/opt*/debfiles/bins`.

##### Detailed Capture of Architectural Elements
- Debian chroot environment as a distinct execution context for building.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    ES1["Debian Security Tracker JSON"]:::external
    ES2["Debian Snapshot URL Generator"]:::external

    %% Crawler/URL Generator
    subgraph "Crawler / URL Generator"
        CS["crawl_debian_snapshot.py"]:::pipeline
        CU["create_debian_url.py"]:::pipeline
    end

    %% Downloader & Builder
    subgraph "Downloader & Builder"
        AUTO["binpool_automation/"]:::pipeline
        FULL["full_script.sh"]:::pipeline
        CHROOT["Debian chroot environment\n(Opt0-Opt3)"]:::pipeline
        BUILDS[".deb packages"]:::pipeline
    end

    %% Extractor
    subgraph "Extractor"
        ED["extract_deb_files.py"]:::pipeline
        ET["extract_deb_tar.py"]:::pipeline
        EB["extract_binpool_bins.py"]:::pipeline
        ESRC["extract_source.py"]:::pipeline
        EMETA["extract_metadata.py"]:::pipeline
    end

    %% Metadata Processor
    subgraph "Metadata Processor"
        MDIR["metadata/"]:::storage
        FR["find_debian_release.py"]:::pipeline
        FV["find_debian_version.py"]:::pipeline
        FF["find_function_bounderies.py"]:::pipeline
    end

    %% Stats & Datasheet Generator
    subgraph "Stats & Datasheet Generator"
        STAT["extract_statistics.py"]:::pipeline
        DS["create_binpool_datasheet.py"]:::pipeline
        BI["binpool_info.py"]:::pipeline
        BIJ["binpool_info.json"]:::pipeline
        SD["structure_dataset.py"]:::pipeline
    end

    %% Storage
    subgraph "Storage"
        RAW["Local FS: raw downloads\n& chroot envs"]:::storage
        DATASET["Final Dataset FS\n(per-CVE bins)"]:::storage
    end

    %% Documentation Assets
    IMG["images/"]:::docs

    %% Connections
    ES1 -->|"CVE JSON"| CS
    ES2 -->|".dsc link"| CU
    CS --> CU
    CU --> AUTO
    AUTO --> FULL
    FULL --> CHROOT
    CHROOT --> BUILDS
    BUILDS --> ED
    BUILDS --> ET
    ED --> EB
    ET --> EB
    EB --> DATASET
    ED --> ESRC
    ET --> ESRC
    EB --> ESRC
    ED --> EMETA
    ET --> EMETA
    ESRC --> EMETA
    EMETA --> MDIR
    MDIR --> FR
    MDIR --> FV
    MDIR --> FF
    FF --> STAT
    FV --> STAT
    FR --> STAT
    STAT --> DS
    DS --> BI
    BI --> BIJ
    BIJ --> SD
    SD --> DATASET
    RAW --> AUTO

    %% Click Events
    click ES1 "https://github.com/simaarasteh/binpool/blob/main/debian_security.json"
    click ES2 "https://github.com/simaarasteh/binpool/blob/main/create_debian_url.py"
    click CS "https://github.com/simaarasteh/binpool/blob/main/crawl_debian_snapshot.py"
    click CU "https://github.com/simaarasteh/binpool/blob/main/create_debian_url.py"
    click AUTO "https://github.com/simaarasteh/binpool/tree/main/binpool_automation/"
    click FULL "https://github.com/simaarasteh/binpool/blob/main/full_script.sh"
    click ED "https://github.com/simaarasteh/binpool/blob/main/extract_deb_files.py"
    click ET "https://github.com/simaarasteh/binpool/blob/main/extract_deb_tar.py"
    click EB "https://github.com/simaarasteh/binpool/blob/main/extract_binpool_bins.py"
    click ESRC "https://github.com/simaarasteh/binpool/blob/main/extract_source.py"
    click EMETA "https://github.com/simaarasteh/binpool/blob/main/extract_metadata.py"
    click MDIR "https://github.com/simaarasteh/binpool/tree/main/metadata/"
    click FR "https://github.com/simaarasteh/binpool/blob/main/find_debian_release.py"
    click FV "https://github.com/simaarasteh/binpool/blob/main/find_debian_version.py"
    click FF "https://github.com/simaarasteh/binpool/blob/main/find_function_bounderies.py"
    click STAT "https://github.com/simaarasteh/binpool/blob/main/extract_statistics.py"
    click DS "https://github.com/simaarasteh/binpool/blob/main/create_binpool_datasheet.py"
    click BI "https://github.com/simaarasteh/binpool/blob/main/binpool_info.py"
    click BIJ "https://github.com/simaarasteh/binpool/blob/main/binpool_info.json"
    click SD "https://github.com/simaarasteh/binpool/blob/main/structure_dataset.py"
    click IMG "https://github.com/simaarasteh/binpool/tree/main/images/"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000
    classDef pipeline fill:#98FB98,stroke:#000
    classDef storage fill:#FFA500,stroke:#000
    classDef docs fill:#D3D3D3,stroke:#000
```
