# bigcode-project/the-stack-v2 architecture

> Architecture diagram of the GitHub repository bigcode-project/the-stack-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigcode-project/the-stack-v2
- Repository: https://github.com/bigcode-project/the-stack-v2

## Overview

##### Project Type & Purpose
- This is a data‐pipeline project for building “The Stack v2” and “StarCoder2Data” corpora.
- It ingests code repositories (GH Archive, Software Heritage), Kaggle datasets, GitHub issues/PRs, applies preprocessing (directory traversal, downloading, language‐specific compilation via IR), and outputs standardized, sharded datasets for model training.

##### File Structure Analysis
Top‐level directories and their roles:
– the_stack/… – Core ETL pipeline split into steps 0 through 3
– kaggle/… – Kaggle ingestion, dataset retrieval, manual sharding & processing utilities
– pull_requests_and_issues/… – GH issue & PR event ingestion & parsing (Ray‐based processing)
– ir/… – Language‐specific “compile” scripts to normalize code via an intermediate representation
– spark/… – Spark/Slurm scripts & env for distributed compute
– Jupyter_notebooks & lhq/… – Analysis & experimentation notebooks
Configuration:
– *.slurm & spark_env.sh for HPC job scheduling
– ray_server.py & ray_util.py for Ray cluster orchestration
Patterns:
– Modular ETL stages (numbered directories under the_stack)
– Separation of concerns (ingestion vs. preprocessing vs. metadata vs. compilation)

##### README Insights
- Dataset building for BigCode’s The Stack and StarCoder training data.

a. Main Components
– Data Sources (GH Archive, Software Heritage, Kaggle API, GitHub API for issues/PRs)
– Ingestion Modules (the_stack/0_repo_data, kaggle/, pull_requests_and_issues/)
– Preprocessing Stages (directory traversal, file downloading, IR compilation)
– Distributed Compute Layer (Slurm + Spark + Ray clusters)
– Utilities & Helpers (processing_utils, manual_sharding, enry setup)
– Interactive Layer (Jupyter notebooks, lhq.ipynb)
– Storage & Output (local FS / HDFS / S3 intermediates, final dataset shards, Hugging Face)

b. Relationships & Data Flows
– Branch from issues/PR ingestion into preprocessing or metadata enrichment

c. Architectural Patterns & Principles
– Pipeline decomposition: each numbered directory is an independent stage with defined inputs/outputs
– Distributed processing: leverages HPC scheduler (Slurm), Spark for bulk data ops, Ray for event‐driven parsing
– Language‐agnostic IR compilation ensures uniform code representation across multiple languages

d. Key Technologies & Frameworks
– Python (core), Slurm (spark/example.slurm), Spark (spark_env.sh), Ray (ray_server.py)
– Kaggle API & custom utils, enry for language classification, Jupyter for exploration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        GHArchive["GH Archive"]:::source
        SoftwareHeritage["Software Heritage"]:::source
        KaggleAPI["Kaggle API"]:::source
        GitHubAPI["GitHub Issues & PRs"]:::source
    end

    %% Ingestion Modules
    subgraph "Ingestion Modules"
        RepoData["0_repo_data Ingestion"]:::ingest
        KaggleMod["Kaggle Ingestion"]:::ingest
        PRIngest["Issues & PR Ingestion"]:::ingest
    end

    %% Preprocessing Stages
    subgraph "Preprocessing Stages"
        DirTrav["1_directory_traversal"]:::ingest
        FileDL["2_download_files"]:::ingest
        FileDS["3_file_dataset"]:::ingest
    end

    %% IR Compilation
    IR["IR Compilation Scripts"]:::ingest

    %% Distributed Compute
    subgraph "Distributed Compute"
        Slurm["Slurm Scheduler"]:::compute
        Spark["Spark Cluster"]:::compute
        Ray["Ray Cluster"]:::compute
    end

    %% Storage & Publishing
    subgraph "Storage & Output"
        InterStorage["Intermediate Storage (HDFS/S3)"]:::storage
        FinalDS["Final Dataset (Sharded)"]:::storage
        HF["Hugging Face Publishing"]:::storage
    end

    %% Interactive Layer
    Jupyter["Interactive Analysis Notebooks"]:::interactive

    %% Data Flows
    GHArchive -->|".json.gz"| RepoData
    SoftwareHeritage -->|".json.gz"| RepoData
    KaggleAPI -->|".zip / .tar"| KaggleMod
    GitHubAPI -->|"events stream"| PRIngest

    RepoData -->|"repo metadata"| DirTrav
    PRIngest -->|"parsed PRs/issues"| DirTrav
    KaggleMod -->|"raw datasets"| DirTrav

    DirTrav -->|"file paths list"| FileDL
    FileDL -->|"source files"| FileDS
    FileDS -->|"metadata.csv"| InterStorage

    InterStorage --> Spark
    InterStorage --> Ray
    InterStorage --> Slurm

    FileDS --> IR
    IR --> FinalDS

    FinalDS --> HF

    Spark --> FinalDS
    Ray --> FileDS
    Slurm --> Spark
    Slurm --> Ray

    FinalDS --> Jupyter

    %% Click Events - the_stack/0_repo_data
    click RepoData "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/0_repo_data/download_gharchive.py"
    click RepoData "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/0_repo_data/merge_gha_swh.py"
    click RepoData "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/0_repo_data/merge_gha_swh.slurm"

    %% Click Events - the_stack/1_directory_traversal
    click DirTrav "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/1_directory_traversal/find_file_paths.py"
    click DirTrav "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/1_directory_traversal/get_unique_files.py"
    click DirTrav "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/1_directory_traversal/*.slurm"

    %% Click Events - the_stack/2_download_files
    click FileDL "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/2_download_files/get_file_contents.py"
    click FileDL "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/2_download_files/get_file_contents.slurm"
    click FileDL "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/2_download_files/setup_enry.sh"

    %% Click Events - the_stack/3_file_dataset
    click FileDS "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/3_file_dataset/get_license_types.py"
    click FileDS "https://github.com/bigcode-project/the-stack-v2/blob/main/the_stack/3_file_dataset/get_license_types.slurm"

    %% Click Events - Kaggle module
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/retrieve_kaggle_datasets.py"
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/get_kaggle_raw_dataset.py"
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/retreive_metadata.py"
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/requirements.txt"
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/utils/manual_sharding.py"
    click KaggleMod "https://github.com/bigcode-project/the-stack-v2/blob/main/kaggle/utils/processing_utils.py"

    %% Click Events - PR ingestion
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/0_get_gharchive_events.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/process_pr_events.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/pr_parser.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/issue_parser.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/pr_issue_parser.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/ray_server.py"
    click PRIngest "https://github.com/bigcode-project/the-stack-v2/blob/main/pull_requests_and_issues/ray_util.py"

    %% Click Events - IR scripts
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_C++.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_C.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_D.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_Fortran.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_Go.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_Haskell.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_Nim.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_ObjectiveC.py"
    click IR "https://github.com/bigcode-project/the-stack-v2/blob/main/ir/Compile_Rust.py"

    %% Click Events - Spark & Slurm
    click Slurm "https://github.com/bigcode-project/the-stack-v2/blob/main/spark/example.slurm"
    click Spark "https://github.com/bigcode-project/the-stack-v2/blob/main/spark/spark_env.sh"

    %% Click Events - Interactive
    click Jupyter "https://github.com/bigcode-project/the-stack-v2/tree/main/Jupyter_notebooks/"
    click Jupyter "https://github.com/bigcode-project/the-stack-v2/blob/main/lhq/lhq.ipynb"

    %% Styles
    classDef source fill:#ffff99,stroke:#333,stroke-width:1px
    classDef ingest fill:#99ccff,stroke:#333,stroke-width:1px
    classDef compute fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef storage fill:#99ff99,stroke:#333,stroke-width:1px
    classDef interactive fill:#ffeb99,stroke:#333,stroke-width:1px
```
