# pasta41/book-extraction-science architecture

> Architecture diagram of the GitHub repository pasta41/book-extraction-science, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pasta41/book-extraction-science
- Repository: https://github.com/pasta41/book-extraction-science

## Overview

##### Project type and purpose
- This is an academic research codebase, not a web or mobile app.
- Purpose: implement and evaluate a probabilistic extraction procedure on open-weight LLMs and reconstruct a target book (e.g., Harry Potter).
- Two entry points at top level:
– probabilistic_extraction.py: runs the extraction pipeline
– reconstruct_hp1.py: demonstrates one reconstruction recipe

##### File structure
Top-level
- config/basic.yaml – global settings (model paths, hyperparameters)
- data/… – raw text of a sample book
- probabilistic_extraction.py – script for extraction experiments
- reconstruct_hp1.py – script showcasing a full reconstruction run
script/memorization (core library)
- config.py, constants.py – internal defaults and constants
- execute.py, inference.py, pz.py – core probabilistic logic & distribution computations
- runner.py – orchestrates a full experiment run (loads config, data, model, executes extraction/inference, saves results)
- loaders/
– data_loader.py – reads and preprocesses book text
– model_loader.py – loads the LLM (e.g., Llama 3.1 70B via HF)
- utils/ – path helpers, tokenization utilities

##### Extract insights from README
- Confirms two main workflows (extraction vs reconstruction)
- Emphasizes running on 4 A100 GPUs with a Llama model
- Data is a CC-BY sample book; experiments are read-only

a. Main components
1. Configuration Layer
– config/basic.yaml & script/memorization/config.py
2. Data Layer
– data folder (raw text)
– Data Loader
3. Model Layer
– Model Loader (LLM checkpoint via HF/torch)
4. Extraction Engine
– inference.py (calls model, computes token probabilities)
– pz.py / execute.py (probabilistic scoring)
5. Reconstruction Pipeline
– runner.py orchestrates rounds of inference + sampling logic
– reconstruct_hp1.py wraps runner for HP-specific parameters
6. Visualization & Reporting
– plots/ modules to generate charts, saving outputs to disk
7. Entry-point Scripts
– probabilistic_extraction.py
– reconstruct_hp1.py

b. Relationships & data flow
– Start: Entry-point script reads basic.yaml →
– Runner invokes Data Loader → raw text → tokenization util → token streams →
– Model Loader initializes LLM →
– Inference module sends token chunks → model → probability distributions →
– Probabilistic Engine scores and extracts next tokens →
– Reconstruction logic aggregates outputs →
– Final text written to disk →
– Plotting module reads intermediate logs → visualizes results

c. Architectural patterns & principles
- Modular pipeline: clear separation of data loading, model loading, inference, post-processing.
- Single responsibility per module (loaders, inference, plotting).
- Config-driven: behavior controlled by YAML + constants.

d. Key technologies & libraries
- PyTorch / HuggingFace Transformers (model_loader)
- YAML for configuration
- matplotlib / seaborn (plots)
- Tokenization utilities (script/memorization/utils/tokenize.py)

##### Detail without over-splitting
- Essential flows: how raw text becomes extracted sequences.
- How parameters (from config/basic.yaml) influence every stage.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        ConfigY["basic.yaml"]:::config
        ConfigP["config.py"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        DataD["Raw Text Data"]:::data
        DLoader["Data Loader"]:::core
    end

    %% Model Layer
    subgraph "Model Layer"
        MLoader["Model Loader"]:::core
    end

    %% Extraction Engine
    subgraph "Extraction Engine"
        Inference["LLM Inference"]:::core
        PZ["Probability Computation"]:::core
        Execute["Extraction Orchestration"]:::core
    end

    %% Reconstruction Pipeline
    subgraph "Reconstruction Pipeline"
        Runner["Runner Orchestration"]:::core
    end

    %% Utilities
    subgraph "Utilities"
        Paths["Path Helpers"]:::core
        Tokenize["Tokenization Utils"]:::core
    end

    %% Visualization & Reporting
    subgraph "Visualization & Reporting"
        PlotUtils["Plot Utilities"]:::viz
        Palettes["Color Palettes"]:::viz
    end

    %% Entry-Point Scripts
    subgraph "Entry-Point Scripts"
        ExtractScript["Probabilistic Extraction"]:::script
        ReconstructScript["Reconstruct HP1"]:::script
    end

    %% Data Store for outputs
    Output(("Extracted Text")):::data

    %% Connections
    ExtractScript -->|"uses config"| ConfigY
    ReconstructScript -->|"uses config"| ConfigY
    ConfigP --> Execute
    ConfigP --> Runner

    ExtractScript --> Execute
    ReconstructScript --> Runner

    Runner --> DLoader
    DLoader -->|reads| DataD
    DLoader --> Tokenize
    DLoader --> Paths

    Runner --> MLoader
    MLoader --> Inference
    Inference -->|"token probabilities"| PZ
    PZ --> Execute
    Execute --> Runner

    Execute -->|writes| Output
    Runner --> PlotUtils
    PlotUtils --> Palettes

    %% Click Events
    click ConfigY "https://github.com/pasta41/book-extraction-science/blob/main/config/basic.yaml"
    click ConfigP "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/config.py"
    click DataD "https://github.com/pasta41/book-extraction-science/tree/main/data/"
    click DLoader "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/loaders/data_loader.py"
    click MLoader "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/loaders/model_loader.py"
    click Inference "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/inference.py"
    click PZ "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/pz.py"
    click Execute "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/execute.py"
    click Runner "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/runner.py"
    click ExtractScript "https://github.com/pasta41/book-extraction-science/blob/main/probabilistic_extraction.py"
    click ReconstructScript "https://github.com/pasta41/book-extraction-science/blob/main/reconstruct_hp1.py"
    click PlotUtils "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/plots/plot_utils.py"
    click Palettes "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/plots/palettes.py"
    click Paths "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/utils/paths.py"
    click Tokenize "https://github.com/pasta41/book-extraction-science/blob/main/script/memorization/utils/tokenize.py"

    %% Styles
    classDef config fill:#cce5ff,stroke:#004085,color:#004085
    classDef data fill:#b8daff,stroke:#004085,color:#004085
    classDef core fill:#d4edda,stroke:#155724,color:#155724
    classDef script fill:#fff3cd,stroke:#856404,color:#856404
    classDef viz fill:#e2d6f3,stroke:#5a3e85,color:#5a3e85
```
