How pasta41/book-extraction-science works
Project type and purpose
• This is an academic research codebase, not a web or mobile app.
• Purpose: implement and evaluate a probabilistic extraction procedure on open-weight LLMs and reconstruct a target book (e.g., Harry Potter).
• Two entry points at top level:
– probabilistic_extraction.py: runs the extraction pipeline
– reconstruct_hp1.py: demonstrates one reconstruction recipe
File structure
Top-level
• config/basic.yaml – global settings (model paths, hyperparameters)
• data/… – raw text of a sample book
• probabilistic_extraction.py – script for extraction experiments
• reconstruct_hp1.py – script showcasing a full reconstruction run
script/memorization (core library)
• config.py, constants.py – internal defaults and constants
• execute.py, inference.py, pz.py – core probabilistic logic & distribution computations
• runner.py – orchestrates a full experiment run (loads config, data, model, executes extraction/inference, saves results)
• loaders/
– data_loader.py – reads and preprocesses book text
– model_loader.py – loads the LLM (e.g., Llama 3.1 70B via HF)
• utils/ – path helpers, tokenization utilities
Extract insights from README
• Confirms two main workflows (extraction vs reconstruction)
• Emphasizes running on 4 A100 GPUs with a Llama model
• Data is a CC-BY sample book; experiments are read-only
a. Main components
1. Configuration Layer
– config/basic.yaml & script/memorization/config.py
2. Data Layer
– data folder (raw text)
– Data Loader
3. Model Layer
– Model Loader (LLM checkpoint via HF/torch)
4. Extraction Engine
– inference.py (calls model, computes token probabilities)
– pz.py / execute.py (probabilistic scoring)
5. Reconstruction Pipeline
– runner.py orchestrates rounds of inference + sampling logic
– reconstruct_hp1.py wraps runner for HP-specific parameters
6. Visualization & Reporting
– plots/ modules to generate charts, saving outputs to disk
7. Entry-point Scripts
– probabilistic_extraction.py
– reconstruct_hp1.py
b. Relationships & data flow
– Start: Entry-point script reads basic.yaml →
– Runner invokes Data Loader → raw text → tokenization util → token streams →
– Model Loader initializes LLM →
– Inference module sends token chunks → model → probability distributions →
– Probabilistic Engine scores and extracts next tokens →
– Reconstruction logic aggregates outputs →
– Final text written to disk →
– Plotting module reads intermediate logs → visualizes results
c. Architectural patterns & principles
• Modular pipeline: clear separation of data loading, model loading, inference, post-processing.
• Single responsibility per module (loaders, inference, plotting).
• Config-driven: behavior controlled by YAML + constants.
d. Key technologies & libraries
• PyTorch / HuggingFace Transformers (model_loader)
• YAML for configuration
• matplotlib / seaborn (plots)
• Tokenization utilities (script/memorization/utils/tokenize.py)
Detail without over-splitting
• Essential flows: how raw text becomes extracted sequences.
• How parameters (from config/basic.yaml) influence every stage.