How paulodder/debot works
Project type and purpose
• This is a data-pipeline and model-training project (“debot”) that ingests raw parliamentary documents, parses and stores them in a database, generates prompts, fine-tunes large language models (BloomZ and Alpaca) on that data, and provides scripts and deployment artifacts (Modal, Replicate) for inference.
• It is not a traditional web frontend/backend stack — it is a CLI-driven ETL + ML pipeline with optional API endpoints (api/api.py) and deployment layers.
File structure
• Top-level folders:
• api/ → lightweight HTTP API layer
• debot/ → core Python package
– db/ → connection, I/O, ORM abstraction
– parsers/ → document-specific parsing modules
– sync/ → sync actions, registrars, syncers to fetch & register docs in DB
– models/, llms/ → prompt‐generation, training utilities, model wrappers
• scripts/ → entry-point scripts (sync.py, inference.py, train_*.py)
• modal/ → Modal.com deployment definitions
• replicate_resources/ → Replicate.com (Cog) deployment artifacts
• docs/ → architecture & component documentation
• reports/ → prototypes, experimental notes
• Config files:
• .env(.example) for environment variables
• requirements.txt & requirements_training.txt
• setup.py (editable mode install)
Key components from README
• Data ingestion via scripts/sync.py against REMOTE_BASE_URL
• PostgreSQL as the persistent store (DB_HOST, DB_NAME, etc.)
• PDF/text parsing (pymupdf, debot/parsers)
• Prompt generation & dataset assembly (debot/prompt_* modules)
• Model training (scripts/train_bloom.py, scripts/train_alpaca.py) using Hugging Face Trainer + wandb logging
• Inference script (scripts/inference.py)
• Cloud deployment (Modal + Replicate)
Main components and interactions
a. External Data Source
• HTTP endpoint (REMOTE_BASE_URL) hosting raw data archives
b. Sync & Parsing Service
• scripts/sync.py orchestrates debot/sync/{resources,actions,registrars,syncers} + debot/parsers
• Two subcomponents:
• Downloader (download_doc, download_json syncers)
• Parser & Registrar (document_structure, enregistrars writing to DB via ORM)
c. Persistence Layer
• PostgreSQL DB
• File system directories for raw & processed files (DATA_DIR)
d. Prompt Generation & Dataset Builder
• debot/prompt_generator.py, prompt_utils, prompt_dicts
• Produces training JSON from parsed data in DB/FS
e. Model Training
• scripts/train_*.py invoking transformers.Trainer
• Inputs: base model weights (BASE_MODELS_DIRECTORY), dataset (from prior step)
• Outputs: fine-tuned model weights in TRAINED_MODELS_DIRECTORY + wandb logs
f. Inference
• scripts/inference.py / api/api.py
• Inputs: trained model weights + new data
• Outputs: predictions (JSON) or HTTP responses
g. Deployment
• Modal definitions (modal/*.py) for orchestration on Modal.com
• Replicate setup (replicate_resources/) for on-demand GPU inference
• Raw Doc Storage → Sync & Parse Pipeline (“download”)
• Sync & Parse Pipeline → PostgreSQL + FS (“store parsed docs & metadata”)
• PostgreSQL + FS → Training Data Generator (“query + assemble prompts”)
• Training Data Generator → Model Trainer (“training JSON + base weights”)
• Model Trainer → TRAINED_MODELS_DIRECTORY (“fine-tuned weights”)
• TRAINED_MODELS_DIRECTORY → Inference Service (“load model”)
• Inference Service ↔ Clients (CLI or HTTP via api/)
• Model Trainer & Inference Service → Modal + Replicate (“deployment artifacts”)
Architectural patterns & design principles
• Modular “syncer” plugin pattern: debot/sync resources, registrars & syncers decoupled
• Separation of concerns: ingestion/parsing vs. persistence vs. ML vs. serving
• Use of environment variables (.env) to decouple config from code
• CLI orchestration via scripts (no monolithic app)
Technologies & libraries to annotate
• Python 3.11
• FastAPI (if used in api/api.py) or simple Flask/WSGI
• psycopg2 / SQLAlchemy ORM (in debot/db/orm.py)
• Hugging Face Transformers & Tokenizers (tokenizers/)
• wandb for experiment logging
• pymupdf for PDF parsing
• Modal SDK, Replicate Cog
Essential elements to capture
• Clear component boundaries & labels