# paulodder/debot architecture

> Architecture diagram of the GitHub repository paulodder/debot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paulodder/debot
- Repository: https://github.com/paulodder/debot

## Overview

##### Project type and purpose
- This is a data-pipeline and model-training project (“debot”) that ingests raw parliamentary documents, parses and stores them in a database, generates prompts, fine-tunes large language models (BloomZ and Alpaca) on that data, and provides scripts and deployment artifacts (Modal, Replicate) for inference.
- It is not a traditional web frontend/backend stack — it is a CLI-driven ETL + ML pipeline with optional API endpoints (api/api.py) and deployment layers.

##### File structure
- Top-level folders:
- api/            → lightweight HTTP API layer
- debot/          → core Python package
– db/           → connection, I/O, ORM abstraction
– parsers/     → document-specific parsing modules
– sync/        → sync actions, registrars, syncers to fetch & register docs in DB
– models/, llms/ → prompt‐generation, training utilities, model wrappers
- scripts/        → entry-point scripts (sync.py, inference.py, train_*.py)
- modal/          → Modal.com deployment definitions
- replicate_resources/ → Replicate.com (Cog) deployment artifacts
- docs/           → architecture & component documentation
- reports/        → prototypes, experimental notes
- Config files:
- .env(.example) for environment variables
- requirements.txt & requirements_training.txt
- setup.py (editable mode install)

##### Key components from README
- Data ingestion via scripts/sync.py against REMOTE_BASE_URL
- PostgreSQL as the persistent store (DB_HOST, DB_NAME, etc.)
- PDF/text parsing (pymupdf, debot/parsers)
- Prompt generation & dataset assembly (debot/prompt_* modules)
- Model training (scripts/train_bloom.py, scripts/train_alpaca.py) using Hugging Face Trainer + wandb logging
- Inference script (scripts/inference.py)
- Cloud deployment (Modal + Replicate)

##### Main components and interactions
a. External Data Source
- HTTP endpoint (REMOTE_BASE_URL) hosting raw data archives
b. Sync & Parsing Service
- scripts/sync.py orchestrates debot/sync/{resources,actions,registrars,syncers} + debot/parsers
- Two subcomponents:
- Downloader (download_doc, download_json syncers)
- Parser & Registrar (document_structure, enregistrars writing to DB via ORM)
c. Persistence Layer
- PostgreSQL DB
- File system directories for raw & processed files (DATA_DIR)
d. Prompt Generation & Dataset Builder
- debot/prompt_generator.py, prompt_utils, prompt_dicts
- Produces training JSON from parsed data in DB/FS
e. Model Training
- scripts/train_*.py invoking transformers.Trainer
- Inputs: base model weights (BASE_MODELS_DIRECTORY), dataset (from prior step)
- Outputs: fine-tuned model weights in TRAINED_MODELS_DIRECTORY + wandb logs
f. Inference
- scripts/inference.py / api/api.py
- Inputs: trained model weights + new data
- Outputs: predictions (JSON) or HTTP responses
g. Deployment
- Modal definitions (modal/*.py) for orchestration on Modal.com
- Replicate setup (replicate_resources/) for on-demand GPU inference

- Raw Doc Storage → Sync & Parse Pipeline (“download”)
- Sync & Parse Pipeline → PostgreSQL + FS (“store parsed docs & metadata”)
- PostgreSQL + FS → Training Data Generator (“query + assemble prompts”)
- Training Data Generator → Model Trainer (“training JSON + base weights”)
- Model Trainer → TRAINED_MODELS_DIRECTORY (“fine-tuned weights”)
- TRAINED_MODELS_DIRECTORY → Inference Service (“load model”)
- Inference Service ↔ Clients (CLI or HTTP via api/)
- Model Trainer & Inference Service → Modal + Replicate (“deployment artifacts”)

##### Architectural patterns & design principles
- Modular “syncer” plugin pattern: debot/sync resources, registrars & syncers decoupled
- Separation of concerns: ingestion/parsing vs. persistence vs. ML vs. serving
- Use of environment variables (.env) to decouple config from code
- CLI orchestration via scripts (no monolithic app)

##### Technologies & libraries to annotate
- Python 3.11
- FastAPI (if used in api/api.py) or simple Flask/WSGI
- psycopg2 / SQLAlchemy ORM (in debot/db/orm.py)
- Hugging Face Transformers & Tokenizers (tokenizers/)
- wandb for experiment logging
- pymupdf for PDF parsing
- Modal SDK, Replicate Cog

##### Essential elements to capture
- Clear component boundaries & labels

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Raw Data
    RawDoc["Raw Doc Storage"]:::external

    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        SyncParse["Sync & Parse Pipeline"]:::compute
        SyncPlugins["Sync Actions & Plugins"]:::compute
        Parsers["Document Parsers"]:::compute
        SyncParse --> SyncPlugins
        SyncParse --> Parsers
    end

    %% Persistence
    DBFS["PostgreSQL + FS"]:::storage

    %% Training Data Generation
    TrainingData["Training Data Generator"]:::compute

    %% Model Training
    ModelTrainer["Model Trainer (HF Trainer + wandb)"]:::compute

    %% Trained Models Storage
    TrainedModels["Trained Models Dir"]:::storage

    %% Inference
    InferenceService["Inference Service (CLI & API)"]:::compute

    %% Deployment
    Modal["Modal"]:::external
    Replicate["Replicate"]:::external

    %% Clients
    Clients["Clients\n(CLI or HTTP)"]:::external

    %% Data Flows
    RawDoc -->|download| SyncParse
    SyncParse -->|store parsed docs & metadata| DBFS
    DBFS -->|query + assemble prompts| TrainingData
    TrainingData -->|training JSON + base weights| ModelTrainer
    ModelTrainer -->|fine-tuned weights| TrainedModels
    TrainedModels -->|load model| InferenceService
    Clients -->|request| InferenceService
    InferenceService -->|response| Clients
    ModelTrainer -->|deployment artifacts| Modal
    InferenceService -->|deployment artifacts| Modal
    ModelTrainer -->|deployment artifacts| Replicate
    InferenceService -->|deployment artifacts| Replicate

    %% Click Events
    click SyncParse "https://github.com/paulodder/debot/blob/main/scripts/sync.py"
    click SyncPlugins "https://github.com/paulodder/debot/tree/main/debot/sync/"
    click Parsers "https://github.com/paulodder/debot/tree/main/debot/parsers/"
    click DBFS "https://github.com/paulodder/debot/tree/main/debot/db/"
    click TrainingData "https://github.com/paulodder/debot/blob/main/debot/prompt_generator.py"
    click ModelTrainer "https://github.com/paulodder/debot/blob/main/scripts/train_bloom.py"
    click ModelTrainer "https://github.com/paulodder/debot/blob/main/scripts/train_alpaca.py"
    click InferenceService "https://github.com/paulodder/debot/blob/main/scripts/inference.py"
    click InferenceService "https://github.com/paulodder/debot/blob/main/api/api.py"
    click Modal "https://github.com/paulodder/debot/tree/main/modal/"
    click Replicate "https://github.com/paulodder/debot/tree/main/replicate_resources/"

    %% Styles
    classDef compute fill:#a3d5f7,stroke:#333,stroke-width:1px
    classDef storage fill:#9be7a5,stroke:#333,stroke-width:1px
    classDef external fill:#f7c58f,stroke:#333,stroke-width:1px
```
