# tanzania-ai-community/twiga-warehouse architecture

> Architecture diagram of the GitHub repository tanzania-ai-community/twiga-warehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tanzania-ai-community/twiga-warehouse
- Repository: https://github.com/tanzania-ai-community/twiga-warehouse

## Overview

##### Project type and purpose
- This is a Python-based data-ingestion/warehouse tool (“Warehouse repository for Twiga”).
- It provides CLI scripts to register new classes, resources, subjects, and validate schemas, plus parsing modules to transform unstructured content into structured form, and persists everything in a database.

##### File structure
- Top level:
- .env.template / .python-version / pyproject.toml: environment and dependency management
- database/: ORM models, enums, utility functions for database access
- scripts/: CLI entry points (cli.py) and commands to scaffold new artifacts (new_class.py, new_resource.py, etc.)
- src/: parsing logic (langchain_chunking.py, pymu_parsing.py, unstructured_parsing.py)
- Separation between:
- CLI layer (scripts)
- Parsing layer (src)
- Persistence layer (database)

##### README
- Minimal, but confirms this is the “Warehouse” component of a larger Twiga ecosystem.
- External link points to broader context (Twiga data platform).

##### Main components and their relationships
a. Components:
1. CLI Layer
– scripts/cli.py: main command dispatcher
– scaffolding scripts (new_class, new_resource, new_subject)
– schema_validator.py: ensures JSON/YAML schemas adhere to rules
2. Parsing Layer
– src/langchain_chunking.py
– src/pymu_parsing.py
– src/unstructured_parsing.py
3. Database Layer
– database/models.py: ORM entity definitions
– database/enums.py: categorization constants
– database/utils.py: connection, session management, CRUD helpers
4. Configuration & Environment
– .env.template: DB credentials, API keys
– pyproject.toml / uv.lock: dependency definitions
b. Interactions:
- The user calls a CLI command → cli.py invokes the appropriate script.
- The script either:
- Runs a scaffolding action (creates new YAML/JSON resource files)
- Or triggers a parsing workflow:
– The parsing module reads raw input (e.g., documents, JSON blobs)
– Performs chunking or parsing (via unstructured / langchain libraries)
– Converts results into domain entities
– Passes entities to database/utils for persistence
- database/utils opens a DB session (via SQLAlchemy or similar), writes models, closes session.
c. Architectural patterns:
- Layered architecture (CLI → Parsing → Persistence)
- Separation of concerns: each script/module has one responsibility
- Use of Repository/Unit-of-Work via database/utils for transactions
d. Key technologies:
- Python 3.x (managed via pyproject.toml & poetry/poetry lock)
- SQLAlchemy (implied by models/utils)
- Unstructured (for document parsing)
- LangChain (for chunking)
- Click or Argparse (for CLI in scripts/cli.py)

##### Elements
- “CLI Layer” (scripts/cli.py + command scripts)
- “Parsing Modules” (each src/*.py)
- “Database Layer” (models, utils)
- “Configuration” (.env, pyproject.toml)
- CLI → Parsing Modules → Database
- Config → CLI & Parsing Modules & Database
- Parsing modules in another (e.g., green)
- Persistence layer in a third (e.g., orange)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration & Environment"
        ENV[".env.template"]:::config
        PYPROJECT["pyproject.toml"]:::config
        UVLOCK["uv.lock"]:::config
        PYVER[".python-version"]:::config
    end

    subgraph "CLI Layer"
        CLI["scripts/cli.py"]:::cli
        NEWCLASS["new_class.py"]:::cli
        NEWRES["new_resource.py"]:::cli
        NEWSUB["new_subject.py"]:::cli
        SCHEMA["schema_validator.py"]:::cli
    end

    subgraph "Parsing Modules"
        LC["langchain_chunking.py"]:::parse
        PMU["pymu_parsing.py"]:::parse
        UNSTR["unstructured_parsing.py"]:::parse
    end

    subgraph "Database Layer"
        MODELS["models.py"]:::db
        ENUMS["enums.py"]:::db
        UTILS["utils.py"]:::db
    end

    ENV -->|"provides config"| CLI
    ENV -->|"provides config"| LC
    ENV -->|"provides config"| UTILS

    CLI -->|"validate schema"| SCHEMA
    CLI -->|"execute scaffold"| NEWCLASS
    CLI -->|"execute scaffold"| NEWRES
    CLI -->|"execute scaffold"| NEWSUB

    CLI -->|"trigger parsing"| LC
    CLI -->|"trigger parsing"| PMU
    CLI -->|"trigger parsing"| UNSTR

    LC -->|"persist entity"| UTILS
    PMU -->|"persist entity"| UTILS
    UNSTR -->|"persist entity"| UTILS

    UTILS -->|"uses definitions"| MODELS
    UTILS -->|"uses constants"| ENUMS

    click CLI "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/scripts/cli.py"
    click NEWCLASS "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/scripts/new_class.py"
    click NEWRES "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/scripts/new_resource.py"
    click NEWSUB "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/scripts/new_subject.py"
    click SCHEMA "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/scripts/schema_validator.py"
    click LC "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/src/langchain_chunking.py"
    click PMU "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/src/pymu_parsing.py"
    click UNSTR "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/src/unstructured_parsing.py"
    click MODELS "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/database/models.py"
    click ENUMS "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/database/enums.py"
    click UTILS "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/database/utils.py"
    click ENV "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/.env.template"
    click PYPROJECT "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/pyproject.toml"
    click UVLOCK "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/uv.lock"
    click PYVER "https://github.com/tanzania-ai-community/twiga-warehouse/blob/main/.python-version"

    classDef cli fill:#bbdefb,stroke:#1e88e5,stroke-width:2px
    classDef parse fill:#c8e6c9,stroke:#43a047,stroke-width:2px
    classDef db fill:#ffcdd2,stroke:#e53935,stroke-width:2px
    classDef config fill:#eeeeee,stroke-dasharray: 5 5,stroke:#616161,stroke-width:2px
```
