# absurdcoder/mini_llm_project architecture

> Architecture diagram of the GitHub repository absurdcoder/mini_llm_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/absurdcoder/mini_llm_project
- Repository: https://github.com/absurdcoder/mini_llm_project

## Overview

##### Project Type and Purpose
- This is a command-line application (“Mini LLM Project – Enhanced CLI with Transfer Learning”) for training, fine-tuning, testing, and running inference on both custom and Hugging Face transformer‐based language models. It provides modular tokenizers, model architectures, data pipelines, and utilities, all orchestrated via a unified CLI (`main.py`).

##### File Structure Analysis
- Top-level:
- CLI Entrypoints: `main.py`, `enhanced_main.py`
- Package modules: `ctokenizers/`, `models/`, `training/`, `testing/`, `ui/`, `utils/`
- Data directories: `data/`, plus runtime folders for `checkpoints/`, `hf_models/`, `test_results/`, `generations/`
- Configuration/build: `requirements.txt`, `setup.py`
- Patterns:
- Clear separation of concerns (tokenization, modeling, training, testing, UI, utilities)
- Plugin-style model/tokenizer interfaces (`base_tokenizer.py`, `base_model.py`)
- CLI orchestrator drives all operations by invoking sub-modules.

##### README Insights
- Unified CLI interface (`--mode train|retrain|test|generate`), modular support for custom vs. pretrained HF models.
- Multi-format data loader, flexible data split strategies.
- Transfer learning: offline HF download, fine-tune, retrain.
- Testing & generation pipelines.
- Detailed logging configurable with `--verbose`.

##### Diagramming the Architecture
a. Main Components
1. CLI Orchestrator
– `main.py` / `enhanced_main.py`
2. Data Ingestion Layer
– `utils/file_loader.py`, `training/data_utils.py`, `training/enhanced_data_utils.py`
3. Tokenization Layer
– Custom: `ctokenizers/*`
– HF: via `transformers` library, offline downloader `utils/hf_offline_downloader.py`
4. Model Layer
– Interfaces: `models/base_model.py`
– Implementations: `transformer_model.py`, `encoder_model.py`, `enhanced_transformer_components.py`, `rag_utils.py`
– HF integration: loaded via `transformers`
5. Training Engine
– `training/trainer.py`, `training/enhanced_trainer.py`, `training/enhanced_detailed_trainer.py`, `training/model_extraction.py`
6. Testing & Evaluation
– `testing/model_tester.py`, `testing/test_*.py`
7. Inference/UI
– CLI generate mode in `main.py`
– Optional local web UI in `ui/app.py`, `ui/rewritten_app.py`, `ui/simplified_app.py`
8. Persistence & Artifacts
– Data: `data/`
– Checkpoints: `checkpoints/`
– Offline HF models: `hf_models/`
– Test outputs: `test_results/`
– Generations: `generations/`
9. Utilities & Logging
– `utils/helpers.py`, `utils/validation.py`
– Logging via CLI flags and internal logger calls
10. External Dependencies
– PyTorch, Hugging Face Transformers & Tokenizers, HF Hub, tqdm, BeautifulSoup, PDF parser

b. Relationships & Data Flow
- CLI → Data Ingestion → Tokenization → Model Construction/Loading → Trainer/Test/Generate → Persistence
- Offline downloader → HF Model/Tokenizer directories → Model Loading
- Data files → Loader
- Loader → Tokenizer
- Tokenizer → Batch → Model/Trainer
- Trainer → Checkpoints/Logs
- Test/Eval → Test Results
- Generate → Generations

c. Architectural Patterns
- Modular plugin interfaces (BaseTokenizer, BaseModel) for extensibility
- Single-point CLI orchestration with sub‐commands (train/retrain/test/generate)
- Separation of compute (models/trainer) vs. data I/O vs. configuration
- Offline/online hybrid model sourcing (HF downloader + local)

d. Key Technologies & Frameworks
- Python CLI (argparse)
- PyTorch for model implementation/training
- Hugging Face Transformers & Tokenizers for pretrained models
- Custom BPE/character tokenizers
- tqdm for progress bars
- BeautifulSoup & pypdf for multi‐format data loading

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Orchestrator
    subgraph "Orchestration Layer"
        direction TB
        CLI1["CLI (main.py)"]:::compute
        CLI2["CLI Enhanced (enhanced_main.py)"]:::compute
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        DL1["File Loader"]:::compute
        DL2["Data Utils"]:::compute
        DL3["Enhanced Data Utils"]:::compute
    end

    %% Tokenization Layer
    subgraph "Tokenization Layer"
        direction TB
        TK1("Base Tokenizer"):::compute
        TK2["BPE Tokenizer"]:::compute
        TK3["Character Tokenizer"]:::compute
        TK4["Fixed BPE Tokenizer"]:::compute
        TK5["HF Offline Downloader"]:::external
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        M0("Base Model Interface"):::compute
        M1["Transformer Model"]:::compute
        M2["Encoder Model"]:::compute
        M3["Transformer Components"]:::compute
        M4["Enhanced Transformer Components"]:::compute
        M5["RAG Utilities"]:::compute
    end

    %% Training Engine
    subgraph "Training Engine"
        direction TB
        T1["Trainer"]:::compute
        T2["Enhanced Trainer"]:::compute
        T3["Enhanced Detailed Trainer"]:::compute
        T4["Model Extraction"]:::compute
    end

    %% Testing & Evaluation
    subgraph "Testing & Evaluation"
        direction TB
        TE1["Model Tester"]:::compute
        TE2["Test Enhanced Functionality"]:::compute
        TE3["Test Enhancements"]:::compute
    end

    %% Inference / UI
    subgraph "Inference/UI"
        direction TB
        UI1["App UI"]:::compute
        UI2["Rewritten App"]:::compute
        UI3["Simplified App"]:::compute
    end

    %% Persistence & Artifacts
    subgraph "Persistence & Artifacts"
        direction TB
        P1["Data (data/)"]:::data
        P2["Checkpoints"]:::data
        P3["HF Models"]:::data
        P4["Test Results"]:::data
        P5["Generations"]:::data
    end

    %% Utilities & Validation
    subgraph "Utilities"
        direction TB
        U1["Helpers"]:::util
        U2["Validation"]:::util
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        EX1(["PyTorch"]):::external
        EX2(["HF Transformers"]):::external
    end

    %% Flows
    CLI1 -->|"load data"| DL1
    CLI2 --> DL1
    DL1 --> DL2
    DL2 --> DL3
    DL3 --> TK1
    TK1 --> TK2
    TK1 --> TK3
    TK1 --> TK4
    TK5 --> TK1
    TK2 --> M0
    TK3 --> M0
    TK4 --> M0
    M0 --> M1
    M0 --> M2
    M1 --> M3
    M1 --> M4
    M1 --> M5
    M2 --> M3
    M3 --> T1
    M4 --> T2
    M5 --> T3
    T1 --> P2
    T2 --> P2
    T3 --> P2
    T4 --> P2
    CLI1 --> T1
    CLI1 --> TE1
    CLI1 --> UI1
    TE1 --> P4
    TE2 --> P4
    TE3 --> P4
    UI1 --> P5
    UI2 --> P5
    UI3 --> P5

    %% Click Events
    click CLI1 "https://github.com/absurdcoder/mini_llm_project/blob/main/main.py"
    click CLI2 "https://github.com/absurdcoder/mini_llm_project/blob/main/enhanced_main.py"
    click DL1 "https://github.com/absurdcoder/mini_llm_project/blob/main/utils/file_loader.py"
    click DL2 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/data_utils.py"
    click DL3 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/enhanced_data_utils.py"
    click TK1 "https://github.com/absurdcoder/mini_llm_project/blob/main/ctokenizers/base_tokenizer.py"
    click TK2 "https://github.com/absurdcoder/mini_llm_project/blob/main/ctokenizers/bpe_tokenizer.py"
    click TK3 "https://github.com/absurdcoder/mini_llm_project/blob/main/ctokenizers/character_tokenizer.py"
    click TK4 "https://github.com/absurdcoder/mini_llm_project/blob/main/ctokenizers/fixed_bpe_tokenizer.py"
    click TK5 "https://github.com/absurdcoder/mini_llm_project/blob/main/utils/hf_offline_downloader.py"
    click M0 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/base_model.py"
    click M1 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/transformer_model.py"
    click M2 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/encoder_model.py"
    click M3 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/transformer_components.py"
    click M4 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/enhanced_transformer_components.py"
    click M5 "https://github.com/absurdcoder/mini_llm_project/blob/main/models/rag_utils.py"
    click T1 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/trainer.py"
    click T2 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/enhanced_trainer.py"
    click T3 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/enhanced_detailed_trainer.py"
    click T4 "https://github.com/absurdcoder/mini_llm_project/blob/main/training/model_extraction.py"
    click TE1 "https://github.com/absurdcoder/mini_llm_project/blob/main/testing/model_tester.py"
    click TE2 "https://github.com/absurdcoder/mini_llm_project/blob/main/testing/test_enhanced_functionality.py"
    click TE3 "https://github.com/absurdcoder/mini_llm_project/blob/main/testing/test_enhancements.py"
    click UI1 "https://github.com/absurdcoder/mini_llm_project/blob/main/ui/app.py"
    click UI2 "https://github.com/absurdcoder/mini_llm_project/blob/main/ui/rewritten_app.py"
    click UI3 "https://github.com/absurdcoder/mini_llm_project/blob/main/ui/simplified_app.py"
    click P1 "https://github.com/absurdcoder/mini_llm_project/tree/main/data/"
    click U1 "https://github.com/absurdcoder/mini_llm_project/blob/main/utils/helpers.py"
    click U2 "https://github.com/absurdcoder/mini_llm_project/blob/main/utils/validation.py"

    %% Styles
    classDef compute fill:#c8e6c9,stroke:#333
    classDef data fill:#bbdefb,stroke:#333
    classDef external fill:#ffe0b2,stroke:#333
    classDef util fill:#eeeeee,stroke:#333
    class CLI1,CLI2,DL1,DL2,DL3,TK1,TK2,TK3,TK4,M0,M1,M2,M3,M4,M5,T1,T2,T3,T4,TE1,TE2,TE3,UI1,UI2,UI3 compute
    class P1,P2,P3,P4,P5 data
    class TK5,EX1,EX2 external
    class U1,U2 util
```
