# zichenwen1/epic architecture

> Architecture diagram of the GitHub repository zichenwen1/epic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zichenwen1/epic
- Repository: https://github.com/zichenwen1/epic

## Overview

Step 1: Identify Project Type and Purpose
- This is a full-stack research and engineering toolkit for multi-modal large language models (MLLMs), implementing the EPIC framework: Progressive Consistency Distillation.
- Core use cases:
- Training/compressing multi-modal LLMs (token- and layer-wise compression).
- Evaluating on vision–language tasks and benchmarks.
- Serving models via CLI or Gradio for inference.

Step 2: Analyze the File Structure
- Top-level directories:
- llava/
– model/ (model definition, pruning-method factories, encoders, projectors)
– train/ (training scripts, KD and TCD entry points)
– eval/ (evaluation scripts + web interface assets)
– serve/ (CLI, controller, Gradio server, worker processes)
– utils, constants, conversation handlers
- scripts/ (shell scripts for pretrain, finetune, eval orchestration, DeepSpeed configs)
- assets/, docs/ (illustrations, markdown guides)
- requirements.txt, pyproject.toml (dependency manifests)
- Patterns:
- Factory & builder patterns for pruning methods and trainers.
- Modular separation: data ingestion (external JSON/images), model construction, training loop, evaluation, serving.

Step 3: Key Insights from README
- Architecture: base LLM + vision tower + multi-modal encoder + projector; progressive token compression via “DART”, “FastV”, “Random” strategies.
- Training configurations managed by shell scripts invoking Python trainers with DeepSpeed.
- Evaluation outputs both console metrics and interactive HTML dashboards.
- Serving supports CLI and Gradio interfaces with a controller dispatching to worker processes.

1. Data Layer
- External datasets (COCO, GQA, TextVQA, VisualGenome) stored on local FS.
- Annotation JSON files under playground/data.
2. Model Construction
- Vision Tower (e.g., CLIP)
- Language Backbone (LLaMA/Mistral/MPT)
- Multimodal Encoder + Projector
- Pruning Method Modules (“dart”, “fastv”, “random”), assembled by factory.
3. Training Pipeline
- Entry scripts (train.py, train_TCD.py, memory-efficient KD scripts)
- Progressive consistency distillation logic
- DeepSpeed engine
- Checkpoint and logs sink
4. Evaluation Pipeline
- eval scripts (eval_*.py)
- M4C evaluator, science QA, VQA
- Web page generation (HTML/CSS/JS)
5. Serving Layer
- CLI & controller
- Gradio Web Server
- Worker processes (model_worker.py, sglang_worker.py)
6. Artifacts & Outputs
- Model checkpoints, evaluation dashboards, inference logs.

Step 5: Relationships & Interactions
- Data Layer → Training Pipeline → Model Construction → DeepSpeed → Checkpoint Storage
- Checkpoint Storage → Evaluation Pipeline → Results Dashboard
- Checkpoint Storage → Serving Layer → Inference Requests → Responses
- Config files / scripts drive parameterization of all pipelines.

Step 6: Architectural Patterns & Technologies
- Patterns:
- Factory (pruning/trainer factories)
- Builder (model, encoder, projector builders)
- Modular pipelines for train, eval, serve
- Technologies:
- Python, PyTorch, DeepSpeed (ZeRO-3), FlashAttention
- HuggingFace Transformers
- PEFT (LoRA), TensorBoard
- Gradio for UI
- Shell scripts for orchestration

Step 7: Tailoring Guidelines
- Full-Stack Application Emphasis:
- Clear separation between Data, Compute (training & eval), Serving.
- File system and external dataset boundaries.
- API layers: CLI vs Gradio.
- Extensibility Points:
- Where new pruning methods plug in (factory modules).
- Where new evaluation benchmarks slot in.
- For Multi-Modal Nature:
- Distinguish Vision Tower vs Language Backbone vs Multimodal Encoder.

Step 8: Diagramming Conventions
- Rectangles for components/modules
- Parallelograms for scripts/orchestration layers
- Clouds for external services
- Blue: Data
- Green: Model Construction
- Orange: Training & Distillation
- Purple: Evaluation
- Teal: Serving & Inference
- Left→Right or Top→Bottom data/control flow
- Annotate:
- Config scripts (e.g., finetune_TCD.sh) alongside the relevant pipeline.
- Factory entry points (factory.py, builder.py) where pruning strategies plug in.

Step 9: Final Advice

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS1[(External Datasets)]:::data
        JSON[(Annotation JSONs)]:::data
    end

    %% Scripts Orchestration
    subgraph "Script Orchestration"
        ORC[/scripts/*.sh/]:::scripts
    end

    %% Model Construction
    subgraph "Model Construction"
        VT["Vision Tower (CLIP Encoder)"]:::model
        click VT "https://github.com/zichenwen1/epic/blob/main/llava/model/multimodal_encoder/clip_encoder.py"
        MEB["Multimodal Encoder Builder"]:::model
        click MEB "https://github.com/zichenwen1/epic/blob/main/llava/model/multimodal_encoder/builder.py"
        MPB["Multimodal Projector Builder"]:::model
        click MPB "https://github.com/zichenwen1/epic/blob/main/llava/model/multimodal_projector/builder.py"
        LB1["LLaMA Backbone"]:::model
        click LB1 "https://github.com/zichenwen1/epic/blob/main/llava/model/language_model/llava_llama.py"
        LB2["Mistral Backbone"]:::model
        click LB2 "https://github.com/zichenwen1/epic/blob/main/llava/model/language_model/llava_mistral.py"
        LB3["MPT Backbone"]:::model
        click LB3 "https://github.com/zichenwen1/epic/blob/main/llava/model/language_model/llava_mpt.py"
        subgraph "Pruning Methods"
            DART["DART"]:::model
            click DART "https://github.com/zichenwen1/epic/tree/main/llava/model/language_model/dart/"
            FASTV["FastV"]:::model
            click FASTV "https://github.com/zichenwen1/epic/tree/main/llava/model/language_model/fastv/"
            RAND["Random"]:::model
            click RAND "https://github.com/zichenwen1/epic/tree/main/llava/model/language_model/random/"
        end
        PF["Pruning Factory & Config"]:::model
        click PF "https://github.com/zichenwen1/epic/blob/main/llava/model/pruning_methods/factory.py"
        click PF "https://github.com/zichenwen1/epic/blob/main/llava/model/pruning_methods/config.py"
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        T1[/train.py/]:::training
        click T1 "https://github.com/zichenwen1/epic/blob/main/llava/train/train.py"
        T2[/train_TCD.py/]:::training
        click T2 "https://github.com/zichenwen1/epic/blob/main/llava/train/train_TCD.py"
        T3[/train_mem_KD_TCD.py/]:::training
        click T3 "https://github.com/zichenwen1/epic/blob/main/llava/train/train_mem_KD_TCD.py"
        BT["Base Trainer"]:::training
        click BT "https://github.com/zichenwen1/epic/blob/main/llava/train/llava_trainer.py"
        KD["KD Trainer Variants"]:::training
        click KD "https://github.com/zichenwen1/epic/blob/main/llava/train/llava_trainer_KD*.py"
        DP1["FlashAttention Patch"]:::training
        click DP1 "https://github.com/zichenwen1/epic/blob/main/llava/train/llama_flash_attn_monkey_patch.py"
        DP2["Xformers Patch"]:::training
        click DP2 "https://github.com/zichenwen1/epic/blob/main/llava/train/llama_xformers_attn_monkey_patch.py"
        DS[(DeepSpeed Engine)]:::training
        CK[(Checkpoints & Logs)]:::storage
    end

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        E1[/eval_*.py/]:::eval
        click E1 "https://github.com/zichenwen1/epic/blob/main/llava/eval/eval_*.py"
        M4C["M4C Evaluator"]:::eval
        click M4C "https://github.com/zichenwen1/epic/blob/main/llava/eval/m4c_evaluator.py"
        WP["Webpage Data Gen"]:::eval
        click WP "https://github.com/zichenwen1/epic/blob/main/llava/eval/generate_webpage_data_from_table.py"
        subgraph "Web Assets"
            HTML["index.html"]:::eval
            click HTML "https://github.com/zichenwen1/epic/blob/main/llava/eval/webpage/index.html"
            CSS["styles.css"]:::eval
            click CSS "https://github.com/zichenwen1/epic/blob/main/llava/eval/webpage/styles.css"
            JS["script.js"]:::eval
            click JS "https://github.com/zichenwen1/epic/blob/main/llava/eval/webpage/script.js"
        end
        RD[(Results Dashboard)]:::storage
    end

    %% Serving Layer
    subgraph "Serving Layer"
        CLI["CLI Interface"]:::serve
        click CLI "https://github.com/zichenwen1/epic/blob/main/llava/serve/cli.py"
        CTRL["Controller"]:::serve
        click CTRL "https://github.com/zichenwen1/epic/blob/main/llava/serve/controller.py"
        GR["Gradio Web Server"]:::serve
        click GR "https://github.com/zichenwen1/epic/blob/main/llava/serve/gradio_web_server.py"
        MW["Model Worker"]:::serve
        click MW "https://github.com/zichenwen1/epic/blob/main/llava/serve/model_worker.py"
        SW["SgLang Worker"]:::serve
        click SW "https://github.com/zichenwen1/epic/blob/main/llava/serve/sglang_worker.py"
        REG["Worker Registration"]:::serve
        click REG "https://github.com/zichenwen1/epic/blob/main/llava/serve/register_worker.py"
        IR[(Inference Requests)]:::serve
    end

    %% Connections
    DS1 -->|"images & annotations"| ORC
    JSON -->|"annotations"| ORC
    ORC -->|"invoke scripts"| T1
    ORC -->|"invoke scripts"| T2
    ORC -->|"invoke scripts"| T3

    T1 -->|"builds model via"| VT
    T1 -->|"builds model via"| MEB
    T1 -->|"builds model via"| MPB
    T1 -->|"builds model via"| LB1
    T1 -->|"builds model via"| LB2
    T1 -->|"builds model via"| LB3
    T1 -->|"selects pruning"| DART
    T1 -->|"selects pruning"| FASTV
    T1 -->|"selects pruning"| RAND
    T1 -->|"factory config"| PF

    T1 -->|"uses trainer"| BT
    T2 -->|"uses trainer"| KD
    T3 -->|"uses trainer"| KD
    BT --> DP1
    BT --> DP2
    KD --> DP1
    KD --> DP2
    BT --> DS
    KD --> DS

    DS -->|"writes"| CK

    CK -->|"load checkpoint"| E1
    E1 --> M4C
    M4C --> WP
    WP --> HTML
    WP --> CSS
    WP --> JS
    HTML --> RD
    CSS --> RD
    JS --> RD

    CK -->|"load checkpoint"| CLI
    CK -->|"load checkpoint"| GR
    CLI --> CTRL
    GR --> CTRL
    CTRL --> MW
    CTRL --> SW
    REG --> CTRL
    IR --> CLI
    IR --> GR

    %% Styles
    classDef data fill:#cfe2f3,stroke:#084298;
    classDef scripts fill:#e2e3e5,stroke:#6c757d;
    classDef model fill:#d1e7dd,stroke:#0f5132;
    classDef training fill:#fce5cd,stroke:#d1491f;
    classDef eval fill:#e2e3f3,stroke:#3730a3;
    classDef serve fill:#d1ecf1,stroke:#0c5460;
    classDef storage fill:#f8f9fa,stroke:#6c757d,stroke-dasharray: 5 5;
```
