# ibrahimethemhamamci/ct-chat architecture

> Architecture diagram of the GitHub repository ibrahimethemhamamci/ct-chat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibrahimethemhamamci/ct-chat
- Repository: https://github.com/ibrahimethemhamamci/ct-chat

## Overview

##### Project Type and Purpose
- CT-CHAT is a full-stack research codebase implementing a visual-language chat model for 3D chest CT volumes.
- It provides data-generation scripts (VQA), a training pipeline leveraging CT-CLIP encoder + LLM backbones (Llama/Mistral/MPT), evaluation tools, and a multi-component serve layer (CLI, controller, Gradio web UI, worker processes).

##### High-Level Components
a. Data Generation & Preprocessing
- VQA_dataset/… scripts – generate questions, free-response descriptions, retrieve PubMed text.
- CT-CLIP encoder (external dependency, invoked via llava/serve/encode_script.py) – converts CT slices/volumes into embeddings.
b. Model Building & Training
- llava/model – defines multimodal encoder, projector, and ties into language_model submodules (llava_llama, llava_mistral, llava_mpt).
- llava/model/apply_delta.py, make_delta.py, consolidate.py – handle LoRA/Delta weight injection and merging.
- llava/train/* – training scripts (train.py, train_mem.py), monkey patches for optimized attention, metadata files.
- Shell wrappers (pretrain_*.sh, finetune_*.sh) – simplify launching large-scale GPU jobs.
c. Evaluation
- evaluations/evaluate_llm.py, new_llm_metrics.py – calculate VQA accuracy, other LLM metrics.
d. Serving & Inference
- llava/serve/controller.py – central RPC/HTTP controller orchestrating workers.
- llava/serve/model_worker.py, sglang_worker.py – run inference processes loaded with model weights.
- llava/serve/gradio_web_server.py (and niigz variants) – GUI endpoint.
- CLI scripts (ctchat_validation_*.py) – batch/in-terminal inference.
- Helper scripts: register_worker.py, send_eval.sh, send_validation.sh.
e. Auxiliary
- zero3.json – ZeRO-3 config for distributed training.
- figures/, video/ – demo assets for documentation.

##### Relationships & Data Flows
1. Raw CT volume + metadata → VQA_dataset scripts → JSON QA pairs
2. QA pairs + CT volumes → encode_script.py (CT-CLIP) → latent embeddings
3. Embeddings + text prompts → llava/train pipeline → trained multimodal LLM weights
4. Trained weights → llava/model/apply_delta & consolidate → final model artifact
5. Model artifact deployed to llava/serve workers → registered with controller
6. User interaction via CLI or Gradio UI → controller routes request → worker loads model → returns answer

##### Architectural Patterns & Principles
- Modular “builder” pattern in llava/model for plugging in different encoders or LLM backbones.
- Delta-weight pattern (LoRA) for parameter-efficient fine-tuning.
- Microservice-style serving: separation of controller vs. worker processes, pluggable via HTTP/RPC.
- Pipeline stages: data prep → training → consolidation → serving → evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        VQA1["conversation_data_generate.py"]:::script
        VQA2["generate_freeresponse_description.py"]:::script
        VQA3["generate_questions.py"]:::script
        VQA4["retrieve_pubmed.py"]:::script
        CTCLIP["CT-CLIP Encoder"]:::external
        QA["Generated QA Pairs"]:::data
        Emb["Latent Embeddings"]:::data
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        subgraph "Model Building" 
            direction TB
            Builder["builder.py"]:::code
            Utils["utils.py"]:::code
            Arch["llava_arch.py"]:::code
            LlamaLM["llava_llama.py"]:::code
            MistralLM["llava_mistral.py"]:::code
            MptLM["llava_mpt.py"]:::code
        end
        subgraph "Delta Handling"
            direction TB
            Apply["apply_delta.py"]:::code
            MakeDelta["make_delta.py"]:::code
            Consolidate["consolidate.py"]:::code
        end
        subgraph "Training Pipeline"
            direction TB
            TrainCore["train.py"]:::script
            TrainMem["train_mem.py"]:::script
            Trainer["llava_trainer.py"]:::script
            Xformers["train_xformers.py"]:::script
            Patch1["llama_flash_attn_monkey_patch.py"]:::code
            Patch2["llama_xformers_attn_monkey_patch.py"]:::code
            Meta1["train_metadata.csv"]:::data
            Meta2["validation_metadata.csv"]:::data
        end
        subgraph "Shell Wrappers"
            direction TB
            Pre1["pretrain_vicuna.sh"]:::script
            Pre2["pretrain_mistral.sh"]:::script
            Pre3["LLaMA3.1-V_pretrain_ctchat.sh"]:::script
            Fine1["finetune_vicuna.sh"]:::script
            Fine2["finetune_mistral.sh"]:::script
            Fine3["LLaMA3.1-V_finetune_lora_ctchat.sh"]:::script
        end
    end

    %% Model Artifact
    Artifact["LoRA & Consolidated Model"]:::artifact

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        Controller["controller.py"]:::server
        CLIEntry["cli.py"]:::script
        subgraph "Workers"
            direction TB
            ModelW["model_worker.py"]:::server
            SgLangW["sglang_worker.py"]:::server
        end
        subgraph "Web UI"
            direction TB
            Gradio1["gradio_web_server.py"]:::server
            Gradio2["gradio_server_updated.py"]:::server
            Niigz1["nii_gz_gradio.py"]:::server
            WebuiN["webui_niigz.py"]:::server
        end
        subgraph "Validation CLI"
            direction TB
            Val1["ctchat_validation_llama.py"]:::script
            Val2["ctchat_validation_mistral.py"]:::script
            Val3["ctchat_validation_vicuna.py"]:::script
        end
        Register["register_worker.py"]:::script
        SendE["send_eval.sh"]:::script
        SendV["send_validation.sh"]:::script
        TestMsg["test_message.py"]:::script
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        Eval1["evaluate_llm.py"]:::code
        Eval2["new_llm_metrics.py"]:::code
    end

    %% Relationships
    VQA1 --> QA
    VQA2 --> QA
    VQA3 --> QA
    VQA4 --> QA
    QA --> CTCLIP
    CTCLIP --> Emb
    Emb --> TrainCore
    TrainCore --> Artifact
    TrainMem --> Artifact
    Apply --> Artifact
    MakeDelta --> Artifact
    Consolidate --> Artifact
    Artifact --> Controller
    Controller --> ModelW
    Controller --> SgLangW
    Controller --> CLIEntry
    Controller --> Gradio1
    Controller --> Gradio2
    Controller --> Niigz1
    Controller --> WebuiN
    CLIEntry --> ModelW
    CLIEntry --> SgLangW
    Val1 --> Controller
    Val2 --> Controller
    Val3 --> Controller
    Artifact --> Eval1
    Artifact --> Eval2

    %% Click Events
    click VQA1 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/VQA_dataset/conversation_data_generate.py"
    click VQA2 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/VQA_dataset/generate_freeresponse_description.py"
    click VQA3 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/VQA_dataset/generate_questions.py"
    click VQA4 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/VQA_dataset/retrieve_pubmed.py"
    click CTCLIP "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/encode_script.py"
    click Builder "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/builder.py"
    click Utils "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/utils.py"
    click Arch "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/llava_arch.py"
    click Apply "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/apply_delta.py"
    click MakeDelta "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/make_delta.py"
    click Consolidate "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/consolidate.py"
    click LlamaLM "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/language_model/llava_llama.py"
    click MistralLM "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/language_model/llava_mistral.py"
    click MptLM "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/model/language_model/llava_mpt.py"
    click ModelW "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/model_worker.py"
    click SgLangW "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/sglang_worker.py"
    click Controller "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/controller.py"
    click CLIEntry "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/cli.py"
    click Gradio1 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/gradio_web_server.py"
    click Gradio2 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/gradio_server_updated.py"
    click Niigz1 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/nii_gz_gradio.py"
    click WebuiN "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/webui_niigz.py"
    click Val1 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/ctchat_validation_llama.py"
    click Val2 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/ctchat_validation_mistral.py"
    click Val3 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/ctchat_validation_vicuna.py"
    click Register "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/register_worker.py"
    click SendE "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/send_eval.sh"
    click SendV "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/llava/serve/send_validation.sh"
    click Eval1 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/evaluations/evaluate_llm.py"
    click Eval2 "https://github.com/ibrahimethemhamamci/ct-chat/blob/main/evaluations/new_llm_metrics.py"

    %% Styles
    classDef code fill:#cce5ff,stroke:#004085,color:#004085
    classDef script fill:#e2e3e5,stroke:#6c757d,color:#383d41
    classDef data fill:#d4edda,stroke:#155724,color:#155724
    classDef artifact fill:#fff3cd,stroke:#856404,color:#856404
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24
    classDef server fill:#d1ecf1,stroke:#0c5460,color:#0c5460
```
