How ibrahimethemhamamci/ct-chat works
Project Type and Purpose
• CT-CHAT is a full-stack research codebase implementing a visual-language chat model for 3D chest CT volumes.
• It provides data-generation scripts (VQA), a training pipeline leveraging CT-CLIP encoder + LLM backbones (Llama/Mistral/MPT), evaluation tools, and a multi-component serve layer (CLI, controller, Gradio web UI, worker processes).
High-Level Components
a. Data Generation & Preprocessing
• VQA_dataset/… scripts – generate questions, free-response descriptions, retrieve PubMed text.
• CT-CLIP encoder (external dependency, invoked via llava/serve/encode_script.py) – converts CT slices/volumes into embeddings.
b. Model Building & Training
• llava/model – defines multimodal encoder, projector, and ties into language_model submodules (llava_llama, llava_mistral, llava_mpt).
• llava/model/apply_delta.py, make_delta.py, consolidate.py – handle LoRA/Delta weight injection and merging.
• llava/train/* – training scripts (train.py, train_mem.py), monkey patches for optimized attention, metadata files.
• Shell wrappers (pretrain_*.sh, finetune_*.sh) – simplify launching large-scale GPU jobs.
c. Evaluation
• evaluations/evaluate_llm.py, new_llm_metrics.py – calculate VQA accuracy, other LLM metrics.
d. Serving & Inference
• llava/serve/controller.py – central RPC/HTTP controller orchestrating workers.
• llava/serve/model_worker.py, sglang_worker.py – run inference processes loaded with model weights.
• llava/serve/gradio_web_server.py (and niigz variants) – GUI endpoint.
• CLI scripts (ctchat_validation_*.py) – batch/in-terminal inference.
• Helper scripts: register_worker.py, send_eval.sh, send_validation.sh.
e. Auxiliary
• zero3.json – ZeRO-3 config for distributed training.
• figures/, video/ – demo assets for documentation.
Relationships & Data Flows
1. Raw CT volume + metadata → VQA_dataset scripts → JSON QA pairs
2. QA pairs + CT volumes → encode_script.py (CT-CLIP) → latent embeddings
3. Embeddings + text prompts → llava/train pipeline → trained multimodal LLM weights
4. Trained weights → llava/model/apply_delta & consolidate → final model artifact
5. Model artifact deployed to llava/serve workers → registered with controller
6. User interaction via CLI or Gradio UI → controller routes request → worker loads model → returns answer
Architectural Patterns & Principles
• Modular “builder” pattern in llava/model for plugging in different encoders or LLM backbones.
• Delta-weight pattern (LoRA) for parameter-efficient fine-tuning.
• Microservice-style serving: separation of controller vs. worker processes, pluggable via HTTP/RPC.
• Pipeline stages: data prep → training → consolidation → serving → evaluation.