How zichenwen1/epic works
Step 1: Identify Project Type and Purpose
• This is a full-stack research and engineering toolkit for multi-modal large language models (MLLMs), implementing the EPIC framework: Progressive Consistency Distillation.
• Core use cases:
• Training/compressing multi-modal LLMs (token- and layer-wise compression).
• Evaluating on vision–language tasks and benchmarks.
• Serving models via CLI or Gradio for inference.
Step 2: Analyze the File Structure
• Top-level directories:
• llava/
– model/ (model definition, pruning-method factories, encoders, projectors)
– train/ (training scripts, KD and TCD entry points)
– eval/ (evaluation scripts + web interface assets)
– serve/ (CLI, controller, Gradio server, worker processes)
– utils, constants, conversation handlers
• scripts/ (shell scripts for pretrain, finetune, eval orchestration, DeepSpeed configs)
• assets/, docs/ (illustrations, markdown guides)
• requirements.txt, pyproject.toml (dependency manifests)
• Patterns:
• Factory & builder patterns for pruning methods and trainers.
• Modular separation: data ingestion (external JSON/images), model construction, training loop, evaluation, serving.
Step 3: Key Insights from README
• Architecture: base LLM + vision tower + multi-modal encoder + projector; progressive token compression via “DART”, “FastV”, “Random” strategies.
• Training configurations managed by shell scripts invoking Python trainers with DeepSpeed.
• Evaluation outputs both console metrics and interactive HTML dashboards.
• Serving supports CLI and Gradio interfaces with a controller dispatching to worker processes.
1. Data Layer
• External datasets (COCO, GQA, TextVQA, VisualGenome) stored on local FS.
• Annotation JSON files under playground/data.
2. Model Construction
• Vision Tower (e.g., CLIP)
• Language Backbone (LLaMA/Mistral/MPT)
• Multimodal Encoder + Projector
• Pruning Method Modules (“dart”, “fastv”, “random”), assembled by factory.
3. Training Pipeline
• Entry scripts (train.py, train_TCD.py, memory-efficient KD scripts)
• Progressive consistency distillation logic
• DeepSpeed engine
• Checkpoint and logs sink
4. Evaluation Pipeline
• eval scripts (eval_*.py)
• M4C evaluator, science QA, VQA
• Web page generation (HTML/CSS/JS)
5. Serving Layer
• CLI & controller
• Gradio Web Server
• Worker processes (model_worker.py, sglang_worker.py)
6. Artifacts & Outputs
• Model checkpoints, evaluation dashboards, inference logs.
Step 5: Relationships & Interactions
• Data Layer → Training Pipeline → Model Construction → DeepSpeed → Checkpoint Storage
• Checkpoint Storage → Evaluation Pipeline → Results Dashboard
• Checkpoint Storage → Serving Layer → Inference Requests → Responses
• Config files / scripts drive parameterization of all pipelines.
Step 6: Architectural Patterns & Technologies
• Patterns:
• Factory (pruning/trainer factories)
• Builder (model, encoder, projector builders)
• Modular pipelines for train, eval, serve
• Technologies:
• Python, PyTorch, DeepSpeed (ZeRO-3), FlashAttention
• HuggingFace Transformers
• PEFT (LoRA), TensorBoard
• Gradio for UI
• Shell scripts for orchestration
Step 7: Tailoring Guidelines
• Full-Stack Application Emphasis:
• Clear separation between Data, Compute (training & eval), Serving.
• File system and external dataset boundaries.
• API layers: CLI vs Gradio.
• Extensibility Points:
• Where new pruning methods plug in (factory modules).
• Where new evaluation benchmarks slot in.
• For Multi-Modal Nature:
• Distinguish Vision Tower vs Language Backbone vs Multimodal Encoder.
Step 8: Diagramming Conventions
• Rectangles for components/modules
• Parallelograms for scripts/orchestration layers
• Clouds for external services
• Blue: Data
• Green: Model Construction
• Orange: Training & Distillation
• Purple: Evaluation
• Teal: Serving & Inference
• Left→Right or Top→Bottom data/control flow
• Annotate:
• Config scripts (e.g., finetune_TCD.sh) alongside the relevant pipeline.
• Factory entry points (factory.py, builder.py) where pruning strategies plug in.
Step 9: Final Advice