How yanzhaoshi/mepnet works
Project Type and Purpose
• This is a multi-modal machine-learning pipeline for brain CT report generation (“MEPNet”), built on top of the open-source Llama-Recipes framework.
• It ingests CT scan data, extracts visual features with a frozen ResNet101 encoder, builds medical-entity knowledge graphs, and fine-tunes a LLaMA-3 8B LLM via customized multi-modal prompts to generate radiology reports.
A. Data Layer
• CTRG_dataset / BCT-CHR dataset (raw CT scans, labels, entity lists)
• data_processing/build_graph_data.py (prepares entity graphs & JSON files)
B. Feature Extraction
• Precomputed ResNet101 visual features (frozen encoder)
• Optionally CLIP or BioMedCLIP feature sources
C. Knowledge & Scoring Modules
• Knowledge-Driven Joint Attention (distills explicit/implicit medical entity patterns)
• Learning Status Scorer (evaluates per-entity learning progress)
D. Prompt Construction & Model Interface
• Multi-modal Prompt Generator (combines visual embeddings + status scores)
• llama_recipes/src/llama_recipes/inference (chat_utils, prompt_format_utils)
E. Training Framework
• llama_recipes/recipes/finetuning/finetuning_work4.py
• Configuration layer: llama_recipes/src/llama_recipes/configs/training.py
• Underlying trainers & data loaders (HuggingFace Trainer, PEFT, FSDP, etc.)
F. Evaluation & Inference
• recipes/testing/testing_work4.py (automated metrics)
• recipes/inference (live inference scripts or notebooks, Gradio UI)
G. Checkpointing & Storage
• model_checkpointing/checkpoint_handler.py
• local filesystem or external model store
H. External Dependencies
• Llama-Recipes library (installed via pip -e .)
• Pretrained LLaMA weights, vision encoder weights
• Optional cloud APIs (Azure, OctoAI) for benchmarks
1. Raw CT Data → data_processing → Entity Graphs & JSON
2. Raw CT Data → ResNet101 Encoder → Visual Embeddings (stored offline)
3. Visual Embeddings + Entity Graphs → Knowledge-Driven Joint Attention → Entity Embedding Vectors
4. Entity Embeddings → Learning Status Scorer → Entity Status Scores
5. Entity Embeddings + Status Scores → Prompt Generator → Multi-modal Prompt
6. Multi-modal Prompt → LLaMA3 Fine-tuner → Trained MEPNet Weights → Checkpoint
7. Checkpoint + Prompt → Inference Engine → Generated CT Report
8. Generated Report + Ground Truth → Evaluation Scripts → Metrics (BLEU, CIDER, clinical accuracy)
Architectural Patterns & Design Principles
• Layered pipeline: Data → Processing → Feature Extraction → Modeling → Evaluation
• Modular recipe-based design (plug in new datasets, encoders, scoring modules)
• Separation of concerns: configurations, data, models, inference utilities live in distinct directories
• Reuse of Llama-Recipes core for training loop, trainers, checkpointing