# yanzhaoshi/mepnet architecture

> Architecture diagram of the GitHub repository yanzhaoshi/mepnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yanzhaoshi/mepnet
- Repository: https://github.com/yanzhaoshi/mepnet

## Overview

##### Project Type and Purpose
- This is a multi-modal machine-learning pipeline for brain CT report generation (“MEPNet”), built on top of the open-source Llama-Recipes framework.
- It ingests CT scan data, extracts visual features with a frozen ResNet101 encoder, builds medical-entity knowledge graphs, and fine-tunes a LLaMA-3 8B LLM via customized multi-modal prompts to generate radiology reports.

A. Data Layer
- CTRG_dataset / BCT-CHR dataset (raw CT scans, labels, entity lists)
- data_processing/build_graph_data.py (prepares entity graphs & JSON files)
B. Feature Extraction
- Precomputed ResNet101 visual features (frozen encoder)
- Optionally CLIP or BioMedCLIP feature sources
C. Knowledge & Scoring Modules
- Knowledge-Driven Joint Attention (distills explicit/implicit medical entity patterns)
- Learning Status Scorer (evaluates per-entity learning progress)
D. Prompt Construction & Model Interface
- Multi-modal Prompt Generator (combines visual embeddings + status scores)
- llama_recipes/src/llama_recipes/inference (chat_utils, prompt_format_utils)
E. Training Framework
- llama_recipes/recipes/finetuning/finetuning_work4.py
- Configuration layer: llama_recipes/src/llama_recipes/configs/training.py
- Underlying trainers & data loaders (HuggingFace Trainer, PEFT, FSDP, etc.)
F. Evaluation & Inference
- recipes/testing/testing_work4.py (automated metrics)
- recipes/inference (live inference scripts or notebooks, Gradio UI)
G. Checkpointing & Storage
- model_checkpointing/checkpoint_handler.py
- local filesystem or external model store
H. External Dependencies
- Llama-Recipes library (installed via pip -e .)
- Pretrained LLaMA weights, vision encoder weights
- Optional cloud APIs (Azure, OctoAI) for benchmarks

1. Raw CT Data → data_processing → Entity Graphs & JSON
2. Raw CT Data → ResNet101 Encoder → Visual Embeddings (stored offline)
3. Visual Embeddings + Entity Graphs → Knowledge-Driven Joint Attention → Entity Embedding Vectors
4. Entity Embeddings → Learning Status Scorer → Entity Status Scores
5. Entity Embeddings + Status Scores → Prompt Generator → Multi-modal Prompt
6. Multi-modal Prompt → LLaMA3 Fine-tuner → Trained MEPNet Weights → Checkpoint
7. Checkpoint + Prompt → Inference Engine → Generated CT Report
8. Generated Report + Ground Truth → Evaluation Scripts → Metrics (BLEU, CIDER, clinical accuracy)

##### Architectural Patterns & Design Principles
- Layered pipeline: Data → Processing → Feature Extraction → Modeling → Evaluation
- Modular recipe-based design (plug in new datasets, encoders, scoring modules)
- Separation of concerns: configurations, data, models, inference utilities live in distinct directories
- Reuse of Llama-Recipes core for training loop, trainers, checkpointing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer (Blue)
    subgraph "Data Layer"
        direction TB
        RawCT["Raw CT Scans & Labels"]:::data
        CTRG["CTRG_dataset"]:::data
        BuildGraph["build_graph_data.py"]:::process
    end

    %% Preprocessing & Feature Extraction (Green)
    subgraph "Preprocessing & Feature Extraction"
        direction TB
        ResNetModel["ResNet101 Encoder"]:::process
        PrecomputedFeatures["Precomputed Visual Embeddings"]:::process
        ResNet_LLM_Wrap["ResNet+LLM Wrapper"]:::process
    end

    %% Knowledge & Scoring (Orange)
    subgraph "Knowledge & Scoring Modules"
        direction TB
        JointAtt["Knowledge-Driven Joint Attention"]:::model
        StatusScorer["Learning Status Scorer"]:::model
    end

    %% Prompt & Interface (Orange)
    subgraph "Prompt Construction & Model Interface"
        direction TB
        ChatUtils["chat_utils.py"]:::model
        PromptFmt["prompt_format_utils.py"]:::model
        LLMInvoke["llm.py"]:::model
    end

    %% Training Framework (Purple)
    subgraph "Training Framework"
        direction TB
        Config["training.py"]:::train
        DataLoader["work4_ctrg_dataset.py"]:::train
        FinetuneRecipe["finetuning_work4.py"]:::train
        HFTrainer["HuggingFace Trainer & PEFT"]:::train
    end

    %% Evaluation & Inference (Purple)
    subgraph "Evaluation & Inference"
        direction TB
        Testing["testing_work4.py"]:::train
        LocalInfer["inference.py"]:::train
        ChatExample["chat_completion.py"]:::train
    end

    %% Checkpointing & External (Gray)
    subgraph "Storage & External"
        direction TB
        CheckpointHandler["checkpoint_handler.py"]:::external
        ModelStore["Model Store / Filesystem"]:::external
        LlamaRecipes["llama-recipes (core library)"]:::external
        PretrainedWeights["Pretrained LLaMA & Vision Weights"]:::external
    end

    %% Data Flows
    RawCT -->|"raw scans"| BuildGraph
    BuildGraph -->|"JSON Entity Graphs"| JointAtt

    RawCT -->|"raw scans"| ResNetModel
    ResNetModel -->|"torch.Tensor"| PrecomputedFeatures

    PrecomputedFeatures --> JointAtt
    JointAtt -->|"entity embeddings"| StatusScorer
    StatusScorer -->|"entity status scores"| ChatUtils

    ChatUtils --> PromptFmt
    PromptFmt --> LLMInvoke
    LLMInvoke -->|"multi-modal prompt"| FinetuneRecipe

    Config -.-> FinetuneRecipe
    DataLoader --> FinetuneRecipe
    FinetuneRecipe --> HFTrainer
    HFTrainer --> CheckpointHandler
    CheckpointHandler --> ModelStore

    ModelStore --> LocalInfer
    LLMInvoke --> LocalInfer
    LocalInfer --> ChatExample
    LocalInfer --> Testing

    ChatExample --> Testing
    Testing -->|"metrics"| HFTrainer

    LlamaRecipes -.-> FinetuneRecipe
    PretrainedWeights --> ResNetModel
    PretrainedWeights --> LLMInvoke

    %% Click Events
    click CTRG "https://github.com/yanzhaoshi/mepnet/tree/main/CTRG_dataset/"
    click BuildGraph "https://github.com/yanzhaoshi/mepnet/blob/main/data_processing/build_graph_data.py"
    click ResNetModel "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/models/ResNet_MLP_llama/modeling_ResNet_MLP_llama.py"
    click ResNet_LLM_Wrap "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/finetuning_ResNet_MLP_llama.py"
    click JointAtt "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/models/work4_llama/modeling_work4_llama.py"
    click StatusScorer "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/utils/work4_train_utils.py"
    click ChatUtils "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/inference/chat_utils.py"
    click PromptFmt "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/inference/prompt_format_utils.py"
    click LLMInvoke "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/inference/llm.py"
    click FinetuneRecipe "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/recipes/finetuning/finetuning_work4.py"
    click Config "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/configs/training.py"
    click DataLoader "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/datasets/work4_ctrg_dataset.py"
    click Testing "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/recipes/testing/testing_work4.py"
    click LocalInfer "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/recipes/inference/local_inference/inference.py"
    click ChatExample "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/recipes/inference/local_inference/chat_completion/chat_completion.py"
    click CheckpointHandler "https://github.com/yanzhaoshi/mepnet/blob/main/llama-recipes/src/llama_recipes/model_checkpointing/checkpoint_handler.py"
    click LlamaRecipes "https://github.com/yanzhaoshi/mepnet/tree/main/llama-recipes/"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,color:#1B2631
    classDef process fill:#ABEBC6,stroke:#1D8348,color:#145A32
    classDef model fill:#F9E79F,stroke:#B7950B,color:#784212
    classDef train fill:#D2B4DE,stroke:#6C3483,color:#4A235A
    classDef external fill:#D5DBDB,stroke:#566573,color:#2C3E50
```
