# zyzhangustc/mellm architecture

> Architecture diagram of the GitHub repository zyzhangustc/mellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zyzhangustc/mellm
- Repository: https://github.com/zyzhangustc/mellm

## Overview

MELLM is a micro-expression understanding demo: a user supplies onset and apex face frames, the pipeline loads MEFlowNet, derives a flow-guided prompt, and a locally loaded causal language model generates an interpretation. The diagram preserves the flow/ROI and visualization utilities, the flow network’s feature backbones, and the Depth Anything/DINOv2 internals. Training is not included in this checkout. Several utility relationships are supported by the source index; unsampled behavior is not expanded into speculative links.

Main entry points: [`MELLM_pipeline/pipeline.py`](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/pipeline.py) (Demo pipeline), [`MELLM_pipeline/model/meflownet.py`](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/meflownet.py) (MEFlowNet), [`MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py`](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py) (DPT depth head), [`MELLM_pipeline/flow_feature.py`](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/flow_feature.py) (Flow prompt)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Inference workflow"]
  node_pipeline["Demo pipeline<br/>[pipeline.py]"]
  node_config["Model config<br/>[parser.py]"]
  node_frames["Onset/apex frames"]
  node_inference["Inference wrapper<br/>[inference_tools.py]"]
  node_llm["Local language model"]
  node_interpretation["Generated interpretation"]
end

subgraph group_motion["Motion processing"]
  node_flow_prompt["Flow prompt<br/>[flow_feature.py]"]
  node_roi["ROI combination<br/>[roi_combine.py]"]
  node_flow_viz["Flow visualization<br/>[flow_vis.py]"]
end

subgraph group_network["Flow network"]
  node_mellm["MEFlowNet<br/>[meflownet.py]"]
  node_vit["Vision transformer<br/>[vit.py]"]
  node_patch_embed["Patch embedding<br/>[patch_embed.py]"]
  node_model_utils["Model utilities<br/>[utils.py]"]
end

subgraph group_depth["Depth backbone"]
  node_depth_feature["Depth feature adapter<br/>[depthanythingv2.py]"]
  node_dpt["DPT depth head<br/>[dpt.py]"]
  node_dinov2["DINOv2 encoder<br/>[dinov2.py]"]
  node_dino_layers["DINOv2 layers"]
  node_dpt_blocks["DPT fusion blocks<br/>[blocks.py]"]
  node_dpt_transform["Depth transforms<br/>[transform.py]"]
end

node_user(("User"))

node_user -->|"runs demo"| node_pipeline
node_user -->|"provides frames"| node_frames
node_pipeline -->|"reads config"| node_config
node_pipeline -->|"loads frames"| node_frames
node_pipeline -->|"constructs model"| node_mellm
node_pipeline -->|"loads checkpoint"| node_model_utils
node_pipeline -->|"wraps model"| node_inference
node_pipeline -->|"requests prompt"| node_flow_prompt
node_flow_prompt -->|"uses ROI"| node_roi
node_flow_prompt -->|"uses visualization"| node_flow_viz
node_pipeline -->|"generates text"| node_llm
node_llm -->|"returns text"| node_interpretation
node_mellm -->|"extracts depth features"| node_depth_feature
node_mellm -->|"uses transformer"| node_vit
node_mellm -->|"uses utilities"| node_model_utils
node_vit -->|"embeds patches"| node_patch_embed
node_depth_feature -->|"uses depth model"| node_dpt
node_dpt -->|"uses encoder"| node_dinov2
node_dpt -->|"uses fusion blocks"| node_dpt_blocks
node_dpt -->|"uses transforms"| node_dpt_transform
node_dinov2 -->|"uses layer modules"| node_dino_layers

click node_pipeline "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/pipeline.py"
click node_config "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/config/parser.py"
click node_flow_prompt "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/flow_feature.py"
click node_roi "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/roi_combine.py"
click node_flow_viz "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/flow_vis.py"
click node_inference "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/inference_tools.py"
click node_mellm "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/meflownet.py"
click node_vit "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/vit.py"
click node_patch_embed "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/patch_embed.py"
click node_depth_feature "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/depthanythingv2.py"
click node_dpt "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py"
click node_dinov2 "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py"
click node_dino_layers "https://github.com/zyzhangustc/mellm/tree/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2_layers"
click node_dpt_blocks "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/blocks.py"
click node_dpt_transform "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/transform.py"
click node_model_utils "https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/utils/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_config,node_frames,node_inference,node_llm,node_interpretation,node_user toneBlue
class node_flow_prompt,node_roi,node_flow_viz toneAmber
class node_mellm,node_vit,node_patch_embed,node_model_utils toneMint
class node_depth_feature,node_dpt,node_dinov2,node_dino_layers,node_dpt_blocks,node_dpt_transform toneRose
```

## Components

### Inference workflow

- [Demo pipeline](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/pipeline.py): `MELLM_pipeline/pipeline.py`
- [Model config](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/config/parser.py): `MELLM_pipeline/config/parser.py`
- **Onset/apex frames**
- [Inference wrapper](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/inference_tools.py): `MELLM_pipeline/inference_tools.py`
- **Local language model**
- **Generated interpretation**

### Motion processing

- [Flow prompt](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/flow_feature.py): `MELLM_pipeline/flow_feature.py`
- [ROI combination](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/roi_combine.py): `MELLM_pipeline/roi_combine.py`
- [Flow visualization](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/flow_vis.py): `MELLM_pipeline/flow_vis.py`

### Flow network

- [MEFlowNet](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/meflownet.py): `MELLM_pipeline/model/meflownet.py`
- [Vision transformer](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/vit.py): `MELLM_pipeline/model/backbone/vit.py`
- [Patch embedding](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/patch_embed.py): `MELLM_pipeline/model/backbone/patch_embed.py`
- [Model utilities](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/utils/utils.py): `MELLM_pipeline/utils/utils.py`

### Depth backbone

- [Depth feature adapter](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/model/backbone/depthanythingv2.py): `MELLM_pipeline/model/backbone/depthanythingv2.py`
- [DPT depth head](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py): `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py`
- [DINOv2 encoder](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py): `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py`
- [DINOv2 layers](https://github.com/zyzhangustc/mellm/tree/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2_layers): `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2_layers`
- [DPT fusion blocks](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/blocks.py): `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/blocks.py`
- [Depth transforms](https://github.com/zyzhangustc/mellm/blob/main/MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/transform.py): `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/transform.py`

### Other components

- **User**

## Connections

- User → Demo pipeline: runs demo (evidence: `readme.md`)
- User → Onset/apex frames: provides frames (evidence: `readme.md`)
- Demo pipeline → Model config: reads config (evidence: `MELLM_pipeline/pipeline.py`)
- Demo pipeline → Onset/apex frames: loads frames (evidence: `MELLM_pipeline/pipeline.py`)
- Demo pipeline → MEFlowNet: constructs model (evidence: `MELLM_pipeline/pipeline.py`)
- Demo pipeline → Model utilities: loads checkpoint (evidence: `MELLM_pipeline/pipeline.py`)
- Demo pipeline → Inference wrapper: wraps model (evidence: `MELLM_pipeline/pipeline.py`)
- Demo pipeline → Flow prompt: requests prompt (evidence: `MELLM_pipeline/pipeline.py`)
- Flow prompt → ROI combination: uses ROI (evidence: `MELLM_pipeline/flow_feature.py`)
- Flow prompt → Flow visualization: uses visualization (evidence: `MELLM_pipeline/flow_feature.py`)
- Demo pipeline → Local language model: generates text (evidence: `MELLM_pipeline/pipeline.py`)
- Local language model → Generated interpretation: returns text
- MEFlowNet → Depth feature adapter: extracts depth features (evidence: `MELLM_pipeline/model/meflownet.py`)
- MEFlowNet → Vision transformer: uses transformer (evidence: `MELLM_pipeline/model/meflownet.py`)
- MEFlowNet → Model utilities: uses utilities (evidence: `MELLM_pipeline/model/meflownet.py`)
- Vision transformer → Patch embedding: embeds patches (evidence: `MELLM_pipeline/model/backbone/vit.py`)
- Depth feature adapter → DPT depth head: uses depth model (evidence: `MELLM_pipeline/model/backbone/depthanythingv2.py`)
- DPT depth head → DINOv2 encoder: uses encoder (evidence: `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py`)
- DPT depth head → DPT fusion blocks: uses fusion blocks (evidence: `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py`)
- DPT depth head → Depth transforms: uses transforms (evidence: `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py`)
- DINOv2 encoder → DINOv2 layers: uses layer modules (evidence: `MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py`)
