How zyzhangustc/mellm works
MELLM is a micro-expression understanding demo: a user supplies onset and apex face frames, the pipeline loads MEFlowNet, derives a flow-guided prompt, and a locally loaded causal language model generates an interpretation. The diagram preserves the flow/ROI and visualization utilities, the flow network’s feature backbones, and the Depth Anything/DINOv2 internals. Training is not included in this checkout. Several utility relationships are supported by the source index; unsampled behavior is not expanded into speculative links.
Main entry points: pipeline.py Demo pipeline · meflownet.py MEFlowNet · dpt.py DPT depth head · flow_feature.py Flow prompt
Components 20
Inference workflow
- Demo pipeline
MELLM_pipeline/pipeline.py - Model config
MELLM_pipeline/config/parser.py - Onset/apex frames
- Inference wrapper
MELLM_pipeline/inference_tools.py - Local language model
- Generated interpretation
Motion processing
- Flow prompt
MELLM_pipeline/flow_feature.py - ROI combination
MELLM_pipeline/roi_combine.py - Flow visualization
MELLM_pipeline/flow_vis.py
Flow network
- MEFlowNet
MELLM_pipeline/model/meflownet.py - Vision transformer
MELLM_pipeline/model/backbone/vit.py - Patch embedding
MELLM_pipeline/model/backbone/patch_embed.py - Model utilities
MELLM_pipeline/utils/utils.py
Depth backbone
- Depth feature adapter
MELLM_pipeline/model/backbone/depthanythingv2.py - DPT depth head
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py - DINOv2 encoder
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py - DINOv2 layers
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2_layers - DPT fusion blocks
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/blocks.py - Depth transforms
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/util/transform.py
Other components
- User
Connections 21
- User → Demo pipeline: runs demo
readme.md - User → Onset/apex frames: provides frames
readme.md - Demo pipeline → Model config: reads config
MELLM_pipeline/pipeline.py - Demo pipeline → Onset/apex frames: loads frames
MELLM_pipeline/pipeline.py - Demo pipeline → MEFlowNet: constructs model
MELLM_pipeline/pipeline.py - Demo pipeline → Model utilities: loads checkpoint
MELLM_pipeline/pipeline.py - Demo pipeline → Inference wrapper: wraps model
MELLM_pipeline/pipeline.py - Demo pipeline → Flow prompt: requests prompt
MELLM_pipeline/pipeline.py - Flow prompt → ROI combination: uses ROI
MELLM_pipeline/flow_feature.py - Flow prompt → Flow visualization: uses visualization
MELLM_pipeline/flow_feature.py - Demo pipeline → Local language model: generates text
MELLM_pipeline/pipeline.py - Local language model → Generated interpretation: returns text
- MEFlowNet → Depth feature adapter: extracts depth features
MELLM_pipeline/model/meflownet.py - MEFlowNet → Vision transformer: uses transformer
MELLM_pipeline/model/meflownet.py - MEFlowNet → Model utilities: uses utilities
MELLM_pipeline/model/meflownet.py - Vision transformer → Patch embedding: embeds patches
MELLM_pipeline/model/backbone/vit.py - Depth feature adapter → DPT depth head: uses depth model
MELLM_pipeline/model/backbone/depthanythingv2.py - DPT depth head → DINOv2 encoder: uses encoder
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py - DPT depth head → DPT fusion blocks: uses fusion blocks
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py - DPT depth head → Depth transforms: uses transforms
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dpt.py - DINOv2 encoder → DINOv2 layers: uses layer modules
MELLM_pipeline/thirdparty/DepthAnythingV2/depth_anything_v2/dinov2.py