How pwang322/dlf works
Project Type and Purpose
• This is a Python-based research codebase implementing “Disentangled-Language-Focused” multimodal sentiment analysis (MSA).
• It provides a full training and evaluation pipeline for MOSI/MOSEI datasets using PyTorch, CUDA, and custom multimodal encoders, disentanglement, enhancement, fusion, and prediction modules.
File Structure
• Top‐Level Scripts and Config:
• run.py, train.py, test.py orchestrate the pipeline.
• config.py & config/config.json store global settings.
• data_loader.py handles dataset ingestion.
• Model & Training Modules (trains/):
• trains/singleTask: core DLF model classes (DLF.py), loss functions (HingeLoss.py), distillation kernels, utilities.
• trains/subNets: per-modality encoders (BertTextEncoder.py, AlignNets.py) and a Transformers encoder (multihead_attention, transformer, position_embedding).
• trains/utils: training loops, metric calculators.
• Utilities: global helpers under utils/ and trains/utils/.
• Artifacts:
• pt/: saved PyTorch checkpoints.
• result/: CSV outputs (MOSI, MOSEI).
• log/: training logs.
• imgs/: documentation images.
README Insights
• Five‐stage pipeline: Feature Extraction → Disentanglement → Enhancement → Fusion → Prediction.
• Language-Focused Attractor (LFA) is a specialized cross-attention module.
• Hierarchical predictions combine pre-fused and post-fused outputs.
Main Components
a. External Data Sources
• MOSI/MOSEI datasets (pkl files under dataset/ referenced by config/config.json).
b. Data Ingestion & Preprocessing
• data_loader.py reads raw features, applies any normalization.
c. Modality-Specific Encoders (Feature Extraction)
• Text: BertTextEncoder
• Audio/Visual (if present): custom AlignNets or other subNets
• Shared: Transformer encoder components in trains/subNets/transformers_encoder
d. Disentanglement Module
• Distillation kernels (get_distillation_kernel.py & homo variant) under trains/singleTask/distillnets
e. Language-Focused Attractor (Enhancement)
• Cross-attention module that refines features in LFA block (within DLF.py)
f. Fusion & Prediction
• Fusion mechanism inside DLF.py combining modalities
• Hierarchical prediction heads (pre-fused and post-fused classifiers)
g. Training Orchestration
• train.py invokes training loops in trains/utils/functions.py and metricsTop.py
• HingeLoss and other loss modules
h. Model Persistence & Logging
• Saving checkpoints to pt/
• Writing logs to log/
i. Evaluation & Results
• test.py reads checkpoints, runs evaluation, outputs CSVs to result/
Relationships & Data Flow
• Flow: Config → Data Loader → Encoders → Disentanglement → LFA → Fusion → Predictors → Loss → Backprop → Checkpoint
• Evaluation: Checkpoint → Model load → Same forward path → Metrics → CSV
Architectural Patterns & Technologies
• Pattern: Pipeline/Multi‐stage data flow; modular “singleTask” & “subNets” packages
• Frameworks: Python 3.9, PyTorch 1.13 + CUDA 11.7
• Design Principles: Separation of concerns (data, config, model, utils), modular encoder/decoder blocks, plug-and-play distillation kernels
• Use clear labels for every component (e.g., “Text Encoder (BertTextEncoder)”).
• Blue for data artifacts (datasets, checkpoints, CSVs)
• Green for compute modules (encoders, LFA, fusion)
• Orange for orchestration scripts (train.py, test.py)
• Gray for utilities (config, logging, metrics)
• external dependencies (PyTorch, CUDA) at the top or side.