# pwang322/dlf architecture

> Architecture diagram of the GitHub repository pwang322/dlf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pwang322/dlf
- Repository: https://github.com/pwang322/dlf

## Overview

##### Project Type and Purpose
- This is a Python-based research codebase implementing “Disentangled-Language-Focused” multimodal sentiment analysis (MSA).
- It provides a full training and evaluation pipeline for MOSI/MOSEI datasets using PyTorch, CUDA, and custom multimodal encoders, disentanglement, enhancement, fusion, and prediction modules.

##### File Structure
- Top‐Level Scripts and Config:
- run.py, train.py, test.py orchestrate the pipeline.
- config.py & config/config.json store global settings.
- data_loader.py handles dataset ingestion.
- Model & Training Modules (trains/):
- trains/singleTask: core DLF model classes (DLF.py), loss functions (HingeLoss.py), distillation kernels, utilities.
- trains/subNets: per-modality encoders (BertTextEncoder.py, AlignNets.py) and a Transformers encoder (multihead_attention, transformer, position_embedding).
- trains/utils: training loops, metric calculators.
- Utilities: global helpers under utils/ and trains/utils/.
- Artifacts:
- pt/: saved PyTorch checkpoints.
- result/: CSV outputs (MOSI, MOSEI).
- log/: training logs.
- imgs/: documentation images.

##### README Insights
- Five‐stage pipeline: Feature Extraction → Disentanglement → Enhancement → Fusion → Prediction.
- Language-Focused Attractor (LFA) is a specialized cross-attention module.
- Hierarchical predictions combine pre-fused and post-fused outputs.

##### Main Components
a. **External Data Sources**
- MOSI/MOSEI datasets (pkl files under dataset/ referenced by config/config.json).
b. **Data Ingestion & Preprocessing**
- data_loader.py reads raw features, applies any normalization.
c. **Modality-Specific Encoders (Feature Extraction)**
- Text: BertTextEncoder
- Audio/Visual (if present): custom AlignNets or other subNets
- Shared: Transformer encoder components in trains/subNets/transformers_encoder
d. **Disentanglement Module**
- Distillation kernels (get_distillation_kernel.py & homo variant) under trains/singleTask/distillnets
e. **Language-Focused Attractor (Enhancement)**
- Cross-attention module that refines features in LFA block (within DLF.py)
f. **Fusion & Prediction**
- Fusion mechanism inside DLF.py combining modalities
- Hierarchical prediction heads (pre-fused and post-fused classifiers)
g. **Training Orchestration**
- train.py invokes training loops in trains/utils/functions.py and metricsTop.py
- HingeLoss and other loss modules
h. **Model Persistence & Logging**
- Saving checkpoints to pt/
- Writing logs to log/
i. **Evaluation & Results**
- test.py reads checkpoints, runs evaluation, outputs CSVs to result/

##### Relationships & Data Flow
- **Flow**: Config → Data Loader → Encoders → Disentanglement → LFA → Fusion → Predictors → Loss → Backprop → Checkpoint
- **Evaluation**: Checkpoint → Model load → Same forward path → Metrics → CSV

##### Architectural Patterns & Technologies
- **Pattern**: Pipeline/Multi‐stage data flow; modular “singleTask” & “subNets” packages
- **Frameworks**: Python 3.9, PyTorch 1.13 + CUDA 11.7
- **Design Principles**: Separation of concerns (data, config, model, utils), modular encoder/decoder blocks, plug-and-play distillation kernels

- Use **clear labels** for every component (e.g., “Text Encoder (BertTextEncoder)”).
- Blue for data artifacts (datasets, checkpoints, CSVs)
- Green for compute modules (encoders, LFA, fusion)
- Orange for orchestration scripts (train.py, test.py)
- Gray for utilities (config, logging, metrics)
- **external dependencies** (PyTorch, CUDA) at the top or side.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Dependencies
    Ext["PyTorch & CUDA"]:::external

    %% Configuration
    ConfigJSON["config/config.json"]:::util
    ConfigPy["config.py"]:::util
    Ext --> ConfigPy
    ConfigPy --> ConfigJSON

    %% Entry Point
    RunScript["run.py"]:::orchestration
    RunScript --> TrainScript
    RunScript --> TestScript

    %% Orchestration Scripts
    TrainScript["train.py"]:::orchestration
    TestScript["test.py"]:::orchestration

    %% Data Ingestion & Preprocessing
    DataLoader["Data Loader\n(data_loader.py)"]:::compute
    TrainScript --> ConfigPy
    TrainScript --> DataLoader
    TestScript --> ConfigPy
    TestScript --> DataLoader

    %% Feature Extraction
    subgraph "Feature Extraction"
        TextEnc["Text Encoder\n(BertTextEncoder)"]:::compute
        AlignNets["Audio/Visual AlignNets"]:::compute
        MultiHead["Multi-Head Attention"]:::compute
        PosEmb["Positional Embedding"]:::compute
        TransBlock["Transformer Block"]:::compute
    end
    DataLoader --> TextEnc
    DataLoader --> AlignNets
    DataLoader --> MultiHead
    DataLoader --> PosEmb
    DataLoader --> TransBlock

    %% Disentanglement
    subgraph "Disentanglement"
        DistillKernel["Distillation Kernel\n(get_distillation_kernel)"]:::compute
        DistillHomo["Homogeneous Kernel\n(get_distillation_kernel_homo)"]:::compute
    end
    TextEnc --> DistillKernel
    AlignNets --> DistillKernel
    MultiHead --> DistillKernel
    PosEmb --> DistillKernel
    TransBlock --> DistillKernel
    DistillKernel --> DistillHomo

    %% Enhancement, Fusion & Prediction
    subgraph "LFA, Fusion & Prediction"
        DLFCore["Core DLF Model\n(DLF.py)"]:::compute
        SecondaryModel["Secondary DLF Definition\n(model/DLF.py)"]:::compute
    end
    DistillHomo --> DLFCore
    DLFCore --> SecondaryModel

    %% Loss & Training Utilities
    Hinge["Hinge Loss\n(HingeLoss.py)"]:::util
    TrainingUtils["Training Functions\n(trains/utils/functions.py)"]:::util
    Metrics["Metrics Calculator\n(metricsTop.py)"]:::util

    DLFCore --> Hinge
    Hinge --> TrainingUtils
    TrainingUtils --> Metrics

    %% Persistence & Logging
    PTDir["Checkpoints\n(pt/)"]:::dataartifact
    LogDir["Logs\n(log/)"]:::dataartifact
    Metrics --> PTDir
    Metrics --> LogDir

    %% Evaluation & Results
    MosiCSV["MOSI Results CSV\n(mosi.csv)"]:::dataartifact
    MoseiCSV["MOSEI Results CSV\n(mosei.csv)"]:::dataartifact
    TestScript --> Metrics
    Metrics --> MosiCSV
    Metrics --> MoseiCSV

    %% Click Events
    click ConfigJSON "https://github.com/pwang322/dlf/blob/main/config/config.json"
    click ConfigPy "https://github.com/pwang322/dlf/blob/main/config.py"
    click RunScript "https://github.com/pwang322/dlf/blob/main/run.py"
    click TrainScript "https://github.com/pwang322/dlf/blob/main/train.py"
    click TestScript "https://github.com/pwang322/dlf/blob/main/test.py"
    click DataLoader "https://github.com/pwang322/dlf/blob/main/data_loader.py"
    click TextEnc "https://github.com/pwang322/dlf/blob/main/trains/subNets/BertTextEncoder.py"
    click AlignNets "https://github.com/pwang322/dlf/blob/main/trains/subNets/AlignNets.py"
    click MultiHead "https://github.com/pwang322/dlf/blob/main/trains/subNets/transformers_encoder/multihead_attention.py"
    click PosEmb "https://github.com/pwang322/dlf/blob/main/trains/subNets/transformers_encoder/position_embedding.py"
    click TransBlock "https://github.com/pwang322/dlf/blob/main/trains/subNets/transformers_encoder/transformer.py"
    click DistillKernel "https://github.com/pwang322/dlf/blob/main/trains/singleTask/distillnets/get_distillation_kernel.py"
    click DistillHomo "https://github.com/pwang322/dlf/blob/main/trains/singleTask/distillnets/get_distillation_kernel_homo.py"
    click DLFCore "https://github.com/pwang322/dlf/blob/main/trains/singleTask/DLF.py"
    click SecondaryModel "https://github.com/pwang322/dlf/blob/main/trains/singleTask/model/DLF.py"
    click Hinge "https://github.com/pwang322/dlf/blob/main/trains/singleTask/HingeLoss.py"
    click TrainingUtils "https://github.com/pwang322/dlf/blob/main/trains/utils/functions.py"
    click Metrics "https://github.com/pwang322/dlf/blob/main/trains/utils/metricsTop.py"
    click PTDir "https://github.com/pwang322/dlf/tree/main/pt/"
    click LogDir "https://github.com/pwang322/dlf/tree/main/log/"
    click MosiCSV "https://github.com/pwang322/dlf/blob/main/result/normal/mosi.csv"
    click MoseiCSV "https://github.com/pwang322/dlf/blob/main/result/normal/mosei.csv"

    %% Styles
    classDef dataartifact fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#98FB98,stroke:#333
    classDef orchestration fill:#FFA500,stroke:#333
    classDef util fill:#D3D3D3,stroke:#333
    classDef external fill:#DDA0DD,stroke:#333
```
