# haoyu-ha/lnln architecture

> Architecture diagram of the GitHub repository haoyu-ha/lnln, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haoyu-ha/lnln
- Repository: https://github.com/haoyu-ha/lnln

## Overview

##### Project Type and Purpose
- This is a deep-learning research codebase (PyTorch) implementing “Towards Robust Multimodal Sentiment Analysis with Incomplete Data.”
- It’s a single monolithic training/evaluation pipeline (not microservices), configurable via YAML, producing checkpoints and evaluation metrics.

##### File Structure
- Top-level scripts:
- train.sh + train.py → orchestrate training
- robust_evaluation.py → evaluation of saved checkpoints
- configs/ → YAML files for training and eval on three datasets (MOSI, MOSEI, SIMS)
- core/ → reusable modules: dataset, loss functions, metrics, optimizer & scheduler wrappers, utilities
- models/ → network definitions (basic layers, BERT integration, proposed LNLN architecture)
- ckpt/ & log/ → storage for model checkpoints and run logs

##### Extract Key README Insights
- Uses PyTorch 2.2.1, Python 3.11.7, GPU (Tesla A40)
- Data preparation: external MMSA repository
- Distinct training vs. evaluation flows controlled by config files

##### Main Components & Relationships
a. Components
1. Config Loader (parses YAML)
2. Data Ingestion (core/dataset.py + external dataset files)
3. Model Factory (models/basic_layers.py, bert.py, lnln.py)
4. Training Loop (train.py + core/utils for epoch management)
5. Loss & Metric Calculation (core/losses.py, core/metric.py)
6. Optimizer & Scheduler (core/optimizer.py, core/scheduler.py)
7. Checkpoint Manager (saves/loads from ckpt/)
8. Logging (writes to log/)
9. Evaluation Flow (robust_evaluation.py uses same core & models)
10. External Dependencies (PyTorch, Python, GPU, MMSA data)

b. Interactions
- train.sh → invokes train.py with a chosen config
- train.py →
- reads config → instantiates DataLoader → Model → Optimizer/Scheduler
- for each batch: forward → compute loss (core/losses) → backward → optimizer step → scheduler step → compute metrics → log → optionally save checkpoint
- robust_evaluation.py → loads config → loads checkpoint → runs forward passes on test splits → computes & prints metrics

c. Architectural Patterns & Principles
- Modular layering: config → core framework → models → persistence (ckpt/log)
- Separation of concerns: core/ modules abstract away framework boilerplate
- Config-driven: no hard-coded hyperparameters, makes it extensible

d. Key Technologies
- PyTorch 2.2.1 (deep learning)
- YAML (configuration)
- MMSA datasets (Multimodal Sentiment Analysis)
- BERT (pretrained transformer for text modality)

##### Detailed Recommendations
- That configs drive both scripts, and that core/ modules are shared by train.py and robust_evaluation.py

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Dependencies
    ExtDeps(["External Dependencies<br/>(PyTorch, Python, GPU, MMSA Data)"]):::external

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        TS1(["train.sh"]):::script
        TS2(["train.py"]):::script

        CL(["Config Loader"]):::config
        DI(["Data Ingestion"]):::core
        subgraph "Model Factory"
            direction TB
            ML(["Basic Layers"]):::model
            MB(["BERT Integration"]):::model
            MLN(["LNLN Architecture"]):::model
        end

        subgraph "Compute"
            direction TB
            LS(["Loss Calculation"]):::core
            MT(["Metric Calculation"]):::core
            OP(["Optimizer"]):::core
            SC(["Scheduler"]):::core
            UT(["Utils"]):::core
        end

        subgraph "Persistence"
            direction TB
            CK(["Checkpoint Manager"]):::storage
            LG(["Logging"]):::storage
        end
    end

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        direction TB
        EF(["robust_evaluation.py"]):::script
        CL2(["Config Loader"]):::config
        DI2(["Data Ingestion"]):::core
        subgraph "Model Factory Eval"
            direction TB
            ML2(["Basic Layers"]):::model
            MB2(["BERT Integration"]):::model
            MLN2(["LNLN Architecture"]):::model
        end
        MT2(["Metric Calculation"]):::core
        OM(["Output Metrics"]):::external
    end

    %% Data Source
    DS(["MMSA Datasets"]):::external

    %% Connections Training
    ExtDeps -->|"provides environment"| TS2
    TS1 -->|"calls"| TS2
    TS2 -->|"loads"| CL
    CL -->|"parses YAML"| DI
    DS -->|"raw data"| DI
    DI -->|"batch tensors"| TS2
    TS2 -->|"builds"| ML & MB & MLN
    TS2 -->|"uses"| OP
    TS2 -->|"uses"| SC
    TS2 -->|"calls"| UT
    TS2 -->|"batch"| ML & MB & MLN
    ML & MB & MLN -->|"forward pass"| TS2
    TS2 -->|"computes"| LS
    LS -->|"loss value"| TS2
    TS2 -->|"backprop"| OP
    OP -->|"updates"| ML & MB & MLN
    SC -->|"schedules"| OP
    TS2 -->|"compute"| MT
    MT -->|"metrics"| TS2
    TS2 -->|"save"| CK
    TS2 -->|"log"| LG

    %% Connections Evaluation
    ExtDeps -->|"provides environment"| EF
    EF -->|"loads"| CL2
    CL2 -->|"parses YAML"| DI2
    DS -->|"raw data"| DI2
    DI2 -->|"batch tensors"| EF
    EF -->|"loads checkpoint"| CK
    EF -->|"uses model"| ML2 & MB2 & MLN2
    DI2 -->|"batch"| ML2 & MB2 & MLN2
    ML2 & MB2 & MLN2 -->|"forward pass"| EF
    EF -->|"compute"| MT2
    MT2 -->|"metrics"| OM

    %% Click Events
    click CL "https://github.com/haoyu-ha/lnln/tree/main/configs/"
    click DI "https://github.com/haoyu-ha/lnln/blob/main/core/dataset.py"
    click ML "https://github.com/haoyu-ha/lnln/blob/main/models/basic_layers.py"
    click MB "https://github.com/haoyu-ha/lnln/blob/main/models/bert.py"
    click MLN "https://github.com/haoyu-ha/lnln/blob/main/models/lnln.py"
    click TS1 "https://github.com/haoyu-ha/lnln/blob/main/train.sh"
    click TS2 "https://github.com/haoyu-ha/lnln/blob/main/train.py"
    click LS "https://github.com/haoyu-ha/lnln/blob/main/core/losses.py"
    click MT "https://github.com/haoyu-ha/lnln/blob/main/core/metric.py"
    click OP "https://github.com/haoyu-ha/lnln/blob/main/core/optimizer.py"
    click SC "https://github.com/haoyu-ha/lnln/blob/main/core/scheduler.py"
    click UT "https://github.com/haoyu-ha/lnln/blob/main/core/utils.py"
    click CK "https://github.com/haoyu-ha/lnln/tree/main/ckpt/"
    click LG "https://github.com/haoyu-ha/lnln/tree/main/log/"
    click EF "https://github.com/haoyu-ha/lnln/blob/main/robust_evaluation.py"

    %% Styles
    classDef script fill:#fdfd96,stroke:#333,stroke-width:1px
    classDef config fill:#ffeb99,stroke:#333,stroke-width:1px
    classDef core fill:#add8e6,stroke:#333,stroke-width:1px
    classDef model fill:#90ee90,stroke:#333,stroke-width:1px
    classDef storage fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef external fill:#e6e6fa,stroke:#333,stroke-width:1px
```
