How haoyu-ha/lnln works
Project Type and Purpose
• This is a deep-learning research codebase (PyTorch) implementing “Towards Robust Multimodal Sentiment Analysis with Incomplete Data.”
• It’s a single monolithic training/evaluation pipeline (not microservices), configurable via YAML, producing checkpoints and evaluation metrics.
File Structure
• Top-level scripts:
• train.sh + train.py → orchestrate training
• robust_evaluation.py → evaluation of saved checkpoints
• configs/ → YAML files for training and eval on three datasets (MOSI, MOSEI, SIMS)
• core/ → reusable modules: dataset, loss functions, metrics, optimizer & scheduler wrappers, utilities
• models/ → network definitions (basic layers, BERT integration, proposed LNLN architecture)
• ckpt/ & log/ → storage for model checkpoints and run logs
Extract Key README Insights
• Uses PyTorch 2.2.1, Python 3.11.7, GPU (Tesla A40)
• Data preparation: external MMSA repository
• Distinct training vs. evaluation flows controlled by config files
Main Components & Relationships
a. Components
1. Config Loader (parses YAML)
2. Data Ingestion (core/dataset.py + external dataset files)
3. Model Factory (models/basic_layers.py, bert.py, lnln.py)
4. Training Loop (train.py + core/utils for epoch management)
5. Loss & Metric Calculation (core/losses.py, core/metric.py)
6. Optimizer & Scheduler (core/optimizer.py, core/scheduler.py)
7. Checkpoint Manager (saves/loads from ckpt/)
8. Logging (writes to log/)
9. Evaluation Flow (robust_evaluation.py uses same core & models)
10. External Dependencies (PyTorch, Python, GPU, MMSA data)
b. Interactions
• train.sh → invokes train.py with a chosen config
• train.py →
• reads config → instantiates DataLoader → Model → Optimizer/Scheduler
• for each batch: forward → compute loss (core/losses) → backward → optimizer step → scheduler step → compute metrics → log → optionally save checkpoint
• robust_evaluation.py → loads config → loads checkpoint → runs forward passes on test splits → computes & prints metrics
c. Architectural Patterns & Principles
• Modular layering: config → core framework → models → persistence (ckpt/log)
• Separation of concerns: core/ modules abstract away framework boilerplate
• Config-driven: no hard-coded hyperparameters, makes it extensible
d. Key Technologies
• PyTorch 2.2.1 (deep learning)
• YAML (configuration)
• MMSA datasets (Multimodal Sentiment Analysis)
• BERT (pretrained transformer for text modality)
Detailed Recommendations
• That configs drive both scripts, and that core/ modules are shared by train.py and robust_evaluation.py