# sajayr/triad architecture

> Architecture diagram of the GitHub repository sajayr/triad, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sajayr/triad
- Repository: https://github.com/sajayr/triad

## Overview

##### Project Type and Purpose
- This project is a research prototype for dense cross-modal grounding across video, audio, and text. Its purpose is to learn fine-grained feature correspondences using three backbone models (DINOv2 for vision, HuBERT for audio, and DistilBERT for text) that feed into a shared cross-modal attention mechanism, with contrastive learning applied.

##### Analyzing the File Structure
- The top-level README.md outlines the overall project idea. The src folder contains most of the code modules:
- dataset.py – Likely responsible for data loading and preprocessing from sources like AudioSet and CC3M.
- model.py – Implements the model, including the three backbone networks and the multi-modal attention layer.
- train.py – Contains training routines, possibly including the contrastive learning loop and epoch management.
- viz.py – Handles visualization routines (likely for feature maps or alignment results).
- The src/conts directory houses additional content (e.g., an image example for the README visual).

##### Key Components
a. Data Input and Preprocessing:
- Data Sources: Indicate the video (and image) inputs, audio files, and text inputs.
- Preprocessing/Dataset Module (from src/dataset.py): Normalization, parsing of multi-modal data, dataset construction.

b. Backbone Feature Extractors (from src/model.py):
- Vision Backbone: DINOv2 – for extracting dense visual features.
- Audio Backbone: HuBERT – for generating audio features.
- Text Backbone: DistilBERT – for processing textual inputs.

c. Cross-Modal Attention and Feature Alignment:
- Shared cross-modal attention mechanism that fuses or aligns features produced by the three backbones.
- Contrastive learning module that pushes aligned features together and non-corresponding ones apart (include max-mean pooling strategy if space permits).

d. Training Pipeline:
- Training Module (from src/train.py): Contains the training loop, loss computation for contrastive learning, and backpropagation.
- How data flows from the dataset module through the backbone modules to the attention mechanism, and back via the training loop (update weights).

e. Visualization:
- Visualization Module (from src/viz.py): Illustrates how the model’s output (dense feature maps/grounding localization) is visualized for analysis.

##### Relationships and Interactions
- Data flows from the dataset module to the individual backbone extractors.
- Each extractor outputs modality-specific features that feed into the cross-modal attention module.
- The attention module then aligns these features and produces a unified, dense cross-modal representation.
- The training pipeline uses these outputs and applies a contrastive loss to iteratively improve the alignment.
- Visualization utilities then provide visual feedback on the learned dense correspondences.

##### Architectural Patterns and Technologies
- Modular separation (data, model, training, viz).
- That this design leverages a modular deep learning architecture with specialized preprocessing and training strategies.
- Mention that it’s built upon state-of-the-art backbones (DINOv2, HuBERT, and DistilBERT) and employs contrastive learning – important for understanding the experimental research nature of the project.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Input Layer
    subgraph "Data Input Layer"
        DI["Data Input & Preprocessing"]:::dataset
    end

    %% Backbone Feature Extraction
    subgraph "Backbone Feature Extraction"
        V["Vision Extractor (DINOv2)"]:::backbone
        A["Audio Extractor (HuBERT)"]:::backbone
        T["Text Extractor (DistilBERT)"]:::backbone
    end

    %% Fusion & Learning
    subgraph "Fusion & Learning"
        F["Cross-Modal Attention & Fusion Module<br/>(includes Max-Mean Pooling)"]:::fusion
        CL["Contrastive Learning & Training Loop"]:::training
    end

    %% Visualization Output
    subgraph "Visualization Output"
        VIZ["Visualization Module"]:::viz
    end

    %% Documentation
    subgraph "Documentation"
        DOC["Project Overview & Documentation"]:::doc
        SC["Supplementary Content"]:::doc
    end

    %% Connections between components
    DI --> V
    DI --> A
    DI --> T
    V --> F
    A --> F
    T --> F
    F --> CL
    CL --> VIZ

    %% Click Events
    click DI "https://github.com/sajayr/triad/blob/main/src/dataset.py"
    click V "https://github.com/sajayr/triad/blob/main/src/model.py"
    click A "https://github.com/sajayr/triad/blob/main/src/model.py"
    click T "https://github.com/sajayr/triad/blob/main/src/model.py"
    click F "https://github.com/sajayr/triad/blob/main/src/model.py"
    click CL "https://github.com/sajayr/triad/blob/main/src/train.py"
    click VIZ "https://github.com/sajayr/triad/blob/main/src/viz.py"
    click DOC "https://github.com/sajayr/triad/blob/main/README.md"
    click SC "https://github.com/sajayr/triad/tree/main/src/conts"

    %% Styles
    classDef dataset fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef backbone fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef fusion fill:#ABEBC6,stroke:#333,stroke-width:2px;
    classDef training fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef viz fill:#D7BDE2,stroke:#333,stroke-width:2px;
    classDef doc fill:#D5F5E3,stroke:#333,stroke-width:2px;
```
