# jxbbb/adapt architecture

> Architecture diagram of the GitHub repository jxbbb/adapt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jxbbb/adapt
- Repository: https://github.com/jxbbb/adapt

## Overview

##### Main Components
- Data Input & Pre-processing:
– Represent the raw video input and associated sensor data.
- Dataset and Data Loader:
– Show the datasets directory (with BDDX, BDDX_des, BDDX_exp) as a component that provides properly formatted data.
– Indicate the dataloader modules in src/datasets and src/timm/data that pack and batch video and language inputs.
- Core Models and Architectures:
– Component for the Video Backbone: Include video-Swin Transformer modules (under src/modeling/swin and src/modeling/video_swin) which extract visual features.
– Transformer and Language Encoder: Show the BERT layers (in src/layers/bert) which process textual inputs and contribute to the captioning head.
– Multitask Training Architecture: Highlight the integration in src/modeling/multitask_e2e_vid_swin_bert.py and video_captioning_e2e_vid_swin_bert.py that simultaneously handle control signal prediction and caption generation.
– Other Predictive Modules: Include additional modules like src/modeling/signal_predicting.py for control signal prediction.
- Training and Optimization:
– Represent the training tasks (under src/tasks, for example, run_adapt, run_adapt_inference, and run_signal) for orchestrating training routines.
– Showcase the solver and optimizer infrastructure (in src/solver) which implements learning rate schedulers, optimizers (like LARC, bertadam), and deepspeed/fairscale integration.
- Inference & Evaluation:
– Include the inference pipeline (scripts/inference.sh and src/tasks/visualize_attention.py) and evaluation components in src/evalcap that generate predictions and perform quality checks.
- Visualization Tools:
– Indicate the grad-CAM modules (src/pytorch_grad_cam) used for attention visualization and debugging.
- Deployment and Containerization:

##### Relationships and Data Flow
– Start from data ingestion (raw video input → data pre-processing → dataset management).
– Then move on to feeding the processed data into the backbone (video swin modules) and language modules (BERT layers).
– Merge these feature streams in the multitask module so the video representation drives both the driving caption generation and the control signal prediction.
- Feedback loops in training where losses and backpropagation computations cycle back to update the model.
- Additional flows for visualization diagnostics (grad-CAM modules receive intermediate activations).

##### Acknowledge Architectural Patterns and Design Principles
- End-to-end nature of the pipeline (data ingestion → feature extraction → multi-head prediction).
- Shared video representation that serves two tasks (caption generation and control prediction).
- Modular design where pre-processing, data loading, modeling, optimization, and testing are clearly separated into different directories/files.
- Use of state-of-the-art technologies such as transformers, mixed-precision training (Torch.amp, Apex, deepspeed), and Docker containerization to ensure reproducibility.

##### Relevant Technologies, Frameworks, and Libraries
- PyTorch, along with Torch.amp, Nvidia Apex, and DeepSpeed for training.
- Docker (with Dockerfiles and launch_container.sh) for containerized deployment.
- Additional libraries like mpi4py, and components from timm (PyTorch Image Models) for additional model architectures.
- The grad-CAM library implementation for visualization and debugging attention mechanisms.
- BERT and Swin Transformer implementations for language and video feature extraction respectively.

– Data Sources & Preprocessing: "Raw Video Data", "Preprocessor (Frame Extraction, TSV Creation)"
– Data Loading: "Dataset Loader (src/datasets, timm data)"
– Model Components:
- "Video Backbone (Video-Swin Transformer)"
- "Language Processing (BERT Layers)"
- "Multitask Fusion Module (Captioning & Signal Prediction)"
– Training Infrastructure: "Solver & Optimizer (LARC, BertAdam, etc.)", "Training Task Manager (src/tasks)"
– Inference/Evaluation: "Inference Engine", "Evaluation Scripts", "Attention Visualization (Grad-CAM)"
– Deployment Environment: "Docker/Conda Environment"

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input & Pre-processing
    subgraph "Data Input & Pre-processing"
        A1["Raw Video Data"]:::data
        A2["Preprocessor (Frame Extraction,TSV Creation)"]:::data
    end

    %% Dataset Management
    subgraph "Dataset Management"
        B1["Datasets Directory (BDD-X,BDD-X_des,BDD-X_exp)"]:::data
        B2["Data Loader (src/datasets & timm/data)"]:::data
    end

    %% Core Models & Architecture
    subgraph "Modeling & Architecture"
        subgraph "Video Backbone"
            M1["Swin Transformer Module"]:::model
            M2["Video-Swin Transformer Module"]:::model
        end
        M3["Language Processing (BERT Layers)"]:::model
        M4["Multitask Fusion Module (Captioning &Signal Prediction)"]:::model
    end

    %% Training Infrastructure
    subgraph "Training Infrastructure"
        T1["Training Task Manager (src/tasks)"]:::train
        T2["Solver & Optimizer (LARC, BertAdam)"]:::train
    end

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        I1["Inference Engine"]:::infer
        I2["Evaluation Scripts"]:::infer
    end

    %% Visualization Tools
    subgraph "Visualization Tools"
        V1["Attention Visualization (Grad-CAM)"]:::viz
    end

    %% Deployment Environment
    subgraph "Deployment Environment"
        D1["Docker/Conda Environment"]:::deploy
    end

    %% Data Flow Connections
    A1 -->|"feeds"| A2
    A2 -->|"outputs preprocessed data"| B1
    B1 -->|"supplies data"| B2

    %% Feeding data into models
    B2 -->|"feature extraction"| M1
    B2 -->|"text input"| M3
    B2 -->|"video processing"| M2

    %% Fusion of model outputs
    M1 -->|"combined with"| M4
    M2 -->|"combined with"| M4
    M3 -->|"combined with"| M4

    %% Training loop
    M4 -->|"training input"| T1
    T1 -->|"optimization"| T2
    T2 ---|"backpropagation"| M4

    %% Inference and evaluation flow
    M4 -->|"predictive output"| I1
    I1 -->|"post-processing"| I2

    %% Visualization flow from video backbone
    M1 -->|"activations"| V1

    %% Deployment encapsulation (dashed lines)
    D1 ---|"containerizes"| A2
    D1 ---|"containerizes"| B2
    D1 ---|"containerizes"| M4
    D1 ---|"containerizes"| T1
    D1 ---|"containerizes"| I1
    D1 ---|"containerizes"| V1

    %% Click Events for Component Mapping
    click A2 "https://github.com/jxbbb/adapt/tree/main/src/prepro"
    click B1 "https://github.com/jxbbb/adapt/tree/main/src/datasets"
    click B2 "https://github.com/jxbbb/adapt/tree/main/src/timm/data"
    click M1 "https://github.com/jxbbb/adapt/tree/main/src/modeling/swin"
    click M2 "https://github.com/jxbbb/adapt/tree/main/src/modeling/video_swin"
    click M3 "https://github.com/jxbbb/adapt/tree/main/src/layers/bert"
    click M4 "https://github.com/jxbbb/adapt/blob/main/src/modeling/multitask_e2e_vid_swin_bert.py"
    click M4 "https://github.com/jxbbb/adapt/blob/main/src/modeling/video_captioning_e2e_vid_swin_bert.py"
    click M4 "https://github.com/jxbbb/adapt/blob/main/src/modeling/signal_predicting.py"
    click T1 "https://github.com/jxbbb/adapt/tree/main/src/tasks"
    click T2 "https://github.com/jxbbb/adapt/tree/main/src/solver"
    click I1 "https://github.com/jxbbb/adapt/blob/main/scripts/inference.sh"
    click I1 "https://github.com/jxbbb/adapt/blob/main/src/tasks/visualize_attention.py"
    click I2 "https://github.com/jxbbb/adapt/tree/main/src/evalcap"
    click V1 "https://github.com/jxbbb/adapt/tree/main/src/pytorch_grad_cam"
    click D1 "https://github.com/jxbbb/adapt/blob/main/launch_container.sh"
    click D1 "https://github.com/jxbbb/adapt/blob/main/requirements.txt"

    %% Class Definitions for Styling
    classDef data fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef model fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef train fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef infer fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef viz fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef deploy fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
