# ispc-lab/tma architecture

> Architecture diagram of the GitHub repository ispc-lab/tma, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ispc-lab/tma
- Repository: https://github.com/ispc-lab/tma

## Overview

##### Project Type and Purpose
- This is a research‐oriented deep‐learning codebase implementing the TMA network for event‐based optical flow.
- Key phases: data generation (“data_utils”), data loading & preprocessing (“dataloader”), model definition (“model”), training (“train.py” / “train_split.py”), and evaluation (“evaluate.py”).

##### File Structure
- Top level:
- data_utils/        → scripts to convert raw DSEC data into .npz/.npy volumes
- dataloader/        → PyTorch Dataset and augmentation classes
- datasets/          → prepared data on disk
- model/             → network modules (backbone, correlation, aggregation, update, TMA head)
- train.py / train_split.py / train.sh → training entrypoints
- evaluate.py        → inference and benchmark submission
- utils/             → file I/O utilities
- Notice “train_split.py” uses a smaller “dsec_split” for rapid prototyping, while “train.sh” orchestrates full‐dataset runs.

##### Gather Insights from README
- Preprocessing follows E-RAFT style (event volumes + flow arrays).
- Data lives under `datasets/dsec_full` or `datasets/dsec_split`.
- Training can log to WandB, outputs saved under a checkpoint directory.
- Evaluation script produces submissions for the online benchmark.

##### Main Components & Relationships
A. Data Preparation Layer
- data_utils/gen_dsec.py  → reads raw .npz/.npy, writes processed event volumes and flows
- data_utils/gen_dsec_upload.py → prepares test‐time volumes for benchmark upload
B. Dataset Storage
- datasets/dsec_full/…   → trainval/test subfolders with .npz/.npy
- datasets/dsec_split/…  → small train/val splits for development
C. Data Loading & Augmentation
- dataloader/dsec_full.py and dataloader/dsec_split.py create PyTorch Datasets
- dataloader/augment.py applies spatial/temporal augmentations
- dataloader/representation.py converts raw events into network inputs
D. Model Definition
- model/backbone.py      → feature extractor
- model/corr.py          → correlation cost volume
- model/aggregate.py     → temporal motion aggregation module
- model/update.py        → flow field refinement
- model/TMA.py           → overall network assembly
- model/util.py          → helpers (tensor ops, warping)
E. Training & Checkpointing
- train.py               → full‐dataset training loop
- train_split.py         → training on split dataset
- train.sh               → shell wrapper for train.py
- utils/file_utils.py    → checkpoint I/O, logging
- Optionally uses WandB for loss visualization
F. Evaluation & Submission
- evaluate.py            → loads checkpoint, runs inference on test set, writes `.npz` for benchmark upload

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% 1. Data Preparation Layer
    subgraph "1. Data Preparation Layer"
        A1["gen_dsec.py"]:::data
        A2["gen_dsec_upload.py"]:::data
    end

    %% 2. Dataset Storage
    subgraph "2. Dataset Storage"
        B["datasets/dsec_full & dsec_split"]:::storage
    end

    A1 -->|process raw→| B
    A2 -->|prepare test volumes→| B

    %% 3. Data Loading & Augmentation
    subgraph "3. Data Loading & Augmentation"
        C1["dsec_full.py"]:::data
        C2["dsec_split.py"]:::data
        C3["augment.py"]:::data
        C4["representation.py"]:::data
    end

    B -->|.npz/.npy→| C1
    B -->|.npz/.npy→| C2
    C1 -->|dataset→| C3 -->|augmented events→| C4
    C2 -->|dataset→| C3

    %% 4. Model Definition
    subgraph "4. Model Definition"
        D1["backbone.py"]:::model
        D2["corr.py"]:::model
        D3["aggregate.py"]:::model
        D4["update.py"]:::model
        D5["TMA.py"]:::model
        D6["util.py"]:::util
    end

    C4 -->|tensor batches→| D1 --> D2 --> D3 --> D4 --> D5

    %% 5. Training & Checkpointing
    subgraph "5. Training & Checkpointing"
        E1["train.py"]:::process
        E2["train_split.py"]:::process
        E3["train.sh"]:::process
        E4["file_utils.py"]:::util
        W["WandB"]:::external
    end

    D5 -->|model input→| E1
    D5 -->|model input→| E2
    E3 -->|calls full train→| E1
    E1 -->|save checkpoint→| E4
    E1 -->|log metrics→| W
    E1 -.->|update weights→| D5

    %% 6. Evaluation & Submission
    subgraph "6. Evaluation & Submission"
        F["evaluate.py"]:::process
        G["Inference Results (.npz)"]:::data
        H["Benchmark Upload"]:::external
    end

    E4 -->|load checkpoint→| F
    F -->|output npz→| G -->|submit→| H

    %% Click Events
    click A1 "https://github.com/ispc-lab/tma/blob/main/data_utils/gen_dsec.py"
    click A2 "https://github.com/ispc-lab/tma/blob/main/data_utils/gen_dsec_upload.py"
    click B "https://github.com/ispc-lab/tma/tree/main/datasets/dsec_split/"
    click C1 "https://github.com/ispc-lab/tma/blob/main/dataloader/dsec_full.py"
    click C2 "https://github.com/ispc-lab/tma/blob/main/dataloader/dsec_split.py"
    click C3 "https://github.com/ispc-lab/tma/blob/main/dataloader/augment.py"
    click C4 "https://github.com/ispc-lab/tma/blob/main/dataloader/representation.py"
    click D1 "https://github.com/ispc-lab/tma/blob/main/model/backbone.py"
    click D2 "https://github.com/ispc-lab/tma/blob/main/model/corr.py"
    click D3 "https://github.com/ispc-lab/tma/blob/main/model/aggregate.py"
    click D4 "https://github.com/ispc-lab/tma/blob/main/model/update.py"
    click D5 "https://github.com/ispc-lab/tma/blob/main/model/TMA.py"
    click D6 "https://github.com/ispc-lab/tma/blob/main/model/util.py"
    click E1 "https://github.com/ispc-lab/tma/blob/main/train.py"
    click E2 "https://github.com/ispc-lab/tma/blob/main/train_split.py"
    click E3 "https://github.com/ispc-lab/tma/blob/main/train.sh"
    click E4 "https://github.com/ispc-lab/tma/blob/main/utils/file_utils.py"
    click F "https://github.com/ispc-lab/tma/blob/main/evaluate.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef storage fill:#b3e5fc,stroke:#039be5,color:#01579b
    classDef model fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef process fill:#dcedc8,stroke:#7cb342,color:#33691e
    classDef util fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef external fill:#ffccbc,stroke:#f4511e,color:#bf360c
```
