# fsoft-aic/mamba-token-dynamic architecture

> Architecture diagram of the GitHub repository fsoft-aic/mamba-token-dynamic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fsoft-aic/mamba-token-dynamic
- Repository: https://github.com/fsoft-aic/mamba-token-dynamic

## Overview

This project is a research codebase targeting experiments on the token dynamics of deep selective state space models. It is structured to run experiments on both language modeling (using WikiText-103) and image classification (with ImageNet) scenarios, and it incorporates both PyTorch-based deep learning pipelines and custom CUDA/C++ kernels (e.g., for FFT-based convolution). The overall goal is to experiment with different eigenvalue scenarios (positive, negative, and mixed) in the state space models and to evaluate their effects on sequence tasks as well as in vision tasks via the “MambaVision” module.

##### Main Components in the System
- Configuration Manager: The project uses Hydra for dynamic configuration management. All configuration files (organized under mamba_wt103/configs) include settings for callbacks, dataset, experiment parameters, model layers, optimizers, pipelines, schedulers, tasks, and trainer parameters.
- Training and Experiment Runner: The core experiment runner is “train.py” (in mamba_wt103 and mambavision_imagenet/mambavision) which reads configuration via Hydra and launches training routines.
- Data Pipeline: The data-related modules under mamba_wt103/src/dataloaders manage dataset ingestion, augmentation, and preparation for both text (language modeling) and vision tasks. Each dataloader component (for instance, the specialized “lm”, “lm_dataset”, “vision”, etc.) forms part of this pipeline.
- Model Components: Located in mamba_wt103/src/models, these include different architectures for sequence modeling (e.g., baselines, transformers, SSM-based models, “mamba” variants) and for vision tasks found in mambavision_imagenet/mambavision/models.
- Custom Operations: The csrc/fftconv directory contains optimized C++/CUDA kernels for FFT-based convolutions and custom operations. These are integrated into the model’s forward passes as custom ops.
- Optimization and Training Utilities: Under mamba_wt103/src/utils (plus specific modules in src/utils/optim, optim_groups, and schedulers) there are implementations of various optimizers (Adam, AdamW, Lamb, SGD, etc.) and learning rate schedulers.
- Callbacks and Logging: The project includes several callbacks (such as debugging, progressive resizing, timers, and wandb integration) for monitoring and logging experiment performance. The outputs directory shows logs, hydra configuration snapshots, and evaluation metrics.
- Experiment Outputs and Evaluation Modules: The “outputs” directory structures experiment run results and includes logs, configuration trees, and artifact directory trees that are generated automatically by Hydra.

- The Experiment Runner (train.py) initializes the Data Pipeline (dataloaders) to preprocess data based on configuration (e.g., WikiText-103 for language or ImageNet for vision) and loads models from the “src/models” module.
- The Data Pipeline feeds data into the Model. The model uses custom operations (e.g., FFT-based convolutions from csrc/fftconv) and neural network components from “models/nn” and “models/sequence”.
- During training, the Optimizers and Schedulers (from both configuration and src/utils/optim) are applied to update model parameters.
- The Callback mechanisms periodically log metrics (using wandb and local logging via Hydra outputs), trigger checkpoints, and possibly adjust experiments (like progressive resizing).
- Finally, evaluation modules compute performance metrics which then loop back into the reporting/logging components.

##### Architectural Patterns and Design Principles
- Modularity: Each responsibility (configuration, data ingestion, modeling, custom ops, training loop, logging) is separated clearly into directories and modules.
- Config-driven Experimentation: Hydra is central so that experiments can be reconfigured quickly without altering code.
- Extensibility: The design supports multiple domains (text and vision) and can incorporate variations (such as different eigenvalue scenarios) via configuration overrides.
- Integration of Custom Kernels: The custom C++/CUDA components for FFT and convolution are seamlessly integrated with PyTorch in the forward passes.

- Major layers as described.
- Relationships like data ingestion, training, backpropagation, logging, and output generation are clearly depicted.

## Diagram (Mermaid)

```mermaid
flowchart TD
    U["User/Experiment Command"]:::userFill
    HCM["Hydra Config Manager"]:::configFill

    subgraph "Experiment Runner"
        T1["Train.py (Text)"]:::runnerFill
        T2["Train.py (Vision)"]:::runnerFill
    end

    DP["Data Pipeline"]:::datapipelineFill

    subgraph "Model Components (Language)"
        M_seq["Sequence Models"]:::modelFill
        M_nn["NN Models"]:::modelFill
    end

    M_vis["Vision Models"]:::modelFill
    M_agg["Model Aggregator"]:::modelFill

    subgraph "Custom Operations"
        FFT_cuda["Custom FFT Ops (CUDA)"]:::customFill
        FFT_py["Custom FFT Ops (Python)"]:::customFill
    end

    subgraph "Optimization & Training Utilities"
        OPT["Optimizers"]:::optimFill
        SCHED["Schedulers"]:::optimFill
        OPT_groups["Optim Groups"]:::optimFill
    end

    subgraph "Callbacks & Logging"
        CALLBACK["Callbacks"]:::callbackFill
        OUTPUT["Outputs/Logging"]:::callbackFill
    end

    EVAL["Evaluation Modules"]:::evalFill

    U --> HCM
    HCM --> T1
    HCM --> T2
    T1 --> DP
    T2 --> DP
    DP --> M_seq
    DP --> M_nn
    DP --> M_vis
    M_seq --> M_agg
    M_nn --> M_agg
    M_vis --> M_agg
    M_agg --> FFT_cuda
    M_agg --> FFT_py
    M_agg --> OPT
    OPT --> SCHED
    SCHED --> OPT_groups
    OPT_groups --> CALLBACK
    CALLBACK --> OUTPUT
    OUTPUT --> EVAL
    EVAL --> CALLBACK

    click HCM "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/configs"
    click T1 "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mamba_wt103/train.py"
    click T2 "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mambavision_imagenet/mambavision/train.py"
    click DP "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/src/dataloaders"
    click M_seq "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/src/models/sequence"
    click M_nn "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/src/models/nn"
    click M_vis "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mambavision_imagenet/mambavision/models/mamba_vision.py"
    click FFT_cuda "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/csrc/fftconv"
    click FFT_py "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mamba_wt103/src/ops/fftconv.py"
    click OPT "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/src/utils/optim"
    click SCHED "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mamba_wt103/src/utils/optim/schedulers.py"
    click OPT_groups "https://github.com/fsoft-aic/mamba-token-dynamic/blob/master/mamba_wt103/src/utils/optim_groups.py"
    click CALLBACK "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/src/callbacks"
    click OUTPUT "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/outputs"
    click EVAL "https://github.com/fsoft-aic/mamba-token-dynamic/tree/master/mamba_wt103/evals"

    classDef userFill fill:#FDEBD0,stroke:#CA6F1E,stroke-width:2px;
    classDef configFill fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef runnerFill fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef datapipelineFill fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef modelFill fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef customFill fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef optimFill fill:#FADBD8,stroke:#C0392B,stroke-width:2px;
    classDef callbackFill fill:#85C1E9,stroke:#2E86C1,stroke-width:2px;
    classDef evalFill fill:#D5F5E3,stroke:#28B463,stroke-width:2px;
```
