# wangtz19/netmamba architecture

> Architecture diagram of the GitHub repository wangtz19/netmamba, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wangtz19/netmamba
- Repository: https://github.com/wangtz19/netmamba

## Overview

##### Project Type and Purpose
- NetMamba is a research-grade machine learning pipeline for network traffic classification.
- It leverages a customized “Mamba” SSM (State-Space Model) library (mamba-1p1p1) plus PyTorch to pre-train and fine-tune models on packet-level datasets.

##### File Structure
- Top-level directories:
- dataset/ – scripts to parse, preprocess, and generate training data.
- mamba-1p1p1/ – the Mamba library, with C++/CUDA kernels (csrc), Python bindings (mamba_ssm/ops, triton), tests, and benchmarks.
- src/ – high-level orchestration: engine.py (training loop), pre-train.py, fine-tune.py, eval.py, plus model definitions (models_mamba.py) and utilities (src/util).

##### Extract Key Components from README
- Data is downloaded or generated via dataset/ scripts.
- Pre-training and fine-tuning are separate scripts invoked with GPU environment variables.
- The Mamba library is installed in “editable” mode and supplies custom ops for PyTorch.
- Checkpoints are saved to disk; evaluation is performed in eval.py or fine-tune.py.

1. Data Storage
- Raw network captures or folder tree of train/valid/test samples
2. Dataset Preprocessing
- dataset/dataset_*.py scripts
- Output: serialized samples or PyTorch Dataset objects
3. Training Pipeline
- src/engine.py orchestrates training loops
- src/pre-train.py for self-supervised pre-training
- src/fine-tune.py for supervised fine-tuning and evaluation
4. Model Components
- mamba_ssm/models & modules: SSM layers, Mixer, sequence modules
- src/models_mamba.py wrapping NetMamba architectures
5. Mamba Library (mamba-1p1p1)
- Python API (mamba_ssm/ops, utils, generation)
- C++/CUDA kernels (csrc/selective_scan, triton ops)
6. Runtime Environment
- Python 3.10 + PyTorch GPU accel
- CUDA/cuDNN back-end
7. Logging & Checkpointing
- TensorBoard or custom logs in log_dir
- Model weights in output_dir
8. External Services
- Google Drive (dataset download)
- Hugging Face (checkpoint hosting)

- Data Storage → Dataset Preprocessing (runs dataset scripts)
- Dataset Preprocessing → PyTorch DataLoader (engine.py)
- DataLoader → Training Pipeline
- Training Pipeline → Model Components → Mamba Library
- Mamba Library ↔ GPU (executes custom kernels)
- Training Pipeline → Logging & Checkpointing
- Checkpoints → Fine-tune & Eval scripts
- External Services ↔ Data Storage / Checkpoints

c. Architectural Patterns & Principles
- Modular layering: clear separation of data, model, compute kernels, and orchestration scripts.
- Pluggable ops: leveraging Python bindings over C++/CUDA for high performance.
- Two-stage training: pre-training (self-supervised) then fine-tuning (supervised).

d. Key Technologies
- Python, PyTorch
- C++/CUDA, Triton for custom ops
- Conda/pip for environment management
- TensorBoard (or similar) for logging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Storage" 
        direction TB
        DS["dataset/"]:::data
    end

    subgraph "Dataset Preprocessing" 
        direction TB
        DP1["dataset_all.py"]:::data
        DP2["dataset_common.py"]:::data
        DP3["dataset_cic_iot2022.py"]:::data
        DP4["dataset_cross_platform.py"]:::data
        DP5["dataset_iscx_tor2016.py"]:::data
        DP6["dataset_iscx_vpn2016.py"]:::data
        DP7["dataset_ustc_tfc2016.py"]:::data
    end

    DS -->|"runs scripts"| DP1
    DS --> DP2
    DS --> DP3
    DS --> DP4
    DS --> DP5
    DS --> DP6
    DS --> DP7

    %% DataLoader
    DL["PyTorch DataLoader"]:::compute

    DP1 --> DL
    DP2 --> DL
    DP3 --> DL
    DP4 --> DL
    DP5 --> DL
    DP6 --> DL
    DP7 --> DL

    %% Training Pipeline
    subgraph "Training Pipeline Orchestration"
        direction TB
        TP1["engine.py"]:::compute
        TP2["pre-train.py"]:::compute
        TP3["fine-tune.py"]:::compute
        TP4["eval.py"]:::compute
    end

    DL -->|"feeds"| TP1
    TP1 --> TP2
    TP1 --> TP3
    TP1 --> TP4

    %% Model Definitions
    subgraph "Model Components"
        direction TB
        MD1["models_mamba.py"]:::compute
        MD2["models_net_mamba.py"]:::compute
    end

    TP1 --> MD1
    TP1 --> MD2
    TP2 --> MD1
    TP3 --> MD2

    %% Utility Functions
    subgraph "Utilities"
        direction TB
        U1["crop.py"]:::compute
        U2["lars.py"]:::compute
        U3["lr_decay.py"]:::compute
        U4["lr_sched.py"]:::compute
        U5["misc.py"]:::compute
        U6["pos_embed.py"]:::compute
    end

    TP1 --> U1
    TP1 --> U2
    TP1 --> U3
    TP1 --> U4
    TP1 --> U5
    TP1 --> U6

    %% Mamba Library
    subgraph "Mamba Library (mamba-1p1p1)" 
        direction TB
        ML1["mamba_ssm/ops"]:::compute
        ML2["mamba_ssm/models"]:::compute
        ML3["mamba_ssm/modules"]:::compute
        ML4["mamba_ssm/utils"]:::compute
        ML5["csrc/selective_scan/"]:::compute
    end

    MD1 --> ML1
    MD1 --> ML2
    MD1 --> ML3
    MD1 --> ML4
    ML1 --> ML5

    %% GPU Kernel Execution
    GPU["GPU/CUDA"]:::compute
    ML5 -->|"executes kernels on"| GPU

    %% Logging & Checkpointing
    LP["Logging & Checkpointing"]:::compute
    TP1 --> LP
    TP2 --> LP
    TP3 --> LP

    %% External Services
    subgraph "External Services"
        direction TB
        ES1["Google Drive"]:::external
        ES2["Hugging Face"]:::external
    end

    ES1 --> DS
    ES2 --> LP

    %% Click Events
    click DS "https://github.com/wangtz19/netmamba/tree/main/dataset/"
    click DP1 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_all.py"
    click DP2 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_common.py"
    click DP3 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_cic_iot2022.py"
    click DP4 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_cross_platform.py"
    click DP5 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_iscx_tor2016.py"
    click DP6 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_iscx_vpn2016.py"
    click DP7 "https://github.com/wangtz19/netmamba/blob/main/dataset/dataset_ustc_tfc2016.py"
    click TP1 "https://github.com/wangtz19/netmamba/blob/main/src/engine.py"
    click TP2 "https://github.com/wangtz19/netmamba/blob/main/src/pre-train.py"
    click TP3 "https://github.com/wangtz19/netmamba/blob/main/src/fine-tune.py"
    click TP4 "https://github.com/wangtz19/netmamba/blob/main/src/eval.py"
    click MD1 "https://github.com/wangtz19/netmamba/blob/main/src/models_mamba.py"
    click MD2 "https://github.com/wangtz19/netmamba/blob/main/src/models_net_mamba.py"
    click U1 "https://github.com/wangtz19/netmamba/blob/main/src/util/crop.py"
    click U2 "https://github.com/wangtz19/netmamba/blob/main/src/util/lars.py"
    click U3 "https://github.com/wangtz19/netmamba/blob/main/src/util/lr_decay.py"
    click U4 "https://github.com/wangtz19/netmamba/blob/main/src/util/lr_sched.py"
    click U5 "https://github.com/wangtz19/netmamba/blob/main/src/util/misc.py"
    click U6 "https://github.com/wangtz19/netmamba/blob/main/src/util/pos_embed.py"
    click ML1 "https://github.com/wangtz19/netmamba/tree/main/mamba-1p1p1/mamba_ssm/ops/"
    click ML2 "https://github.com/wangtz19/netmamba/tree/main/mamba-1p1p1/mamba_ssm/models/"
    click ML3 "https://github.com/wangtz19/netmamba/tree/main/mamba-1p1p1/mamba_ssm/modules/"
    click ML4 "https://github.com/wangtz19/netmamba/tree/main/mamba-1p1p1/mamba_ssm/utils/"
    click ML5 "https://github.com/wangtz19/netmamba/tree/main/mamba-1p1p1/csrc/selective_scan/"

    %% Styles
    classDef data fill:#E0F7FA,stroke:#0277BD,color:#01579B
    classDef compute fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20
    classDef external fill:#FFF3E0,stroke:#EF6C00,color:#E65100
```
