How wangtz19/netmamba works
Project Type and Purpose
• NetMamba is a research-grade machine learning pipeline for network traffic classification.
• It leverages a customized “Mamba” SSM (State-Space Model) library (mamba-1p1p1) plus PyTorch to pre-train and fine-tune models on packet-level datasets.
File Structure
• Top-level directories:
• dataset/ – scripts to parse, preprocess, and generate training data.
• mamba-1p1p1/ – the Mamba library, with C++/CUDA kernels (csrc), Python bindings (mamba_ssm/ops, triton), tests, and benchmarks.
• src/ – high-level orchestration: engine.py (training loop), pre-train.py, fine-tune.py, eval.py, plus model definitions (models_mamba.py) and utilities (src/util).
Extract Key Components from README
• Data is downloaded or generated via dataset/ scripts.
• Pre-training and fine-tuning are separate scripts invoked with GPU environment variables.
• The Mamba library is installed in “editable” mode and supplies custom ops for PyTorch.
• Checkpoints are saved to disk; evaluation is performed in eval.py or fine-tune.py.
1. Data Storage
• Raw network captures or folder tree of train/valid/test samples
2. Dataset Preprocessing
• dataset/dataset_*.py scripts
• Output: serialized samples or PyTorch Dataset objects
3. Training Pipeline
• src/engine.py orchestrates training loops
• src/pre-train.py for self-supervised pre-training
• src/fine-tune.py for supervised fine-tuning and evaluation
4. Model Components
• mamba_ssm/models & modules: SSM layers, Mixer, sequence modules
• src/models_mamba.py wrapping NetMamba architectures
5. Mamba Library (mamba-1p1p1)
• Python API (mamba_ssm/ops, utils, generation)
• C++/CUDA kernels (csrc/selective_scan, triton ops)
6. Runtime Environment
• Python 3.10 + PyTorch GPU accel
• CUDA/cuDNN back-end
7. Logging & Checkpointing
• TensorBoard or custom logs in log_dir
• Model weights in output_dir
8. External Services
• Google Drive (dataset download)
• Hugging Face (checkpoint hosting)
• Data Storage → Dataset Preprocessing (runs dataset scripts)
• Dataset Preprocessing → PyTorch DataLoader (engine.py)
• DataLoader → Training Pipeline
• Training Pipeline → Model Components → Mamba Library
• Mamba Library ↔ GPU (executes custom kernels)
• Training Pipeline → Logging & Checkpointing
• Checkpoints → Fine-tune & Eval scripts
• External Services ↔ Data Storage / Checkpoints
c. Architectural Patterns & Principles
• Modular layering: clear separation of data, model, compute kernels, and orchestration scripts.
• Pluggable ops: leveraging Python bindings over C++/CUDA for high performance.
• Two-stage training: pre-training (self-supervised) then fine-tuning (supervised).
d. Key Technologies
• Python, PyTorch
• C++/CUDA, Triton for custom ops
• Conda/pip for environment management
• TensorBoard (or similar) for logging