How rainmaker22/smart works
Project Type and Purpose
• This is not a traditional full-stack web application but instead an advanced deep learning framework for autonomous driving motion generation. The project implements SMART – a scalable multi-agent, real-time motion generation system based on next-token predictions. It covers data preprocessing, model definition, training/evaluation, and integration of various modules (e.g., layers, metrics, tokens, and transforms).
Break Down the File Structure
• Top-level directories/files:
• configs: Contains YAML configuration files for training (train_scalable.yaml) and validation (validation_scalable.yaml), illustrating configurable hyperparameters and possibly training/evaluation settings.
• data: Includes raw or demo data in pickle format and a directory structure that hints to how data is organized.
• scripts: Contains supplementary scripts such as installation helpers for PyG and specific tools like traj_clustering.
• smart: This is the main package and is split into several sub-modules
• datamodules and datasets for handling data input and serving batches;
• layers include implementations of attention layers, Fourier embeddings, and MLP layers for building the neural network;
• model contains the main SMART model implementation;
• modules potentially orchestrate the decoding for agents and maps;
• tokens house pre-generated tokens (likely from clustering or quantization) used in the discrete sequence modeling;
• transforms, metrics, and utils provide support for data transformation, performance measurement, and general utility functions.
• train.py and val.py: Orchestrate the training and evaluation loops using the architectural components.
• data_preprocess.py: An independent script for converting raw Waymo data to preprocessed tokens.
• Environment and dependency files (environment.yml, requirements.txt) ensure the right runtime and package versions.
Main Components to Capture
a. Data Input & Preprocessing:
• Raw Waymo Open Motion Dataset (organized externally as described in README under data/waymo).
• Data Preprocessing module, triggered via data_preprocess.py or smart/preprocess/preprocess.py, converts raw data into preprocessed tokens & structured formats.
• Preprocessed output stored under data/ (or data/waymo_processed once generated).
b. Data Management & Loading:
• DataModules (smart/datamodules) and Dataset classes (smart/datasets) load and serve data for training and evaluation.
• Configuration specified by YAML files in the configs directory directs data handling (e.g., scalable settings).
c. Network Architecture:
• Model Core: Located in smart/model (smart.py) aggregates different components for motion generation.
• Layers: smart/layers (attention_layer, fourier_embedding, mlp_layer) form the building blocks of the model.
• Modules: smart/modules (agent_decoder, map_decoder, smart_decoder) handle specific decoding tasks from the model’s output.
• Tokens: smart/tokens represent how map and trajectory data are discretized into sequence tokens.
• Transforms: smart/transforms (target_builder) supports target preparation for training.
d. Training and Evaluation Pipeline:
• Training script (train.py) integrates the model, data modules, and configuration to run the training loop.
• Evaluation script (val.py) uses the trained model and involves metrics (found in smart/metrics) such as min_ade, min_fde, next_token_cls, average_meter, etc.
• Additional scripts and utilities (e.g., smart/utils) provide logging, weight initialization, cluster reading, and geometry processing to support the core pipeline.
e. Configuration and Environment Management:
• The configs directory provides specific settings for training/validation.
• Environment files ensure that dependencies (e.g., PyTorch, PyG, PyTorch Lightning) are properly installed.
Relationships and Data Flow
• Then, branch the flow into “Data Management” or “Data Loading” (smart/datamodules and smart/datasets), which uses configuration from the configs folder.
– “Layers” (attention, MLP, Fourier embeddings)
– “Modules/Decoders” (agent, map, overall SMART decoder)
– “Tokens” (handling discretized trajectory and map data)
• Link the “Data Loading” step to the “Training Pipeline” that invokes train.py.
• External scripts/utilities as supplementary components that support installation (scripts/install_pyg.sh) and specialized processes (scripts/traj_clstering.py).
Architectural Patterns and Guiding Principles
• The system uses modular design principles – clear separation of data handling, model computation, and evaluation.
• Dependency injection via YAML configs allows ease of scalability and adjustments according to model and dataset variations.
• The discrete token-based representation (a unique pattern) is central to converting continuous map/trajectory data into sequence tokens, which is reflected in the specialized directories (smart/tokens, transforms).