How thu-si/spatial-ttt works
1. Repository shape
- This repo is split into two main runtime subsystems plus supporting assets:
- qwen-vl-finetune/: training and model code for Spatial-TTT.
- evaluation/spatial/: benchmark/evaluation harness built around lmms-eval.
- The top-level README.md and assets/ are documentation only; they describe the method and show the pipeline.
2. Core product architecture
- The system implements Spatial-TTT, a Qwen-VL-based visual model with test-time training / fast-weight updates for long video spatial reasoning.
- Architecturally, it is not a generic app; it is a model training + benchmark evaluation platform centered on streaming video inputs and spatial QA tasks.
3. Training pipeline boundary
- The training entrypoint lives in qwen-vl-finetune/spatial_ttt_train.sh, which launches distributed training.
- Inputs come from the configured Spatial-TTT-Data-97k dataset: annotations and media paths are set in qwenvl/data/__init__.py.
- The training flow is: dataset metadata and video/image files → data processor → trainer → Spatial-TTT model → checkpoints in the configured output directory.
4. Model stack
- The model code is concentrated in qwen-vl-finetune/models/.
- Key custom components include:
- spatial_ttt.py: the Spatial-TTT model variant.
- causal_swa_lact.py and causal_swa_lact_streaming_chunked.py: hybrid TTT / sliding-window attention layers.
- ttt_operation.py and ttt_operation_fused_kernel.py: fast-weight / TTT operations.
- This layer is where the repo differentiates from standard Qwen-VL finetuning: the model is modified to support streaming chunk updates and spatial memory accumulation.
5. Performance and kernel boundary
- The model stack depends on custom Triton kernels under models/lact_triton_kernels/.
- These kernels implement fused primitives for normalization, SwiGLU, matmul, and update steps, which are performance-critical for the TTT path.
- This is an important architecture boundary: high-level training code delegates compute-heavy operations to specialized GPU kernels.
6. Data processing and ingestion
- qwenvl/data/ and qwenvl/data/data_processor.py adapt raw multimodal samples into model-ready batches.
- The repo also includes supporting utilities for packing, checking, and preprocessing data under tools/.
- The dataset flow is media-heavy: videos/images are loaded, chunked, and arranged to match the model’s long-context streaming assumptions.
7. Training runtime and distributed orchestration
- Training uses DeepSpeed configuration files in qwen-vl-finetune/scripts/ such as ZeRO stage configs.
- The trainer code under qwenvl/train/ wraps argument parsing, the training loop, and Spatial-TTT-specific training entrypoints.
- The main runtime dependencies called out in the README are PyTorch, Transformers, FlashAttention, Accelerate, PEFT, Triton, TorchCodec, which materially shape the execution model and GPU requirements.
8. Evaluation subsystem
- Evaluation is isolated under evaluation/spatial/, with a dedicated shell entrypoint in evaluation/spatial/scripts/eval_spatial_ttt_2b.sh.
- This side of the repo uses a forked lmms-eval package in evaluation/spatial/lmms-eval/ to run spatial benchmarks and aggregate results.
- The checkpoint flow is: trained model checkpoint → benchmark runner → task-specific adapters and metrics → summarized benchmark outputs.
9. Benchmark/task architecture
- lmms-eval organizes evaluation around:
- API abstractions in lmms_eval/api/ for models, tasks, metrics, samplers, filters, and registry.
- Task definitions in lmms_eval/tasks/, mostly YAML-based benchmark specs with per-benchmark utilities.
- Model wrappers in lmms_eval/models/, including simple adapters for Qwen/VLLM/OpenAI-compatible backends.
- This separation lets the same evaluator run many spatial/video benchmarks without changing the model implementation.
10. Post-processing, filtering, and judgment
- Evaluation supports answer extraction, selection, transformation, and decontamination via lmms_eval/filters/.
- There is also an LLM-as-judge subsystem in lmms_eval/llm_judge/ with provider backends for OpenAI/Azure OpenAI, async variants, and Sglang launching.
- That means evaluation is not only exact-match metrics; some tasks can use model-based scoring or judgment pipelines.
11. Benchmark coverage
- The task set is heavily oriented toward video and spatial understanding:
- mvbench, videomme, videommmu, mindcube, egoschema, longvideobench, sitebench, vsibench, cambrians_vsr/vsc, and others.
- Each benchmark has its own YAML task files and utility code, which makes task configuration a first-class architectural artifact.
- The repo is therefore organized around benchmark-specific adapters, not a single monolithic eval runner.
12. Auxiliary dataset and benchmark tooling
- The tools/ directory under evaluation/spatial/lmms-eval and top-level tools include dataset preparation, video/image HF dataset creation, regression scripts, and benchmark-specific utilities.
- tools/live_bench/ is a separate benchmark generation pipeline with its own API, data generator, website loader, screenshotter, and script notebooks.
- These tools form a secondary data-production boundary: they support constructing or refining evaluation datasets rather than model training itself.
13. External dependencies and service boundaries
- The repo depends on external services and assets in two main ways:
- Model/data hosting on Hugging Face for pretrained checkpoints and datasets.
- LLM provider APIs for judge workflows and benchmark generation in llm_judge and tools/live_bench.
- That makes the system a mix of local GPU training/inference plus optional cloud API calls for evaluation and data generation.
14. What an architecture diagram should show
- The most accurate high-level diagram would have:
- Data sources: Spatial-TTT datasets, benchmark YAMLs, videos/images.
- Training path: data config → processor → Qwen-VL/Spatial-TTT model → distributed trainer/DeepSpeed → checkpoints.
- Evaluation path: checkpoint → lmms-eval model adapter → benchmark tasks/filters/metrics → result summaries.
- Supporting compute: custom Triton kernels and TTT operations.
- Optional external services: OpenAI/Azure/Sglang judge backends and HF-hosted artifacts.
Main entry points: evaluator.py lmms-eval core · train_spatial_ttt.py Trainer · spatial_ttt.py Spatial-TTT · causal_swa_lact.py TTT layers
Components 22
Spatial-TTT Repo
Two runtime subsystems: training/model stack and evaluation harness.
Training Stack
Qwen-VL finetuning and Spatial-TTT model runtime.
- Train script · Distributed Spatial-TTT training launcher.
qwen-vl-finetune/spatial_ttt_train.sh - Data config · Dataset and media path configuration for Spatial-TTT-Data-97k.
qwen-vl-finetune/qwenvl/data/__init__.py - Data processor · Transforms multimodal samples into model-ready batches.
qwen-vl-finetune/qwenvl/data/data_processor.py - Trainer · Argument handling and Spatial-TTT training loop.
qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py - Qwen train · Shared training utilities and entrypoints.
qwen-vl-finetune/qwenvl/train/train_qwen.py - Spatial-TTT · Qwen-VL variant with test-time training and streaming memory updates.
qwen-vl-finetune/models/spatial_ttt.py - TTT layers · Hybrid causal sliding-window and fast-weight update layers.
qwen-vl-finetune/models/causal_swa_lact.py - Streaming chunked · Chunked streaming variant for long video inputs.
qwen-vl-finetune/models/causal_swa_lact_streaming_chunked.py - TTT ops · Core test-time training and update operations.
qwen-vl-finetune/models/ttt_operation.py - Triton kernels · Fused GPU kernels for normalization, matmul, SwiGLU, and updates.
qwen-vl-finetune/models/lact_triton_kernels - DeepSpeed configs · ZeRO and offload configurations for distributed training.
qwen-vl-finetune/scripts/zero3.json
Evaluation Stack
lmms-eval-based benchmark execution and scoring.
- Eval script · Spatial-TTT benchmark evaluation launcher.
evaluation/spatial/scripts/eval_spatial_ttt_2b.sh - lmms-eval core · Evaluator, API abstractions, caching, logging, and protocol.
evaluation/spatial/lmms-eval/lmms_eval/evaluator.py - Model adapters · Wrappers for Spatial-TTT, Qwen, VLLM, and other model backends.
evaluation/spatial/lmms-eval/lmms_eval/models/simple/spatial_ttt.py - Task specs · YAML-defined benchmark tasks and per-benchmark utilities.
evaluation/spatial/lmms-eval/lmms_eval/tasks - Filters · Answer extraction, selection, transformation, and decontamination.
evaluation/spatial/lmms-eval/lmms_eval/filters - LLM judge · Model-based scoring and external provider-backed judgment workflows.
evaluation/spatial/lmms-eval/lmms_eval/llm_judge - Eval tools · Dataset prep, regression, and benchmark support utilities.
evaluation/spatial/lmms-eval/tools - Live bench · Benchmark generation pipeline with website loading, screenshots, and QA creation.
evaluation/spatial/lmms-eval/tools/live_bench/live_bench - Misc scripts · Result summarization and auxiliary evaluation helpers.
evaluation/spatial/misc/summarize_vsibench_results.py
Connections 21
- README → Train script: describes · Documentation explains the training workflow.
- README → Eval script: describes · Documentation explains the evaluation workflow.
- Train script → Data config: loads · Training starts from dataset and media configuration.
- Data config → Data processor: feeds · Configured samples are converted into batches.
- Data processor → Trainer: batches · Processed multimodal batches are consumed by the trainer.
- Trainer → Spatial-TTT: optimizes · Trainer updates the Spatial-TTT model.
- Qwen train → Trainer: extends · Shared Qwen training runtime underpins the Spatial-TTT loop.
- Spatial-TTT → TTT layers: uses · Spatial-TTT is built from custom attention/memory layers.
- TTT layers → Streaming chunked: variants · Streaming chunked logic supports long video reasoning.
- TTT layers → TTT ops: calls · Layer behavior relies on fast-weight operations.
- TTT ops → Triton kernels: accelerates · Compute-heavy updates are offloaded to Triton kernels.
- Trainer → DeepSpeed configs: orchestrates · Distributed training uses DeepSpeed configs.
- Eval script → lmms-eval core: runs · Evaluation script invokes the lmms-eval engine.
- lmms-eval core → Model adapters: loads · Evaluator dispatches through model adapters.
- lmms-eval core → Task specs: executes · Benchmark runs are driven by task YAMLs and utilities.
- lmms-eval core → Filters: postprocesses · Outputs are normalized through filter pipelines.
- lmms-eval core → LLM judge: scores · Optional LLM-as-judge flows provide rubric-based scoring.
- Task specs → Eval tools: depends on · Task execution relies on shared benchmark utilities.
- Eval tools → Live bench: includes · Tooling includes benchmark generation and website-based data production.
- Misc scripts → Task specs: summarizes · Auxiliary scripts aggregate benchmark results.
- Spatial-TTT → lmms-eval core: evaluated by · Trained checkpoints are consumed by the evaluation stack.