# thu-si/spatial-ttt architecture

> Architecture diagram of the GitHub repository thu-si/spatial-ttt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thu-si/spatial-ttt
- Repository: https://github.com/thu-si/spatial-ttt

## Overview

1. **Repository shape**
   - This repo is split into two main runtime subsystems plus supporting assets:
     - **`qwen-vl-finetune/`**: training and model code for Spatial-TTT.
     - **`evaluation/spatial/`**: benchmark/evaluation harness built around `lmms-eval`.
   - The top-level `README.md` and `assets/` are documentation only; they describe the method and show the pipeline.

2. **Core product architecture**
   - The system implements **Spatial-TTT**, a Qwen-VL-based visual model with **test-time training / fast-weight updates** for long video spatial reasoning.
   - Architecturally, it is not a generic app; it is a **model training + benchmark evaluation platform** centered on streaming video inputs and spatial QA tasks.

3. **Training pipeline boundary**
   - The training entrypoint lives in **`qwen-vl-finetune/spatial_ttt_train.sh`**, which launches distributed training.
   - Inputs come from the configured **Spatial-TTT-Data-97k** dataset: annotations and media paths are set in **`qwenvl/data/__init__.py`**.
   - The training flow is: dataset metadata and video/image files → data processor → trainer → Spatial-TTT model → checkpoints in the configured output directory.

4. **Model stack**
   - The model code is concentrated in **`qwen-vl-finetune/models/`**.
   - Key custom components include:
     - **`spatial_ttt.py`**: the Spatial-TTT model variant.
     - **`causal_swa_lact.py`** and **`causal_swa_lact_streaming_chunked.py`**: hybrid TTT / sliding-window attention layers.
     - **`ttt_operation.py`** and **`ttt_operation_fused_kernel.py`**: fast-weight / TTT operations.
   - This layer is where the repo differentiates from standard Qwen-VL finetuning: the model is modified to support streaming chunk updates and spatial memory accumulation.

5. **Performance and kernel boundary**
   - The model stack depends on **custom Triton kernels** under **`models/lact_triton_kernels/`**.
   - These kernels implement fused primitives for normalization, SwiGLU, matmul, and update steps, which are performance-critical for the TTT path.
   - This is an important architecture boundary: high-level training code delegates compute-heavy operations to specialized GPU kernels.

6. **Data processing and ingestion**
   - **`qwenvl/data/`** and **`qwenvl/data/data_processor.py`** adapt raw multimodal samples into model-ready batches.
   - The repo also includes supporting utilities for packing, checking, and preprocessing data under **`tools/`**.
   - The dataset flow is media-heavy: videos/images are loaded, chunked, and arranged to match the model’s long-context streaming assumptions.

7. **Training runtime and distributed orchestration**
   - Training uses **DeepSpeed** configuration files in **`qwen-vl-finetune/scripts/`** such as ZeRO stage configs.
   - The trainer code under **`qwenvl/train/`** wraps argument parsing, the training loop, and Spatial-TTT-specific training entrypoints.
   - The main runtime dependencies called out in the README are **PyTorch, Transformers, FlashAttention, Accelerate, PEFT, Triton, TorchCodec**, which materially shape the execution model and GPU requirements.

8. **Evaluation subsystem**
   - Evaluation is isolated under **`evaluation/spatial/`**, with a dedicated shell entrypoint in **`evaluation/spatial/scripts/eval_spatial_ttt_2b.sh`**.
   - This side of the repo uses a forked **`lmms-eval`** package in **`evaluation/spatial/lmms-eval/`** to run spatial benchmarks and aggregate results.
   - The checkpoint flow is: trained model checkpoint → benchmark runner → task-specific adapters and metrics → summarized benchmark outputs.

9. **Benchmark/task architecture**
   - `lmms-eval` organizes evaluation around:
     - **API abstractions** in `lmms_eval/api/` for models, tasks, metrics, samplers, filters, and registry.
     - **Task definitions** in `lmms_eval/tasks/`, mostly YAML-based benchmark specs with per-benchmark utilities.
     - **Model wrappers** in `lmms_eval/models/`, including simple adapters for Qwen/VLLM/OpenAI-compatible backends.
   - This separation lets the same evaluator run many spatial/video benchmarks without changing the model implementation.

10. **Post-processing, filtering, and judgment**
    - Evaluation supports answer extraction, selection, transformation, and decontamination via **`lmms_eval/filters/`**.
    - There is also an **LLM-as-judge** subsystem in **`lmms_eval/llm_judge/`** with provider backends for OpenAI/Azure OpenAI, async variants, and Sglang launching.
    - That means evaluation is not only exact-match metrics; some tasks can use model-based scoring or judgment pipelines.

11. **Benchmark coverage**
    - The task set is heavily oriented toward **video and spatial understanding**:
      - `mvbench`, `videomme`, `videommmu`, `mindcube`, `egoschema`, `longvideobench`, `sitebench`, `vsibench`, `cambrians_vsr/vsc`, and others.
    - Each benchmark has its own YAML task files and utility code, which makes task configuration a first-class architectural artifact.
    - The repo is therefore organized around **benchmark-specific adapters**, not a single monolithic eval runner.

12. **Auxiliary dataset and benchmark tooling**
    - The **`tools/`** directory under `evaluation/spatial/lmms-eval` and top-level tools include dataset preparation, video/image HF dataset creation, regression scripts, and benchmark-specific utilities.
    - `tools/live_bench/` is a separate benchmark generation pipeline with its own API, data generator, website loader, screenshotter, and script notebooks.
    - These tools form a secondary data-production boundary: they support constructing or refining evaluation datasets rather than model training itself.

13. **External dependencies and service boundaries**
    - The repo depends on external services and assets in two main ways:
      - **Model/data hosting** on Hugging Face for pretrained checkpoints and datasets.
      - **LLM provider APIs** for judge workflows and benchmark generation in `llm_judge` and `tools/live_bench`.
    - That makes the system a mix of local GPU training/inference plus optional cloud API calls for evaluation and data generation.

14. **What an architecture diagram should show**
    - The most accurate high-level diagram would have:
      - **Data sources**: Spatial-TTT datasets, benchmark YAMLs, videos/images.
      - **Training path**: data config → processor → Qwen-VL/Spatial-TTT model → distributed trainer/DeepSpeed → checkpoints.
      - **Evaluation path**: checkpoint → lmms-eval model adapter → benchmark tasks/filters/metrics → result summaries.
      - **Supporting compute**: custom Triton kernels and TTT operations.
      - **Optional external services**: OpenAI/Azure/Sglang judge backends and HF-hosted artifacts.

Main entry points: [`evaluation/spatial/lmms-eval/lmms_eval/evaluator.py`](https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/lmms-eval/lmms_eval/evaluator.py) (lmms-eval core), [`qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py`](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py) (Trainer), [`qwen-vl-finetune/models/spatial_ttt.py`](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/spatial_ttt.py) (Spatial-TTT), [`qwen-vl-finetune/models/causal_swa_lact.py`](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/causal_swa_lact.py) (TTT layers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_repo["Spatial-TTT Repo"]
  node_readme["README<br/>docs<br/>[README.md]"]
  node_assets["Assets"]
end

subgraph group_training["Training Stack"]
  node_train_entry["Train script<br/>entrypoint"]
  node_data_config["Data config<br/>dataset wiring<br/>[__init__.py]"]
  node_data_processor["Data processor<br/>batch prep<br/>[data_processor.py]"]
  node_trainer["Trainer<br/>train loop"]
  node_train_core["Qwen train<br/>training runtime<br/>[train_qwen.py]"]
  node_model["Spatial-TTT<br/>model<br/>[spatial_ttt.py]"]
  node_attn_layers["TTT layers<br/>attention/memory<br/>[causal_swa_lact.py]"]
  node_stream_layers["Streaming chunked<br/>attention/memory"]
  node_ttt_ops["TTT ops<br/>fast-weight ops<br/>[ttt_operation.py]"]
  node_kernels["Triton kernels<br/>gpu kernels"]
  node_deepspeed["DeepSpeed configs<br/>distributed config<br/>[zero3.json]"]
end

subgraph group_evaluation["Evaluation Stack"]
  node_eval_entry["Eval script<br/>entrypoint"]
  node_lmms_core["lmms-eval core<br/>eval framework<br/>[evaluator.py]"]
  node_model_adapters["Model adapters<br/>backend adapters<br/>[spatial_ttt.py]"]
  node_task_specs["Task specs<br/>benchmarks"]
  node_filters["Filters<br/>post-process"]
  node_judge["LLM judge<br/>judging"]
  node_eval_tools["Eval tools<br/>tooling"]
  node_live_bench["Live bench<br/>data generation"]
  node_misc["Misc scripts<br/>supporting scripts"]
end

subgraph group_support["Supporting Assets"]
end

node_readme -->|"describes"| node_train_entry
node_readme -->|"describes"| node_eval_entry
node_train_entry -->|"loads"| node_data_config
node_data_config -->|"feeds"| node_data_processor
node_data_processor -->|"batches"| node_trainer
node_trainer -->|"optimizes"| node_model
node_train_core -->|"extends"| node_trainer
node_model -->|"uses"| node_attn_layers
node_attn_layers -->|"variants"| node_stream_layers
node_attn_layers -->|"calls"| node_ttt_ops
node_ttt_ops -->|"accelerates"| node_kernels
node_trainer -->|"orchestrates"| node_deepspeed
node_eval_entry -->|"runs"| node_lmms_core
node_lmms_core -->|"loads"| node_model_adapters
node_lmms_core -->|"executes"| node_task_specs
node_lmms_core -->|"postprocesses"| node_filters
node_lmms_core -->|"scores"| node_judge
node_task_specs -->|"depends on"| node_eval_tools
node_eval_tools -->|"includes"| node_live_bench
node_misc -->|"summarizes"| node_task_specs
node_model -.->|"evaluated by"| node_lmms_core

click node_readme "https://github.com/thu-si/spatial-ttt/blob/main/README.md"
click node_assets "https://github.com/thu-si/spatial-ttt/tree/main/assets"
click node_train_entry "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/spatial_ttt_train.sh"
click node_data_config "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/data/__init__.py"
click node_data_processor "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/data/data_processor.py"
click node_trainer "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py"
click node_train_core "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/train/train_qwen.py"
click node_model "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/spatial_ttt.py"
click node_attn_layers "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/causal_swa_lact.py"
click node_stream_layers "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/causal_swa_lact_streaming_chunked.py"
click node_ttt_ops "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/ttt_operation.py"
click node_kernels "https://github.com/thu-si/spatial-ttt/tree/main/qwen-vl-finetune/models/lact_triton_kernels"
click node_deepspeed "https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/scripts/zero3.json"
click node_eval_entry "https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/scripts/eval_spatial_ttt_2b.sh"
click node_lmms_core "https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/lmms-eval/lmms_eval/evaluator.py"
click node_model_adapters "https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/lmms-eval/lmms_eval/models/simple/spatial_ttt.py"
click node_task_specs "https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/tasks"
click node_filters "https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/filters"
click node_judge "https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/llm_judge"
click node_eval_tools "https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/tools"
click node_live_bench "https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/tools/live_bench/live_bench"
click node_misc "https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/misc/summarize_vsibench_results.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_readme,node_assets toneBlue
class node_train_entry,node_data_config,node_data_processor,node_trainer,node_train_core,node_model,node_attn_layers,node_stream_layers,node_ttt_ops,node_kernels,node_deepspeed toneAmber
class node_eval_entry,node_lmms_core,node_model_adapters,node_task_specs,node_filters,node_judge,node_eval_tools,node_live_bench,node_misc toneMint
```

## Components

### Spatial-TTT Repo

Two runtime subsystems: training/model stack and evaluation harness.

- [README](https://github.com/thu-si/spatial-ttt/blob/main/README.md): docs · Project overview and method description. · `README.md`
- [Assets](https://github.com/thu-si/spatial-ttt/tree/main/assets): docs/assets · Figures and illustrative pipeline material. · `assets`

### Training Stack

Qwen-VL finetuning and Spatial-TTT model runtime.

- [Train script](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/spatial_ttt_train.sh): entrypoint · Distributed Spatial-TTT training launcher. · `qwen-vl-finetune/spatial_ttt_train.sh`
- [Data config](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/data/__init__.py): dataset wiring · Dataset and media path configuration for Spatial-TTT-Data-97k. · `qwen-vl-finetune/qwenvl/data/__init__.py`
- [Data processor](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/data/data_processor.py): batch prep · Transforms multimodal samples into model-ready batches. · `qwen-vl-finetune/qwenvl/data/data_processor.py`
- [Trainer](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py): train loop · Argument handling and Spatial-TTT training loop. · `qwen-vl-finetune/qwenvl/train/train_spatial_ttt.py`
- [Qwen train](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/qwenvl/train/train_qwen.py): training runtime · Shared training utilities and entrypoints. · `qwen-vl-finetune/qwenvl/train/train_qwen.py`
- [Spatial-TTT](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/spatial_ttt.py): model · Qwen-VL variant with test-time training and streaming memory updates. · `qwen-vl-finetune/models/spatial_ttt.py`
- [TTT layers](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/causal_swa_lact.py): attention/memory · Hybrid causal sliding-window and fast-weight update layers. · `qwen-vl-finetune/models/causal_swa_lact.py`
- [Streaming chunked](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/causal_swa_lact_streaming_chunked.py): attention/memory · Chunked streaming variant for long video inputs. · `qwen-vl-finetune/models/causal_swa_lact_streaming_chunked.py`
- [TTT ops](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/models/ttt_operation.py): fast-weight ops · Core test-time training and update operations. · `qwen-vl-finetune/models/ttt_operation.py`
- [Triton kernels](https://github.com/thu-si/spatial-ttt/tree/main/qwen-vl-finetune/models/lact_triton_kernels): gpu kernels · Fused GPU kernels for normalization, matmul, SwiGLU, and updates. · `qwen-vl-finetune/models/lact_triton_kernels`
- [DeepSpeed configs](https://github.com/thu-si/spatial-ttt/blob/main/qwen-vl-finetune/scripts/zero3.json): distributed config · ZeRO and offload configurations for distributed training. · `qwen-vl-finetune/scripts/zero3.json`

### Evaluation Stack

lmms-eval-based benchmark execution and scoring.

- [Eval script](https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/scripts/eval_spatial_ttt_2b.sh): entrypoint · Spatial-TTT benchmark evaluation launcher. · `evaluation/spatial/scripts/eval_spatial_ttt_2b.sh`
- [lmms-eval core](https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/lmms-eval/lmms_eval/evaluator.py): eval framework · Evaluator, API abstractions, caching, logging, and protocol. · `evaluation/spatial/lmms-eval/lmms_eval/evaluator.py`
- [Model adapters](https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/lmms-eval/lmms_eval/models/simple/spatial_ttt.py): backend adapters · Wrappers for Spatial-TTT, Qwen, VLLM, and other model backends. · `evaluation/spatial/lmms-eval/lmms_eval/models/simple/spatial_ttt.py`
- [Task specs](https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/tasks): benchmarks · YAML-defined benchmark tasks and per-benchmark utilities. · `evaluation/spatial/lmms-eval/lmms_eval/tasks`
- [Filters](https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/filters): post-process · Answer extraction, selection, transformation, and decontamination. · `evaluation/spatial/lmms-eval/lmms_eval/filters`
- [LLM judge](https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/lmms_eval/llm_judge): judging · Model-based scoring and external provider-backed judgment workflows. · `evaluation/spatial/lmms-eval/lmms_eval/llm_judge`
- [Eval tools](https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/tools): tooling · Dataset prep, regression, and benchmark support utilities. · `evaluation/spatial/lmms-eval/tools`
- [Live bench](https://github.com/thu-si/spatial-ttt/tree/main/evaluation/spatial/lmms-eval/tools/live_bench/live_bench): data generation · Benchmark generation pipeline with website loading, screenshots, and QA creation. · `evaluation/spatial/lmms-eval/tools/live_bench/live_bench`
- [Misc scripts](https://github.com/thu-si/spatial-ttt/blob/main/evaluation/spatial/misc/summarize_vsibench_results.py): supporting scripts · Result summarization and auxiliary evaluation helpers. · `evaluation/spatial/misc/summarize_vsibench_results.py`

## Connections

- README → Train script: describes · Documentation explains the training workflow.
- README → Eval script: describes · Documentation explains the evaluation workflow.
- Train script → Data config: loads · Training starts from dataset and media configuration.
- Data config → Data processor: feeds · Configured samples are converted into batches.
- Data processor → Trainer: batches · Processed multimodal batches are consumed by the trainer.
- Trainer → Spatial-TTT: optimizes · Trainer updates the Spatial-TTT model.
- Qwen train → Trainer: extends · Shared Qwen training runtime underpins the Spatial-TTT loop.
- Spatial-TTT → TTT layers: uses · Spatial-TTT is built from custom attention/memory layers.
- TTT layers → Streaming chunked: variants · Streaming chunked logic supports long video reasoning.
- TTT layers → TTT ops: calls · Layer behavior relies on fast-weight operations.
- TTT ops → Triton kernels: accelerates · Compute-heavy updates are offloaded to Triton kernels.
- Trainer → DeepSpeed configs: orchestrates · Distributed training uses DeepSpeed configs.
- Eval script → lmms-eval core: runs · Evaluation script invokes the lmms-eval engine.
- lmms-eval core → Model adapters: loads · Evaluator dispatches through model adapters.
- lmms-eval core → Task specs: executes · Benchmark runs are driven by task YAMLs and utilities.
- lmms-eval core → Filters: postprocesses · Outputs are normalized through filter pipelines.
- lmms-eval core → LLM judge: scores · Optional LLM-as-judge flows provide rubric-based scoring.
- Task specs → Eval tools: depends on · Task execution relies on shared benchmark utilities.
- Eval tools → Live bench: includes · Tooling includes benchmark generation and website-based data production.
- Misc scripts → Task specs: summarizes · Auxiliary scripts aggregate benchmark results.
- Spatial-TTT → lmms-eval core: evaluated by · Trained checkpoints are consumed by the evaluation stack.
