# autonomousvision/transfuser architecture
> Architecture diagram of the GitHub repository autonomousvision/transfuser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/autonomousvision/transfuser
- Repository: https://github.com/autonomousvision/transfuser
## Overview
1. This repository is an end-to-end CARLA autonomous driving stack centered on imitation learning. It combines data generation, model training, local evaluation, and leaderboard submission around CARLA 0.9.10.1.
2. The repo is split into three major runtime boundaries:
- `team_code_autopilot`: a privileged expert policy used to generate training data.
- `team_code_transfuser`: the learned sensor-fusion driving agents and training code.
- `leaderboard` plus `scenario_runner`: the CARLA benchmark/evaluation harness and scenario execution layer.
3. Data generation flows from CARLA into a structured offline dataset. A CARLA server is launched, then `leaderboard/scripts/datagen.sh` runs routes and scenario definitions from `leaderboard/data/training/`, while `team_code_autopilot/autopilot.py` produces expert driving actions. The result is stored as scenario/town/route folders containing multimodal observations and supervision.
4. The dataset schema is multimodal and driving-centric rather than generic ML data. Each route contains RGB, depth, semantic segmentation, LiDAR point clouds, top-down maps, raw 3D labels, and ego measurements. This means the model boundary is not just camera input; it fuses vision, LiDAR, and navigation/measurement state.
5. `team_code_transfuser` is the learning subsystem. `train.py` is the main entry point, with data loading in `data.py`, model variants in `transfuser.py`, `latentTF.py`, `geometric_fusion.py`, `late_fusion.py`, and `point_pillar.py`, and shared utilities in `utils.py` and `config.py`. `submission_agent.py` is the inference-facing wrapper used during evaluation and leaderboard submission.
6. The agent package is designed for checkpoint-driven deployment. Model weights live outside the code in `team_code_transfuser/model_ckpt/`, and the submission agent selects the correct architecture from the saved `args.txt` and checkpoint files. This cleanly separates training artifacts from runtime code.
7. Benchmark execution is handled by `leaderboard`. It contains the evaluator, route indexing/manipulation helpers, statistics collection, and result writing. The local and remote evaluation scripts use the same agent interface but different execution contexts, so evaluation is a distinct boundary from training.
8. `scenario_runner` is the lower-level scenario orchestration engine. It defines scenario configurations, scenario classes, atomic behaviors/criteria, traffic and weather utilities, and actor control abstractions. `leaderboard` depends on this layer to instantiate and score route/scenario runs in CARLA.
9. The repository includes a modified benchmark setup for the “Longest6” benchmark. Relevant assets live under `leaderboard/data/longest6/`, including the route/scenario definitions and evaluation XML. This appears to be a custom benchmark layer built on top of CARLA leaderboard mechanics.
10. `team_code_autopilot` is a separate baseline/expert policy, not the learned agent. It contains navigation planning and simple utility code for map/rendering, and exists primarily to drive data collection and provide a privileged evaluation baseline.
11. `tools/dataset` is the dataset construction and inspection layer. It contains generators for route and scenario files, plus visualization tools for point-based inspection. `tools/result_parser.py` post-processes evaluation outputs and overlays infractions on town maps.
12. Assets and configuration files are part of the architecture, not just static data. Map rasters and OpenDRIVE files under `leaderboard/data/maps` and `leaderboard/data/town_maps_xodr`, plus route/scenario XML/JSON files under `leaderboard/data/training`, define the driving tasks, town boundaries, and scenario coverage used by both data generation and evaluation.
13. Deployment is containerized for leaderboard submission. `leaderboard/scripts/make_docker.sh` and the Dockerfiles package the agent with CARLA-compatible runtime dependencies so the same code can be submitted to the CARLA leaderboard or run locally in a standardized environment.
14. In diagram terms, the core flow is: CARLA simulator plus route/scenario definitions feed the autopilot to create a dataset; the dataset trains TransFuser variants; the trained checkpoint is wrapped by `submission_agent.py`; the agent is evaluated through `leaderboard`/`scenario_runner`; and results can be parsed or submitted via the Dockerized leaderboard path.
Main entry points: [`leaderboard/leaderboard/leaderboard_evaluator.py`](https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/leaderboard/leaderboard_evaluator.py) (Leaderboard), [`team_code_transfuser/train.py`](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/train.py) (Train), [`leaderboard/scripts/datagen.sh`](https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/scripts/datagen.sh) (Datagen), [`tools/dataset`](https://github.com/autonomousvision/transfuser/tree/2022/tools/dataset) (Dataset tools)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_group_data["Data generation"]
node_carla_sim[("CARLA Sim
sim runtime")]
node_autopilot["Autopilot
expert policy
[autopilot.py]"]
node_datagen["Datagen
shell pipeline
[datagen.sh]"]
node_dataset[("Offline dataset
multimodal drive data")]
end
subgraph group_group_learning["Learning"]
node_train["Train
[train.py]"]
node_data_loader["Data loader
dataset interface
[data.py]"]
node_model_zoo["Model zoo
fusion models
[transfuser.py]"]
node_train_utils["Train utils
config + helpers
[config.py]"]
node_ckpt_dir[("Checkpoints
deployment artifacts")]
node_submission_agent["Submission agent
inference wrapper"]
end
subgraph group_group_eval["Evaluation"]
node_leaderboard_eval["Leaderboard
evaluation harness"]
node_scenario_runner["Scenario runner
scenario engine
[scenario_runner.py]"]
node_docker["Docker path
submission packaging
[make_docker.sh]"]
end
subgraph group_group_tools["Tools & assets"]
node_route_defs["Route defs
benchmark assets"]
node_dataset_tools["Dataset tools
prep utilities"]
node_leaderboard_data["Longest6 assets
custom benchmark"]
node_maps["Maps
town assets"]
node_result_parser["Result parser
analysis tool
[result_parser.py]"]
end
node_route_defs -->|"defines runs"| node_datagen
node_carla_sim -->|"executes"| node_datagen
node_datagen -->|"queries policy"| node_autopilot
node_autopilot -->|"writes supervision"| node_dataset
node_dataset_tools -->|"builds assets"| node_route_defs
node_dataset_tools -->|"inspects data"| node_dataset
node_dataset -->|"feeds"| node_data_loader
node_train_utils -->|"configures"| node_train
node_data_loader -->|"supplies batches"| node_train
node_model_zoo -->|"optimizes"| node_train
node_train -->|"produces"| node_ckpt_dir
node_ckpt_dir -->|"loads"| node_submission_agent
node_submission_agent -->|"serves agent"| node_leaderboard_eval
node_leaderboard_eval -->|"delegates scenarios"| node_scenario_runner
node_scenario_runner -->|"controls"| node_carla_sim
node_leaderboard_data -->|"configures benchmark"| node_leaderboard_eval
node_maps -->|"provides map context"| node_leaderboard_eval
node_leaderboard_eval -->|"emits results"| node_result_parser
node_leaderboard_eval -->|"packages for submit"| node_docker
click node_route_defs "https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/training"
click node_autopilot "https://github.com/autonomousvision/transfuser/blob/2022/team_code_autopilot/autopilot.py"
click node_datagen "https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/scripts/datagen.sh"
click node_dataset_tools "https://github.com/autonomousvision/transfuser/tree/2022/tools/dataset"
click node_train "https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/train.py"
click node_data_loader "https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/data.py"
click node_model_zoo "https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/transfuser.py"
click node_train_utils "https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/config.py"
click node_submission_agent "https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/submission_agent.py"
click node_leaderboard_eval "https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/leaderboard/leaderboard_evaluator.py"
click node_scenario_runner "https://github.com/autonomousvision/transfuser/blob/2022/scenario_runner/scenario_runner.py"
click node_leaderboard_data "https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/longest6"
click node_maps "https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/maps"
click node_result_parser "https://github.com/autonomousvision/transfuser/blob/2022/tools/result_parser.py"
click node_docker "https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/scripts/make_docker.sh"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_carla_sim,node_autopilot,node_datagen,node_dataset toneBlue
class node_train,node_data_loader,node_model_zoo,node_train_utils,node_ckpt_dir,node_submission_agent toneAmber
class node_leaderboard_eval,node_scenario_runner,node_docker toneMint
class node_route_defs,node_dataset_tools,node_leaderboard_data,node_maps,node_result_parser toneRose
```
## Components
### Data generation
CARLA route execution, expert driving, and offline dataset creation.
- **CARLA Sim**: sim runtime · CARLA 0.9.10.1 simulation environment providing sensors, towns, and driving execution.
- [Autopilot](https://github.com/autonomousvision/transfuser/blob/2022/team_code_autopilot/autopilot.py): expert policy · Privileged baseline driver used to generate imitation-learning supervision. · `team_code_autopilot/autopilot.py`
- [Datagen](https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/scripts/datagen.sh): shell pipeline · Launches route and scenario runs to collect offline driving data. · `leaderboard/scripts/datagen.sh`
- **Offline dataset**: multimodal drive data · Structured route folders with RGB, depth, segmentation, LiDAR, top-down maps, labels, and measurements.
### Learning
TransFuser training, model variants, and checkpoint-based inference.
- [Train](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/train.py): training entry · Main learning entry point for TransFuser variants. · `team_code_transfuser/train.py`
- [Data loader](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/data.py): dataset interface · Loads multimodal driving samples and supervision for training. · `team_code_transfuser/data.py`
- [Model zoo](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/transfuser.py): fusion models · TransFuser, latentTF, geometric fusion, late fusion, and point pillar architectures. · `team_code_transfuser/transfuser.py`
- [Train utils](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/config.py): config + helpers · Shared configuration and training utilities. · `team_code_transfuser/config.py`
- **Checkpoints**: deployment artifacts · Saved model weights and args metadata used for inference selection.
- [Submission agent](https://github.com/autonomousvision/transfuser/blob/2022/team_code_transfuser/submission_agent.py): inference wrapper · Leaderboard-facing agent that loads the correct architecture and checkpoint. · `team_code_transfuser/submission_agent.py`
### Evaluation
Leaderboard and scenario runner layers for CARLA benchmarking and submission.
- [Leaderboard](https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/leaderboard/leaderboard_evaluator.py): evaluation harness · Route indexing, evaluation, statistics, and result writing for CARLA benchmarks. · `leaderboard/leaderboard/leaderboard_evaluator.py`
- [Scenario runner](https://github.com/autonomousvision/transfuser/blob/2022/scenario_runner/scenario_runner.py): scenario engine · Lower-level scenario orchestration, actor control, and criteria execution. · `scenario_runner/scenario_runner.py`
- [Docker path](https://github.com/autonomousvision/transfuser/blob/2022/leaderboard/scripts/make_docker.sh): submission packaging · Build and run scripts for CARLA-compatible leaderboard submission containers. · `leaderboard/scripts/make_docker.sh`
### Tools & assets
Dataset helpers, maps, benchmark definitions, and result inspection.
- [Route defs](https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/training): benchmark assets · Training routes, scenarios, map rasters, and OpenDRIVE assets defining driving tasks. · `leaderboard/data/training`
- [Dataset tools](https://github.com/autonomousvision/transfuser/tree/2022/tools/dataset): prep utilities · Route and scenario generation plus point-based visualization helpers. · `tools/dataset`
- [Longest6 assets](https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/longest6): custom benchmark · Modified benchmark definitions for the Longest6 evaluation setup. · `leaderboard/data/longest6`
- [Maps](https://github.com/autonomousvision/transfuser/tree/2022/leaderboard/data/maps): town assets · Town raster maps and OpenDRIVE files used by routes and scoring. · `leaderboard/data/maps`
- [Result parser](https://github.com/autonomousvision/transfuser/blob/2022/tools/result_parser.py): analysis tool · Post-processes evaluation outputs and overlays infractions on town maps. · `tools/result_parser.py`
## Connections
- Route defs → Datagen: defines runs · Training routes and scenario files drive data generation runs.
- CARLA Sim → Datagen: executes · CARLA provides the runtime for route execution and sensing.
- Datagen → Autopilot: queries policy · The generator uses the expert autopilot for control actions.
- Autopilot → Offline dataset: writes supervision · Expert actions and observations are recorded into the offline dataset.
- Dataset tools → Route defs: builds assets · Helpers generate and manipulate route and scenario files.
- Dataset tools → Offline dataset: inspects data · Visualization tools support point-based dataset inspection.
- Offline dataset → Data loader: feeds · Multimodal driving samples are loaded for training.
- Train utils → Train: configures · Shared settings and helpers support the training loop.
- Data loader → Train: supplies batches · Training consumes loaded samples and targets.
- Model zoo → Train: optimizes · Training selects and fits one of the fusion architectures.
- Train → Checkpoints: produces · Training outputs checkpoints and saved args metadata.
- Checkpoints → Submission agent: loads · Inference wrapper restores the correct checkpointed architecture.
- Submission agent → Leaderboard: serves agent · The agent is executed by the leaderboard evaluator.
- Leaderboard → Scenario runner: delegates scenarios · Leaderboard relies on scenario runner for scenario execution and scoring.
- Scenario runner → CARLA Sim: controls · Scenario runner instantiates actors and controls CARLA runs.
- Longest6 assets → Leaderboard: configures benchmark · Longest6 route and scenario assets define the benchmark setup.
- Maps → Leaderboard: provides map context · Town maps and XODR files support route execution and scoring.
- Leaderboard → Result parser: emits results · Evaluation outputs are post-processed and visualized.
- Leaderboard → Docker path: packages for submit · Docker scripts standardize local and leaderboard execution.