How autonomousvision/transfuser works
1. This repository is an end-to-end CARLA autonomous driving stack centered on imitation learning. It combines data generation, model training, local evaluation, and leaderboard submission around CARLA 0.9.10.1.
2. The repo is split into three major runtime boundaries:
- team_code_autopilot: a privileged expert policy used to generate training data.
- team_code_transfuser: the learned sensor-fusion driving agents and training code.
- leaderboard plus scenario_runner: the CARLA benchmark/evaluation harness and scenario execution layer.
3. Data generation flows from CARLA into a structured offline dataset. A CARLA server is launched, then leaderboard/scripts/datagen.sh runs routes and scenario definitions from leaderboard/data/training/, while team_code_autopilot/autopilot.py produces expert driving actions. The result is stored as scenario/town/route folders containing multimodal observations and supervision.
4. The dataset schema is multimodal and driving-centric rather than generic ML data. Each route contains RGB, depth, semantic segmentation, LiDAR point clouds, top-down maps, raw 3D labels, and ego measurements. This means the model boundary is not just camera input; it fuses vision, LiDAR, and navigation/measurement state.
5. team_code_transfuser is the learning subsystem. train.py is the main entry point, with data loading in data.py, model variants in transfuser.py, latentTF.py, geometric_fusion.py, late_fusion.py, and point_pillar.py, and shared utilities in utils.py and config.py. submission_agent.py is the inference-facing wrapper used during evaluation and leaderboard submission.
6. The agent package is designed for checkpoint-driven deployment. Model weights live outside the code in team_code_transfuser/model_ckpt/, and the submission agent selects the correct architecture from the saved args.txt and checkpoint files. This cleanly separates training artifacts from runtime code.
7. Benchmark execution is handled by leaderboard. It contains the evaluator, route indexing/manipulation helpers, statistics collection, and result writing. The local and remote evaluation scripts use the same agent interface but different execution contexts, so evaluation is a distinct boundary from training.
8. scenario_runner is the lower-level scenario orchestration engine. It defines scenario configurations, scenario classes, atomic behaviors/criteria, traffic and weather utilities, and actor control abstractions. leaderboard depends on this layer to instantiate and score route/scenario runs in CARLA.
9. The repository includes a modified benchmark setup for the “Longest6” benchmark. Relevant assets live under leaderboard/data/longest6/, including the route/scenario definitions and evaluation XML. This appears to be a custom benchmark layer built on top of CARLA leaderboard mechanics.
10. team_code_autopilot is a separate baseline/expert policy, not the learned agent. It contains navigation planning and simple utility code for map/rendering, and exists primarily to drive data collection and provide a privileged evaluation baseline.
11. tools/dataset is the dataset construction and inspection layer. It contains generators for route and scenario files, plus visualization tools for point-based inspection. tools/result_parser.py post-processes evaluation outputs and overlays infractions on town maps.
12. Assets and configuration files are part of the architecture, not just static data. Map rasters and OpenDRIVE files under leaderboard/data/maps and leaderboard/data/town_maps_xodr, plus route/scenario XML/JSON files under leaderboard/data/training, define the driving tasks, town boundaries, and scenario coverage used by both data generation and evaluation.
13. Deployment is containerized for leaderboard submission. leaderboard/scripts/make_docker.sh and the Dockerfiles package the agent with CARLA-compatible runtime dependencies so the same code can be submitted to the CARLA leaderboard or run locally in a standardized environment.
14. In diagram terms, the core flow is: CARLA simulator plus route/scenario definitions feed the autopilot to create a dataset; the dataset trains TransFuser variants; the trained checkpoint is wrapped by submission_agent.py; the agent is evaluated through leaderboard/scenario_runner; and results can be parsed or submitted via the Dockerized leaderboard path.
Main entry points: leaderboard_evaluator.py Leaderboard · train.py Train · datagen.sh Datagen · dataset Dataset tools
Components 18
Data generation
CARLA route execution, expert driving, and offline dataset creation.
- CARLA Sim · CARLA 0.9.10.1 simulation environment providing sensors, towns, and driving execution.
- Autopilot · Privileged baseline driver used to generate imitation-learning supervision.
team_code_autopilot/autopilot.py - Datagen · Launches route and scenario runs to collect offline driving data.
leaderboard/scripts/datagen.sh - Offline dataset · Structured route folders with RGB, depth, segmentation, LiDAR, top-down maps, labels, and measurements.
Learning
TransFuser training, model variants, and checkpoint-based inference.
- Train · Main learning entry point for TransFuser variants.
team_code_transfuser/train.py - Data loader · Loads multimodal driving samples and supervision for training.
team_code_transfuser/data.py - Model zoo · TransFuser, latentTF, geometric fusion, late fusion, and point pillar architectures.
team_code_transfuser/transfuser.py - Train utils · Shared configuration and training utilities.
team_code_transfuser/config.py - Checkpoints · Saved model weights and args metadata used for inference selection.
- Submission agent · Leaderboard-facing agent that loads the correct architecture and checkpoint.
team_code_transfuser/submission_agent.py
Evaluation
Leaderboard and scenario runner layers for CARLA benchmarking and submission.
- Leaderboard · Route indexing, evaluation, statistics, and result writing for CARLA benchmarks.
leaderboard/leaderboard/leaderboard_evaluator.py - Scenario runner · Lower-level scenario orchestration, actor control, and criteria execution.
scenario_runner/scenario_runner.py - Docker path · Build and run scripts for CARLA-compatible leaderboard submission containers.
leaderboard/scripts/make_docker.sh
Tools & assets
Dataset helpers, maps, benchmark definitions, and result inspection.
- Route defs · Training routes, scenarios, map rasters, and OpenDRIVE assets defining driving tasks.
leaderboard/data/training - Dataset tools · Route and scenario generation plus point-based visualization helpers.
tools/dataset - Longest6 assets · Modified benchmark definitions for the Longest6 evaluation setup.
leaderboard/data/longest6 - Maps · Town raster maps and OpenDRIVE files used by routes and scoring.
leaderboard/data/maps - Result parser · Post-processes evaluation outputs and overlays infractions on town maps.
tools/result_parser.py
Connections 19
- Route defs → Datagen: defines runs · Training routes and scenario files drive data generation runs.
- CARLA Sim → Datagen: executes · CARLA provides the runtime for route execution and sensing.
- Datagen → Autopilot: queries policy · The generator uses the expert autopilot for control actions.
- Autopilot → Offline dataset: writes supervision · Expert actions and observations are recorded into the offline dataset.
- Dataset tools → Route defs: builds assets · Helpers generate and manipulate route and scenario files.
- Dataset tools → Offline dataset: inspects data · Visualization tools support point-based dataset inspection.
- Offline dataset → Data loader: feeds · Multimodal driving samples are loaded for training.
- Train utils → Train: configures · Shared settings and helpers support the training loop.
- Data loader → Train: supplies batches · Training consumes loaded samples and targets.
- Model zoo → Train: optimizes · Training selects and fits one of the fusion architectures.
- Train → Checkpoints: produces · Training outputs checkpoints and saved args metadata.
- Checkpoints → Submission agent: loads · Inference wrapper restores the correct checkpointed architecture.
- Submission agent → Leaderboard: serves agent · The agent is executed by the leaderboard evaluator.
- Leaderboard → Scenario runner: delegates scenarios · Leaderboard relies on scenario runner for scenario execution and scoring.
- Scenario runner → CARLA Sim: controls · Scenario runner instantiates actors and controls CARLA runs.
- Longest6 assets → Leaderboard: configures benchmark · Longest6 route and scenario assets define the benchmark setup.
- Maps → Leaderboard: provides map context · Town maps and XODR files support route execution and scoring.
- Leaderboard → Result parser: emits results · Evaluation outputs are post-processed and visualized.
- Leaderboard → Docker path: packages for submit · Docker scripts standardize local and leaderboard execution.