How wayveai/mile works
Project Type and Purpose
• The project is a research implementation in the field of model-based imitation learning for urban driving. It integrates data collection, training, and evaluation components with simulation (using CARLA) to train a driving policy that operates “in imagination.”
• The README, together with the file tree, indicates that the system is designed to interact with the CARLA simulator, process sensor and map data (e.g., bird's-eye view, RGB camera inputs), and drive using both RL-based and imitation learning approaches.
Main Components
a. Simulation & Environment Interface (carla_gym)
• The “carla_gym” package encapsulates the environment interface and simulation specifics:
• The “core” folder includes various observation managers that handle input data (birdview, camera, navigation, etc.), object detection (object_finder), and task-related components.
• “envs” which describe different simulation scenarios and test cases (e.g., LeaderBoard scenarios with various Town configurations).
• “utils” to support key functionalities such as map transformations and weather simulation.
• This part as the simulation/environment wrapper that interfaces directly with the external CARLA simulator.
b. Training, Inference & Model Components (mile)
• The “mile” directory holds the core learning components:
• “agents” that include both the imitation-based “mile_agent” and the RL-based “rl_birdview_agent.”
• “models” representing the neural network architectures (e.g., world model, policy network, auxiliary processing layers).
• “trainer.py” and other utilities (e.g., losses, preprocessing, data handling) for training the driving policy.
• “configs” containing YAML configuration files to set up experiments (debug, one_frame, mile, etc.).
• This section should be illustrated as the “brain” of the system with flows for training input, forward passes through models, and feedback from loss calculations and policy evaluations.
c. Data Collection & Evaluation Pipeline
• Scripts and configurations for:
• Data collection (e.g., run/data_collect.sh, config/data_collect.yaml) that collect simulation data from CARLA.
• Evaluation (e.g., run/evaluate.sh) which runs the pre-trained model and evaluates it in the simulation environment.
• These components are critical for the full lifecycle: they provide the data that feeds into training and assess the performance of the model in simulated environments.
d. Configuration and Utility Layers
• “config” directory: Contains configuration files for agents, training, and test suites.
• “utils”: Common utilities such as saving checkpoints, managing network-related tasks, and server utilities.
• That these utilities support and glue the main components together.
e. External Dependencies
• PyTorch: Indicate that the training/inference backend is implemented using PyTorch.
• Any other relevant libraries (e.g., conda for environment management, YAML for configuration, etc.).
Relationships and Interactions
• Data Flow:
• The output of the model (driving policy) then influences vehicle simulation within the CARLA environment.
• Training loop feedback should be shown from the trainer and loss components back onto model parameters.
• Dependency Links:
• Configuration files (from the config folder) to both the simulation environment (which they help parameterize) and the training pipeline.
• Role of utility functions in each subsystem, acting as common services for processing data, transforming coordinates, managing weather, etc.
Architectural Patterns and Design Principles
• Modularity: The separation between carla_gym (simulation interfacing) and mile (model training and inference) indicates a modular design where changes in simulation or modeling can be developed separately.
• Configuration-Driven Design: With several configuration files driving agent parameters, observation setups, and test suites, note how every component is highly configurable.
• Layered Architecture: From low-level sensor data processing in the observation manager to high-level learning and policy evaluation, the design follows a layered approach.
Final Steps
• Reference the test suites and scenario descriptions as input sources/configurations for simulation runs.