How mazenmtulab/the-barn-challenge works
1. This repo is a ROS-based navigation benchmark and submission harness for the ICRA BARN Navigation Challenge. The core purpose is not a standalone robot app, but a controlled simulation/evaluation environment for running a navigation stack against many predefined Gazebo worlds.
2. The top-level orchestrator is run.py. It appears to be the main entry point used in both local and containerized runs. Its job is to select a world by index, bring up the simulation, and initialize one navigation stack implementation. The README explicitly says the user should only customize the navigation-stack initialization section in this file.
3. gazebo_simulation.py likely encapsulates simulation startup and world loading. Together with jackal_helper/launch/gazebo_launch.launch, it forms the simulation boundary: Gazebo world + Jackal robot + ROS interfaces. This is where the environment is instantiated before any planner runs.
4. The repository supports at least two planner families. The jackal_helper/launch/move_base_DWA.launch and move_base_eband.launch files configure ROS move_base with different local planners. In parallel, navigation_pkg/scripts/Nav-VFH.py and navigation_pkg/scripts/navigation_utils.py implement a separate VFH-based navigation stack. This gives the repo multiple interchangeable “navigation backend” paths under the same simulation harness.
5. The jackal_helper ROS package is the simulation support package. Its configs/params/*.yaml files provide move_base, costmap, local planner, and odometry-navigation parameters. This package is the configuration and robot-integration layer between the simulator and planner nodes.
6. jackal_helper/src/collision_publisher.cpp is the only compiled C++ source shown, so it is likely a small ROS node/plugin responsible for collision detection or collision-state publication during benchmark runs. Architecturally, it sits on the simulator side and feeds termination/evaluation signals back to the harness.
7. jackal_helper/worlds/BARN and jackal_helper/worlds/DynaBARN are the benchmark assets. The many .world files are Gazebo environments, while the BARN path_files/*.npy set likely stores reference paths or scenario metadata used for evaluation. DynaBARN is explicitly noted as an extension with dynamic environments.
8. The navigation stack boundary is intentionally loose: run.py launches whichever planner is being tested, but the repo does not enforce a single API beyond the ROS ecosystem. move_base-based stacks consume ROS topics/services and planner parameters; the VFH stack is a Python implementation with its own utilities. The common contract is the simulated robot in Gazebo and the benchmark episode outcome.
9. Evaluation is handled outside the live simulation loop by test.sh and report_test.py. test.sh appears to batch multiple runs across sampled worlds, while report_test.py aggregates outcomes into average time, metric, success, collision, and timeout rates. This means the repo separates episode execution from score computation.
10. Containerization is a first-class deployment path. Singularityfile.def, entrypoint.sh, and singularity_run.sh define the reproducible runtime used for challenge submission. The container boundary matters because it packages ROS, Python dependencies, and the navigation code so the same run.py entry point can run consistently on the evaluator’s machine.
11. External dependencies that materially shape the architecture are ROS, Gazebo, Jackal robot simulation packages, and move_base/eband_local_planner. The README also requires Python packages like numpy, rospkg, netifaces, and defusedxml, which supports the Python orchestration and evaluation scripts.
12. A clean way to draw the architecture is: benchmark assets and world files on one side; simulation/orchestration in the middle (run.py, Gazebo, ROS launch/config, collision node); navigation stack as a swappable module set (move_base DWA/Eband or Python VFH); and evaluation/container tooling around the edges (test.sh, report_test.py, Singularity scripts).
Main entry points: run.py run.py · jackal_helper Jackal helper · move_base_DWA.launch move_base DWA · move_base_eband.launch move_base eband
Components 24
Orchestration
Top-level run control and simulation bootstrapping
- run.py · Main harness entrypoint that selects worlds and starts a navigation stack
run.py
Simulation
Gazebo, robot integration, and benchmark runtime services
- gazebo sim · Python wrapper for launching Gazebo and loading the chosen world
gazebo_simulation.py - Gazebo launch · Launches the Gazebo simulation boundary for the Jackal benchmark
jackal_helper/launch/gazebo_launch.launch - Collision node · Compiled node that publishes collision state or termination signals
jackal_helper/src/collision_publisher.cpp - Jackal helper · Simulation support package connecting planners, robot interfaces, and benchmark settings
jackal_helper - move_base params · Core move_base and planner parameters
jackal_helper/configs/params/move_base_params.yaml - costmap params · Shared costmap configuration for navigation stacks
jackal_helper/configs/params/costmap_common_params.yaml - local planner · Base local planner tuning for move_base
jackal_helper/configs/params/base_local_planner_params.yaml - odom nav · Odometry-navigation costmap parameters
jackal_helper/configs/params/odom_nav_params
Navigation
Swappable planner backends tested inside the harness
- move_base DWA · Configures move_base with the DWA local planner
jackal_helper/launch/move_base_DWA.launch - move_base eband · Configures move_base with the eband local planner
jackal_helper/launch/move_base_eband.launch - VFH launch · Launches the Python VFH-based navigation stack
navigation_pkg/launch/VFH_launch.launch - Nav-VFH · VFH navigation controller used as an alternate backend
navigation_pkg/scripts/Nav-VFH.py - Nav utils · Shared helpers for the VFH navigation implementation
navigation_pkg/scripts/navigation_utils.py
Benchmark Assets
Worlds and reference path data used for episodes
- BARN worlds · Static benchmark Gazebo worlds for episode sampling
jackal_helper/worlds/BARN - DynaBARN · Dynamic benchmark world set
jackal_helper/worlds/DynaBARN - Path files · Reference paths or scenario metadata for BARN evaluation
jackal_helper/worlds/BARN/path_files - World files · Indexed benchmark world definitions
jackal_helper/worlds/BARN/world_0.world
Evaluation
Batch runs, reporting, and container entrypoints
- Batch test · Runs repeated benchmark episodes across sampled worlds
test.sh - Dynamic test · Batch runner focused on dynamic benchmark episodes
test_dynamic.sh - Report · Aggregates run results into benchmark metrics and success rates
report_test.py - Container · Reproducible submission runtime definition
Singularityfile.def - Entrypoint · Container startup script for the benchmark harness
entrypoint.sh - Run script · Convenience wrapper for launching the containerized benchmark
singularity_run.sh
Connections 25
- run.py → gazebo sim: boots · run.py delegates simulation startup and world selection
- run.py → move_base DWA: can launch · run.py selects a move_base-based backend when configured
- run.py → move_base eband: can launch · run.py selects the alternative move_base backend when configured
- run.py → VFH launch: can launch · run.py can initialize the VFH navigation stack
- gazebo sim → Gazebo launch: uses · Simulation wrapper calls the Gazebo launch boundary
- Gazebo launch → Jackal helper: part of · Gazebo launch belongs to the Jackal helper ROS package
- Collision node → run.py: signals outcome · Collision state feeds benchmark termination and scoring
- move_base DWA → move_base params: loads · DWA move_base launch consumes planner settings
- move_base DWA → costmap params: loads · DWA stack consumes shared costmap settings
- move_base eband → move_base params: loads · Eband move_base launch consumes planner settings
- move_base eband → costmap params: loads · Eband stack consumes shared costmap settings
- VFH launch → Nav-VFH: starts · VFH launch starts the Python planner
- Nav-VFH → Nav utils: depends on · Planner logic uses shared VFH utilities
- Jackal helper → move_base params: owns · Jackal helper provides navigation parameter sets
- Jackal helper → costmap params: owns · Jackal helper provides shared costmap parameters
- Jackal helper → local planner: owns · Jackal helper provides local planner parameters
- Jackal helper → odom nav: owns · Jackal helper provides odometry-navigation parameters
- BARN worlds → World files: contains · BARN benchmark set includes Gazebo world files
- BARN worlds → Path files: contains · BARN benchmark set includes reference path data
- DynaBARN → World files: extends · Dynamic benchmark family is a related world collection
- Batch test → Report: feeds · Batch test results are summarized by the report script
- Dynamic test → Report: feeds · Dynamic batch results are summarized by the report script
- Container → Entrypoint: starts · Container definition uses the entrypoint script
- Entrypoint → run.py: invokes · Container startup ultimately runs the benchmark entrypoint
- Run script → Container: wraps · Convenience script runs the singularity container