How commaai/comma2k19 works
Project type and purpose
• This is an open-source dataset repository (“comma2k19”) with example code and utilities to read, process, and benchmark multi-sensor driving logs (camera video, raw CAN, IMU, GNSS).
• It provides a reproducible data-processing pipeline and Jupyter notebooks for exploration, plus a Dockerfile and CI config for environment setup and tests.
File structure
• Top-level:
• .github/workflows/main.yml – CI pipeline
• Dockerfile – containerized environment
• Example_1/… – one sample dataset chunk containing raw “video.hevc” and compressed “raw_log.bz2”, plus precomputed “processed_log” and “global_pose” subfolders
• assets/ – images for README illustrations
• notebooks/ – Jupyter notebooks and ancillary text files (camera intrinsics, example benchmarks, readers)
• openpilot/ – git submodule providing logreader/framereader tools
• utils/ – Python package with modules for data I/O, coordinate transforms, camera models, benchmarks, and a helper for Microsoft FS quirks
• requirements.txt – Python dependencies
• Patterns:
• Data-driven pipeline: raw data → unzip → numpy arrays → benchmarks/plots
• Modular code: “utils” library used by notebooks and potentially external tools
• Reproducible environment: Docker + CI
README for additional insights
• Describes dataset layout and formats (processed_log, global_pose, IMU/CAN/GNSS message schemas)
• Points out integration with OpenPilot tools and Laika GNSS library
• Highlights use cases: GNSS/INS/vision fusion, mapping, benchmarking
a. Main components
• Data Storage
• Raw Data (raw_log.bz2, video.hevc)
• Preprocessed Data (processed_log/, global_pose/)
• Processing Library
• utils package (dataset reader, coordinate transforms, camera models, benchmarks)
• unzip_msft_fs.py for filesystem compatibility
• External Integrations
• OpenPilot submodule (logreader, framereader)
• Laika GNSS library (external process/library)
• Example Workflows
• Jupyter notebooks (raw_readers.ipynb, processed_readers.ipynb, position_benchmark.ipynb)
• Environment & CI
• Docker container (Dockerfile)
• GitHub Actions (main.yml)
b. Relationships & interactions
• Raw Data → unzip_msft_fs.py (if needed) → utils.dataset reader → numpy arrays
• numpy arrays + openpilot.tools → data ingestion in notebooks
• utils.camera & utils.coordinates → calibrate & project poses (refer to camera_intrinsics.txt)
• Laika library invoked within utils or notebooks for GNSS processing
• notebooks import utils and openpilot modules → generate plots/metrics
• Dockerfile bundles Python environment + openpilot submodule → reproducible runs
• CI workflow builds Docker image, runs unit tests on utils, lints notebooks, verifies example notebooks execute without errors
c. Architectural patterns / design principles
• Data pipeline pattern: clear separation between raw data ingestion, preprocessing, analysis
• Modular library design: utils package exposes clean APIs for reading and processing data streams
• Reproducibility: containerization (Docker), CI testing, sample data segment
• Extensibility: open submodule points for additional processing (e.g., swapping Laika for another GNSS library)
d. Key technologies
• Python 2.7 / 3.6, numpy, Jupyter
• OpenPilot tools (capnp-based logreader, framereader)
• Laika GNSS processing library
• Docker, GitHub Actions
Final advice
• Be exhaustive: break the system into data layers, code layers, and infrastructure layers
• Component with its role, inputs, and outputs