How kesai-labs/py123d works
1. Top-level shape
The repository is a Python package centered on one goal: normalize autonomous-driving datasets into a shared internal model, then expose that model through a single read/visualization API. The architecture is split into clear layers: raw dataset ingestion, canonical datatypes/geometry, Arrow-backed storage APIs, command-line orchestration, and visualization.
2. Primary data flow
The main flow is: dataset-specific downloader/parser reads raw files from external dataset sources, maps them into the package’s canonical datatypes, and writes converted logs/maps into an Apache Arrow-based format. After conversion, consumers use the scene/map APIs to read data with zero-copy, memory-mapped access. The converted log often keeps relative references to original camera/lidar assets rather than duplicating them.
3. Canonical domain model
src/py123d/datatypes and src/py123d/geometry define the common language of the system. Datatypes cover sensors, vehicle state, detections, metadata, map objects, and time. Geometry provides points, vectors, rotations, transforms, bounding boxes, polylines, poses, and occupancy maps. This is the main boundary that all dataset-specific code must target.
4. Scene and map APIs
src/py123d/api/scene and src/py123d/api/map are the public runtime interfaces. They sit on top of the stored Arrow representation and provide dataset-agnostic access to scenes, modalities, metadata, and map layers. The scene API is the main read path for time-indexed multi-modal data; the map API is the read path for HD-map content.
5. Dataset-specific parser layer
src/py123d/parser contains the ingestion/adaptation layer, with one subpackage per supported dataset: AV2, nuScenes, nuPlan, KITTI-360, PandaSet, WOD, CARLA/OpenDRIVE, Physical AI AV, NCore, and L3AD/Lead123D. Each parser package has dataset constants, helper utilities, sensor I/O, and often a separate map parser. This layer is where dataset quirks, schema differences, labels, timestamps, and coordinate conventions are normalized.
6. Map ingestion strategy
Map support is not monolithic: some datasets have native map parsers, while CARLA/OpenDRIVE is handled through a dedicated XODR parser and helper utilities. The code includes specialized map-object reconstruction, lane/road-edge/signals handling, and embedded CARLA map assets. The map pipeline is a distinct branch from sensor/scene ingestion but converges into the same map-object datatypes.
7. Arrow writers and modality builders
src/py123d/api/scene/arrow and src/py123d/api/map/arrow implement the write-side architecture. These modules build Arrow-compatible records for scenes, logs, and maps, with modality-specific handlers for cameras, lidar, ego state, box detections, traffic lights, and custom modalities. The write path is responsible for synchronizing modalities, caching scene state, and enforcing schema consistency before persistence.
8. Modality boundaries
Modalities are modeled explicitly rather than as generic blobs. Camera, lidar, ego state, detections, sync data, and custom fields each have dedicated Arrow modality implementations. This separation matters because the repository supports both synchronized and native-rate/asynchronous capture streams, so the system must preserve timing semantics per modality.
9. Common runtime utilities
src/py123d/common holds cross-cutting infrastructure: dataset path management, file-format IO for camera and lidar codecs, dependency gating, UUID/timer helpers, and execution backends. The IO subpackages support multiple encodings like JPEG/PNG/MP4 for cameras and LAZ/Draco/Arrow IPC for lidar. This layer is reused by parsers, writers, and visualization code rather than tied to any dataset.
10. Execution and scalability
The repository supports multiple execution strategies under src/py123d/common/execution: sequential, thread pool, process pool, and Ray. That indicates conversion and preprocessing can be parallelized, but the framework abstracts the executor choice so the rest of the pipeline is scheduler-agnostic. This is important for large dataset conversion jobs and for keeping the parsing logic independent of compute deployment.
11. CLI and configuration system
src/py123d/script is the orchestration layer. It wires together dataset download, conversion, and viewer startup through Hydra-based YAML configuration. The configs are split by concern: dataset presets, download presets, conversion presets, scene filters, execution backends, and viewer settings. This makes the runtime behavior highly declarative and allows the same core code to serve multiple dataset pipelines.
12. Visualization subsystem
src/py123d/visualization provides two consumer-facing render paths: Matplotlib for static plotting and Viser for interactive 3D inspection. The Viser stack has controllers, an element manager, and renderable elements for cameras, lidar, ego state, boxes, and maps. It consumes the public scene API, so visualization depends on the canonical data model rather than directly on raw dataset parsers.
13. External dependencies and dataset boundaries
The repo interacts with external dataset ecosystems and their storage formats: official dataset downloads, dataset-specific metadata and annotations, and in some cases vendored protobufs or map assets. The architecture deliberately isolates these external dependencies inside parser/download modules so the public API remains uniform even when backend formats differ substantially.
14. Quality and release pipeline
Tests are broad and mirror the package structure, with unit coverage for datatypes, geometry, parsers, executors, IO, and scene/map roundtrips. Docs are built from a dedicated Sphinx tree and tutorials demonstrate the conversion-to-viewer workflow. GitHub Actions handle docs deployment, PyPI release, linting/pre-commit, and pytest, which reinforces that the package is meant to be both a library and a reproducible data-processing toolchain.
Main entry points: datatypes Datatypes · script CLI Scripts · registry.py Parser Registry · arrow Scene Arrow
Components 23
Core model
Canonical datatypes and geometry used across ingestion, storage, and visualization.
Runtime API
Public read/write APIs for scenes and maps, backed by Arrow storage.
- Scene API · Dataset-agnostic access to logs, scenes, modalities, and scene metadata.
src/py123d/api/scene/scene_api.py - Scene Arrow · Arrow-backed scene builders, log writers, and modality handlers.
src/py123d/api/scene/arrow - Map API · Dataset-agnostic access to map content and layers.
src/py123d/api/map/map_api.py - Map Arrow · Arrow-backed map writer and reader for persisted map assets.
src/py123d/api/map/arrow
Ingestion
Dataset adapters and map parsers that normalize external sources into the internal model.
- Parser Registry · Registers and dispatches dataset-specific parsers and downloaders.
src/py123d/parser/registry.py - AV2 · Argoverse 2 ingestion, sensor I/O, and map parsing.
src/py123d/parser/av2 - nuPlan · nuPlan conversion, sensor I/O, and map parsing.
src/py123d/parser/nuplan - nuScenes · nuScenes conversion, sensor I/O, and map parsing.
src/py123d/parser/nuscenes - KITTI-360 · KITTI-360 ingestion and map parsing.
src/py123d/parser/kitti360 - PandaSet · PandaSet ingestion and sensor I/O.
src/py123d/parser/pandaset - WOD · Waymo Open Dataset perception and motion ingestion.
src/py123d/parser/wod - OpenDRIVE · CARLA/OpenDRIVE map ingestion and XODR reconstruction.
src/py123d/parser/opendrive - Physical AI AV · Physical AI AV ingestion and sensor I/O.
src/py123d/parser/physical_ai_av - NCore · NCore ingestion and sensor I/O.
src/py123d/parser/ncore - Lead123D · Dataset download entry for Lead123D.
src/py123d/parser/lead123d/lead123d_download.py
Platform
Cross-cutting utilities for IO, execution, and shared helpers.
- Common IO · Shared camera and lidar codecs plus dataset path handling and utilities.
src/py123d/common/io - Execution · Sequential, threaded, process, and Ray execution backends.
src/py123d/common/execution - Common Utils · Dependency gating, UUID/timing helpers, mixins, and messaging utilities.
src/py123d/common/utils
Orchestration
CLI entrypoints and Hydra-based configuration for download, conversion, and viewing.
- CLI Scripts · Download, conversion, and viewer command entrypoints.
src/py123d/script - Configs · Hydra presets for datasets, conversion, execution, scene filters, and viewer settings.
src/py123d/script/config
Visualization
Static and interactive consumers of the public scene/map APIs.
- Visualization · Matplotlib plots and interactive Viser viewer components.
src/py123d/visualization
Connections 28
- AV2 → Datatypes: normalize into · Maps AV2-specific records into canonical domain objects.
- nuPlan → Datatypes: normalize into · Maps nuPlan records into canonical domain objects.
- nuScenes → Datatypes: normalize into · Maps nuScenes records into canonical domain objects.
- KITTI-360 → Datatypes: normalize into · Maps KITTI-360 records into canonical domain objects.
- PandaSet → Datatypes: normalize into · Maps PandaSet records into canonical domain objects.
- WOD → Datatypes: normalize into · Maps Waymo Open Dataset records into canonical domain objects.
- OpenDRIVE → Geometry: build map geometry · Reconstructs lane and road geometry from OpenDRIVE assets.
- OpenDRIVE → Datatypes: emit map objects · Converts parsed map features into canonical map objects.
- Parser Registry → AV2: dispatches · Routes AV2 dataset jobs to the AV2 adapter.
- Parser Registry → nuPlan: dispatches · Routes nuPlan dataset jobs to the nuPlan adapter.
- Parser Registry → nuScenes: dispatches · Routes nuScenes dataset jobs to the nuScenes adapter.
- Parser Registry → OpenDRIVE: dispatches · Routes CARLA/OpenDRIVE map jobs to the dedicated parser.
- Scene Arrow → Datatypes: serializes · Persists canonical scene data into Arrow-backed storage.
- Scene API → Scene Arrow: reads · Public scene access is served from Arrow scene storage.
- Map Arrow → Datatypes: serializes · Persists canonical map data into Arrow-backed storage.
- Map API → Map Arrow: reads · Public map access is served from Arrow map storage.
- Common IO → Scene Arrow: feeds media · Camera and lidar codecs support the scene write/read path.
- Common IO → Parser Registry: supports ingestion · Shared file and asset IO is reused during dataset parsing.
- Execution → CLI Scripts: backends for · Provides selectable executors for conversion and preprocessing.
- CLI Scripts → Configs: loads · CLI entrypoints are configured through Hydra presets.
- CLI Scripts → Parser Registry: orchestrates · Download and conversion commands invoke dataset parsers.
- CLI Scripts → Scene Arrow: writes · Conversion commands build Arrow scene logs.
- CLI Scripts → Map Arrow: writes · Conversion commands persist map artifacts.
- Visualization → Scene API: consumes · Static and interactive viewers consume the public scene API.
- Visualization → Map API: consumes · Map rendering uses the public map API.
- Scene Arrow → Common Utils: uses · Scene builders share timing, UUID, and dependency helpers.
- Map Arrow → Common Utils: uses · Map writers share common helper utilities.
- Lead123D → CLI Scripts: invoked by · Dataset download flow is surfaced through the CLI layer.