How nvlabs/trajdata works
Main Components (Top Level)
• Use a layered architecture with the following main components from bottom to top:
• Raw Data Layer
• Data Processing Layer
• API Layer
• Application Layer
Component Details
a) Raw Data Layer:
• Them to a "Raw Dataset Interface" component
b) Data Processing Layer (Core):
• Dataset Specific Processors (from dataset_specific/)
• Caching System (from caching/)
• Data Structures (from data_structures/)
• Filtering Components (from filtering/)
c) API Layer:
• UnifiedDataset API (main interface)
• API (for vector map access)
• Simulation API (for simulation capabilities)
d) Application Layer:
• Example applications/use cases:
• Batch Processing
• Operations
• Simulation
• Visualization
• Two-stage process as mentioned in the README:
• Stage 1: Raw Data → Preprocessing → Cache
• Stage 2: Cache → Unified Interface → Application
Special Components
a) Caching System:
• Cache components:
• DataFrame Cache
• Environment Cache
• Scene Cache
• Simulation Cache
b) Processing:
• Vector map processing components
• Connection to supported datasets with maps
Important Details
• Data transformation flow
• Parallel processing capabilities
• Cache management
• Dataset compatibility matrix
• Supported data formats
• Time step handling (dt synchronization)
Additional Elements
• Version/compatibility information
• Supported dataset characteristics (dt, maps, etc.)
Key Relationships to Highlight
• Dataset preprocessing workflow
• Caching mechanism
• API interactions
• Simulation system integration
• API integration
Remember to include the parallel processing capabilities and the optional preprocessing stage, as these are key features of the system's architecture.