# nvlabs/trajdata architecture

> Architecture diagram of the GitHub repository nvlabs/trajdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/trajdata
- Repository: https://github.com/nvlabs/trajdata

## Overview

##### Main Components (Top Level)
- Use a layered architecture with the following main components from bottom to top:
- Raw Data Layer
- Data Processing Layer
- API Layer
- Application Layer

##### Component Details

a) Raw Data Layer:
- Them to a "Raw Dataset Interface" component

b) Data Processing Layer (Core):
- Dataset Specific Processors (from dataset_specific/)
- Caching System (from caching/)
- Data Structures (from data_structures/)
- Filtering Components (from filtering/)

c) API Layer:
- UnifiedDataset API (main interface)
- API (for vector map access)
- Simulation API (for simulation capabilities)

d) Application Layer:
- Example applications/use cases:
- Batch Processing
- Operations
- Simulation
- Visualization

- Two-stage process as mentioned in the README:
- Stage 1: Raw Data → Preprocessing → Cache
- Stage 2: Cache → Unified Interface → Application

##### Special Components

a) Caching System:
- Cache components:
- DataFrame Cache
- Environment Cache
- Scene Cache
- Simulation Cache

b) Processing:
- Vector map processing components
- Connection to supported datasets with maps

##### Important Details
- Data transformation flow
- Parallel processing capabilities
- Cache management
- Dataset compatibility matrix
- Supported data formats
- Time step handling (dt synchronization)

##### Additional Elements
- Version/compatibility information
- Supported dataset characteristics (dt, maps, etc.)

##### Key Relationships to Highlight
- Dataset preprocessing workflow
- Caching mechanism
- API interactions
- Simulation system integration
- API integration

Remember to include the parallel processing capabilities and the optional preprocessing stage, as these are key features of the system's architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and Classes
    classDef raw fill:#gray,stroke:#333
    classDef core fill:#blue,stroke:#333
    classDef api fill:#green,stroke:#333
    classDef cache fill:#yellow,stroke:#333
    classDef app fill:#purple,stroke:#333

    %% Raw Data Layer
    subgraph RawData
        NuScenes["nuScenes Dataset"]:::raw
        Lyft["Lyft Dataset"]:::raw
        Waymo["Waymo Dataset"]:::raw
        Argo["Argoverse2 Dataset"]:::raw
        RawInterface["Raw Dataset Interface"]:::raw
    end

    %% Data Processing Layer
    subgraph Processing
        subgraph DataProcessors
            NuScProc["nuScenes Processor"]:::core
            LyftProc["Lyft Processor"]:::core
            WaymoProc["Waymo Processor"]:::core
            ArgoProc["Argoverse2 Processor"]:::core
        end

        subgraph DataStructures
            Agent["Agent"]:::core
            Batch["Batch"]:::core
            Scene["Scene"]:::core
            State["State"]:::core
        end

        subgraph CacheSystem
            DFCache["DataFrame Cache"]:::cache
            EnvCache["Environment Cache"]:::cache
            SceneCache["Scene Cache"]:::cache
            SimCache["Simulation Cache"]:::cache
        end

        Filters["Filtering Components"]:::core
        ParallelProc["Parallel Processing"]:::core
    end

    %% API Layer
    subgraph APIs
        UnifiedAPI["UnifiedDataset API"]:::api
        MapAPI["Map API"]:::api
        SimAPI["Simulation API"]:::api
    end

    %% Application Layer
    subgraph Applications
        BatchProc["Batch Processing"]:::app
        MapOps["Map Operations"]:::app
        Simulation["Simulation"]:::app
        Visualization["Visualization"]:::app
    end

    %% Relationships
    NuScenes & Lyft & Waymo & Argo --> RawInterface
    RawInterface --> DataProcessors
    DataProcessors --> CacheSystem
    CacheSystem --> UnifiedAPI
    DataStructures --> UnifiedAPI
    Filters --> UnifiedAPI
    ParallelProc --> DataProcessors
    UnifiedAPI & MapAPI & SimAPI --> Applications

    %% Click Events
    click RawInterface "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset_specific/raw_dataset.py"
    click NuScProc "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset_specific/nusc/nusc_dataset.py"
    click WaymoProc "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset_specific/waymo/waymo_dataset.py"
    click LyftProc "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset_specific/lyft/lyft_dataset.py"
    click ArgoProc "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset_specific/argoverse2/av2_dataset.py"
    click DFCache "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/caching/df_cache.py"
    click EnvCache "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/caching/env_cache.py"
    click SceneCache "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/caching/scene_cache.py"
    click SimCache "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/simulation/sim_cache.py"
    click Agent "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/data_structures/agent.py"
    click Batch "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/data_structures/batch.py"
    click Scene "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/data_structures/scene.py"
    click State "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/data_structures/state.py"
    click Filters "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/filtering/filters.py"
    click UnifiedAPI "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/dataset.py"
    click MapAPI "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/maps/map_api.py"
    click SimAPI "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/simulation/sim_scene.py"
    click ParallelProc "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/parallel/data_preprocessor.py"
    click Visualization "https://github.com/NVlabs/trajdata/blob/main/src/trajdata/visualization/vis.py"
```
