# theopendatastackofficial/nyc-mta-data architecture

> Architecture diagram of the GitHub repository theopendatastackofficial/nyc-mta-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theopendatastackofficial/nyc-mta-data
- Repository: https://github.com/theopendatastackofficial/nyc-mta-data

## Overview

##### Project Type and Purpose
- The repository is an end-to-end data platform focused on ingesting, cleaning, transforming, and visualizing MTA (Metropolitan Transportation Authority) datasets.
- Key capabilities include:
– Ingestion of live data from the Socrata and Open Mateo APIs (and even partitioned parquet datasets).
– SQL-based transformation (using DBT) executed on data stored in DuckDB.
– Interactive visualizations and reporting through separate UIs (the Data App UI and Data Dictionary UI).
– Provision for ad-hoc SQL querying (via the Harlequin terminal-based SQL editor) and notebook-based exploration.

##### File Structure for Architectural Insights
- Top-level directories reveal the modular layout:
– app: Contains UI elements and markdown pages that likely serve as documentation or front-end presentation of reports.
– pipeline: Houses the ingestion, asset management, and data transformation orchestration (including modules for ingestion, processing, DBT integration, specialized IO managers, and utilities for DuckDB handling).
– transformations: Contains a DBT project with models, tests, and profiles that form the SQL transformation pipeline.
– dictionary: Provides a data dictionary application (with scripts to create a UI for visualizing datasets and schemas).
– notebooks: Provides interactive notebooks (like an EDA notebook) for data exploration.
– scripts and tools: Utility scripts for tasks like DAG configuration, evidence templating, and runtime support.
- Configuration files and setup scripts (e.g., setup.sh, setup.bat) indicate environment setup (Python’s uv, JS’s Bun) as well as integration with orchestration frameworks like Dagster.

##### Key Components and Their Relationships
- External Data Sources:
– Socrata API for MTA data.
– Open Mateo API for weather data.
– Possibly large static datasets from external repositories.
- Data Ingestion & Pipeline (Pipeline Directory):
– Responsible for fetching data, running through cleaning and ingestion routines, and storing raw data as partitioned parquet files.
– DAG and orchestration components (Dagster) track workflow states and enable materialization.
- Data Storage:
– Use of DuckDB is evident (via SQL scripts in app/sources and pipeline/warehouse modules). DuckDB acts both as the persistent storage layer and as a queryable database for both transformations and ad-hoc queries.
- Data Transformation:
– DBT (in the transformations/dbt directory) defines SQL models that transform raw ingested data into analytic views.
– Integration with pipeline resources (such as DBT assets in pipeline/assets/dbt_assets.py) demonstrates how the transformed data becomes accessible.
- UIs and Data Consumption:
– Data Dictionary UI: Managed under the dictionary directory (with scripts like create.py) to explore dataset schemas, especially toggling for LLM mode.
– Data App UI: Managed in the app directory and executed via Bun (JS runtime) to serve interactive visual presentations to end users.
– Harlequin SQL Editor integration for ad-hoc querying is provided by a CLI command that connects to the DuckDB file.
- Development Tools:
– Notebooks for EDA.
– Scripts and tools that help manage, export, and generate configurations (e.g., generate_dagsteryaml.py in scripts).

##### Recommended Elements
- Clearly delineated labeled components:
– Data Sources: Socrata API, Open Mateo API, and external data repositories.
– Data Ingestion/ETL Pipeline: Represented as modules that call ingestion routines and store raw data.
– Processing/Transformation Layer: DBT jobs (transformation models) that work on the DuckDB warehouse.
– Data Storage: The DuckDB database holding raw tables, views, and intermediate transformation results.
– Orchestration & Workflow Manager: Dagster server controlling pipeline materializations.
– UIs: Separate components for the Data Dictionary (Python-based) and the Data App UI (JS/TS-based with Bun), plus notebooks for manual analysis.
– Ad-hoc Querying Interface: Harlequin SQL Editor.
– Show data flow from external APIs into the ingestion layer.
– Connect ingestion to data storage (parquet files and DuckDB).
– Illustrate the transformation flow from raw storage to corrected, cleaned views using DBT.
– Indicate how the UIs and the SQL editor read from the DuckDB.
- Architectural Patterns & Technologies:
– Emphasize the separation-of-concerns between ingestion, transformation, and presentation.
– Note use of microservice-like segregation such as having distinct directories and services for pipeline tasks versus UI tasks.
– Indicate key technologies: Python (for pipeline, uv management, Dagster, DuckDBWrapper), JavaScript/TypeScript (for the app UI via Bun), SQL/DBT, and orchestration patterns.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% External Data Sources
    subgraph "External Data Sources"
        socrata["Socrata API"]:::external
        openMateo["Open Mateo API"]:::external
        staticDS["Static External Datasets"]:::external
    end

    %% Data Pipeline Layer
    subgraph "Data Pipeline"
        ingestion["Data Ingestion Pipeline"]:::process
        orchestration["Dagster Orchestration"]:::orchestration
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        parquet["Parquet Files Storage"]:::storage
        duckdb["DuckDB Database"]:::storage
    end

    %% Transformation Layer
    subgraph "Transformation Layer"
        dbt["DBT Transformations"]:::transformation
    end

    %% UIs and Query Tools
    subgraph "UIs and Query Tools"
        dataApp["Data App UI"]:::ui
        dataDict["Data Dictionary UI"]:::ui
        sqlSources["SQL Sources"]:::ui
        sqlEditor["Harlequin SQL Editor"]:::ui
        notebooks["Interactive Notebooks (EDA)"]:::ui
    end

    %% Connections: External Sources to Ingestion
    socrata -->|"ingests"| ingestion
    openMateo -->|"ingests"| ingestion
    staticDS -->|"ingests"| ingestion

    %% Orchestration controls Ingestion and Transformation
    orchestration -->|"schedules"| ingestion
    orchestration -->|"triggers"| dbt

    %% Data Ingestion flows to Storage
    ingestion -->|"stores raw data"| parquet
    ingestion -->|"loads into"| duckdb

    %% Transformation reads from DuckDB and refines data
    duckdb -->|"source for"| dbt
    dbt -->|"updates"| duckdb

    %% UIs and Query Tools consume data from DuckDB
    duckdb -->|"serves data to"| dataApp
    duckdb -->|"serves data to"| dataDict
    duckdb -->|"serves data to"| sqlSources
    duckdb -->|"serves data to"| sqlEditor
    duckdb -->|"serves data to"| notebooks

    %% Click Events
    click ingestion "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/pipeline/assets/ingestion/"
    click dbt "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/transformations/dbt/"
    click dataApp "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/app/"
    click dataDict "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/dictionary/"
    click duckdb "https://github.com/theopendatastackofficial/nyc-mta-data/blob/main/pipeline/assets/warehouse/duckdb_warehouse.py"
    click sqlSources "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/app/sources/app/"
    click notebooks "https://github.com/theopendatastackofficial/nyc-mta-data/tree/main/notebooks/"
    click orchestration "https://github.com/theopendatastackofficial/nyc-mta-data/blob/main/scripts/generate_dagsteryaml.py"

    %% Styles
    classDef external fill:#ffcc00,stroke:#333,stroke-width:2px,color:#000;
    classDef process fill:#99ccff,stroke:#333,stroke-width:2px,color:#000;
    classDef storage fill:#a0d468,stroke:#333,stroke-width:2px,color:#000;
    classDef transformation fill:#f6bb42,stroke:#333,stroke-width:2px,color:#000;
    classDef ui fill:#ed5565,stroke:#333,stroke-width:2px,color:#000;
    classDef orchestration fill:#da70d6,stroke:#333,stroke-width:2px,color:#000;
```
