# karanamlokesh/dataexpert-lokesh-capstone architecture

> Architecture diagram of the GitHub repository karanamlokesh/dataexpert-lokesh-capstone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karanamlokesh/dataexpert-lokesh-capstone
- Repository: https://github.com/karanamlokesh/dataexpert-lokesh-capstone

## Overview

Based on our review of the file tree and the README, this project is a real-time data pipeline and analytics system focused on ingesting, processing, and visualizing train movement and weather data. The system leverages a suite of technologies—including Kafka for messaging, Snowflake as a data warehouse, Airflow for orchestration, DBT for transformations and testing, and Streamlit for dashboards—to provide real-time monitoring and analytics for train delays and utilization. The architectural design employs a layered data model (L0, L1, and L2) and utilizes separate components to handle real-time ingestion from public data sources and subsequent data transformation, quality tests, and reporting.

##### Main Components
- Data Ingestion Layer:
- Stomp Server: Continuously connects to real-time APIs (Network Rail Open Data Platform and Open Weather API) to fetch train and weather data.
- API Connectors: Two distinct connectors – one for train movements and one for weather data.
- Messaging System:
- Kafka: Acts as the intermediary messaging layer where both data streams (train and weather) are published. Represent separate topics for each data type.
- Data Storage/Processing Layer:
- Station Codes Table: Represented as a static seed (maintained via DBT) for station code to station name mapping.
- Data Transformation & Testing Layer:
- Apache Airflow: Orchestrates the pipeline using DAGs scheduled (e.g., every minute) to trigger the ETL steps.
- DBT: Handles data transformations (creating stg_train_movement, stg_weather_data, and the final fact table) and data quality tests.
- Visualization/Application Layer:
- Streamlit Dashboard: For interactive data visualization and user-driven analytics such as KPI visualizations (train delays, weather impacts, temporal analyses).
- Deployment & Monitoring:
- Astronomer Cloud: Deployed version of the Airflow DAGs with enterprise-grade monitoring solutions.

##### Relationships and Data Flow
- Airflow periodically triggering jobs that:
- Execute DBT transformations – consume L0 data, perform cleansing/data quality validations, and create L1 datasets.
- Join data from L1 to produce the final fact table (L2) for reporting.
- Finally, illustrate the dashboard (Streamlit) querying the data from Snowflake (L2) for visualizations.
- Optionally, include feedback loops like DBT tests and operational dashboards for data quality and pipeline health.

##### Architectural Patterns
- ETL Pipeline Pattern – with clear separation between data ingestion (raw vs. transformed layers) and a multi-layer (L0, L1, L2) data model.
- Real-time Streaming – with Kafka and API ingestion running continuously.
- Orchestration and Automation – implemented with Airflow in a DAG structure, emphasizing dependencies among tasks.
- Modular and Extensible Design – each layer (ingestion, transformation, visualization) is decoupled. DBT tests ensure data quality; adding new KPIs or ML models will require extension rather than complete redesign.

##### Relevant Technologies and Frameworks
- Kafka, Snowflake, Airflow, DBT, Streamlit, and supporting libraries (Snowpark & stomp.py) should be mentioned near their respective components.
- Optionally, add minor annotations regarding the use of Apache Flink as a potential future streaming technology (mentioned in next steps).

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        TC["Train Events API Connector"]:::ingestion
        WC["Weather Data API Connector"]:::ingestion
        SS["Stomp Server"]:::ingestion
    end

    %% Messaging System
    subgraph "Messaging System"
        KAFKA["Kafka"]:::messaging
        KFTrain["Kafka-to-Snowflake (Train)"]:::messaging
        KFWeather["Kafka-to-Snowflake (Weather)"]:::messaging
    end

    %% Data Storage/Processing (Snowflake)
    subgraph "Data Storage/Processing (Snowflake)"
        L0["L0: Raw Data Ingestion"]:::storage
        L1["L1: Staging Data"]:::storage
        L2["L2: Final Fact Table"]:::storage
        SC["Station Codes Table"]:::storage
    end

    %% Data Transformation & Orchestration
    subgraph "Data Transformation & Orchestration"
        AF["Apache Airflow (DAGs)"]:::orchestration
        DBT["DBT (Transformations & Testing)"]:::transformation
        subgraph "DBT Models & Tests"
            DBTM["DBT Models"]:::transformation
            ST["stg_train_movement.sql"]:::transformation
            SW["stg_weather_data.sql"]:::transformation
            FT["fact_city_movement.sql"]:::transformation
            DBTT["DBT Data Tests"]:::testing
        end
    end

    %% Visualization/Application Layer
    subgraph "Visualization/Application Layer"
        DASH["Streamlit Dashboard"]:::visualization
    end

    %% Deployment
    AC["Astronomer Cloud (Deployment)"]:::orchestration

    %% Connections - Data Ingestion to Messaging
    TC --> SS
    WC --> SS
    SS --> KAFKA

    %% Messaging to Storage
    KAFKA --> KFTrain
    KAFKA --> KFWeather
    KFTrain --> L0
    KFWeather --> L0

    %% Storage to Transformation Flow
    L0 --> DBT
    AF --> DBT

    %% DBT Transformation Steps
    DBT --> DBTM
    DBTM --> ST
    DBTM --> SW
    DBTM --> FT
    DBTM --> DBTT
    ST --> L1
    SW --> L1
    FT --> L2

    %% Visualization Flow
    L2 --> DASH

    %% Airflow Deployment
    AF --> AC

    %% Styles
    classDef ingestion fill:#FFDAB9,stroke:#333,stroke-width:2px;
    classDef messaging fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef storage fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef transformation fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef visualization fill:#DDA0DD,stroke:#333,stroke-width:2px;
    classDef testing fill:#F0E68C,stroke:#333,stroke-width:2px;

    %% Click Events
    click TC "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/scripts/train_events_kafka.py"
    click WC "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/scripts/weather_events_kafka.py"
    click KFTrain "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/scripts/train_events_sf.py"
    click KFWeather "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/scripts/weather_events_sf.py"
    click ST "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/models/staging/stg_train_movement.sql"
    click SW "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/models/staging/stg_weather_data.sql"
    click FT "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/models/marts/fact_city_movement.sql"
    click SC "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/seeds/station_codes.csv"
    click AF "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dags/lokesh/train_dag.py"
    click DASH "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/blob/main/dbt_project/scripts/dashboard.py"
    click DBTM "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/tree/main/dbt_project/models"
    click DBTT "https://github.com/karanamlokesh/dataexpert-lokesh-capstone/tree/main/dbt_project/data-tests"
```
