How karanamlokesh/dataexpert-lokesh-capstone works
Based on our review of the file tree and the README, this project is a real-time data pipeline and analytics system focused on ingesting, processing, and visualizing train movement and weather data. The system leverages a suite of technologies—including Kafka for messaging, Snowflake as a data warehouse, Airflow for orchestration, DBT for transformations and testing, and Streamlit for dashboards—to provide real-time monitoring and analytics for train delays and utilization. The architectural design employs a layered data model (L0, L1, and L2) and utilizes separate components to handle real-time ingestion from public data sources and subsequent data transformation, quality tests, and reporting.
Main Components
• Data Ingestion Layer:
• Stomp Server: Continuously connects to real-time APIs (Network Rail Open Data Platform and Open Weather API) to fetch train and weather data.
• API Connectors: Two distinct connectors – one for train movements and one for weather data.
• Messaging System:
• Kafka: Acts as the intermediary messaging layer where both data streams (train and weather) are published. Represent separate topics for each data type.
• Data Storage/Processing Layer:
• Station Codes Table: Represented as a static seed (maintained via DBT) for station code to station name mapping.
• Data Transformation & Testing Layer:
• Apache Airflow: Orchestrates the pipeline using DAGs scheduled (e.g., every minute) to trigger the ETL steps.
• DBT: Handles data transformations (creating stg_train_movement, stg_weather_data, and the final fact table) and data quality tests.
• Visualization/Application Layer:
• Streamlit Dashboard: For interactive data visualization and user-driven analytics such as KPI visualizations (train delays, weather impacts, temporal analyses).
• Deployment & Monitoring:
• Astronomer Cloud: Deployed version of the Airflow DAGs with enterprise-grade monitoring solutions.
Relationships and Data Flow
• Airflow periodically triggering jobs that:
• Execute DBT transformations – consume L0 data, perform cleansing/data quality validations, and create L1 datasets.
• Join data from L1 to produce the final fact table (L2) for reporting.
• Finally, illustrate the dashboard (Streamlit) querying the data from Snowflake (L2) for visualizations.
• Optionally, include feedback loops like DBT tests and operational dashboards for data quality and pipeline health.
Architectural Patterns
• ETL Pipeline Pattern – with clear separation between data ingestion (raw vs. transformed layers) and a multi-layer (L0, L1, L2) data model.
• Real-time Streaming – with Kafka and API ingestion running continuously.
• Orchestration and Automation – implemented with Airflow in a DAG structure, emphasizing dependencies among tasks.
• Modular and Extensible Design – each layer (ingestion, transformation, visualization) is decoupled. DBT tests ensure data quality; adding new KPIs or ML models will require extension rather than complete redesign.
Relevant Technologies and Frameworks
• Kafka, Snowflake, Airflow, DBT, Streamlit, and supporting libraries (Snowpark & stomp.py) should be mentioned near their respective components.
• Optionally, add minor annotations regarding the use of Apache Flink as a potential future streaming technology (mentioned in next steps).