# lakku153/sml-project-2023-manfredi-meneghin architecture

> Architecture diagram of the GitHub repository lakku153/sml-project-2023-manfredi-meneghin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lakku153/sml-project-2023-manfredi-meneghin
- Repository: https://github.com/lakku153/sml-project-2023-manfredi-meneghin

## Overview

##### Project Type and Purpose
- “DeLight” is a serverless machine‐learning pipeline for predicting flight delays at Stockholm Arlanda Airport.
- It ingests historical and real‐time data from external APIs (weather and flight), processes features, trains daily models, serves batch predictions, and offers a Gradio UI on Hugging Face.

##### Key Architectural Components
a. External Data Sources
- Historical: SMHI OpenData Grid Archive (GRIB), Zyla Flight History API (JSON)
- Real‐Time: SMHI Forecast & MetAnalysis APIs, Swedavia Flight Info v2 API
b. Data Ingestion & Feature Pipelines (Serverless compute via Modal / local scripts)
- Historical Feature Pipeline
– Collector (download & store raw data)
– Extractor (convert GRIB/JSON → CSV)
– Processor (normalize, timestamp adjust, rotate wind vectors)
– Merger (join weather+flight historic features)
– EDA & Uploader to Hopsworks Feature Store
- Real‐Time Feature Pipeline
– Daily backfill script: fetch yesterday’s/today’s/tomorrow’s data → process → upload to Feature Store
c. Feature Store & Model Registry
- Hopsworks Feature Store holds evolving feature groups
- Hopsworks Model Registry stores and version‐controls XGBoost models
d. Training Pipeline
- Initialization (download feature group, perform model‐specific transformations)
- Daily Training (GridSearchCV hyperparameter tuning + XGBoost training)
- Upload updated model to Model Registry
e. Inference Pipeline
- Batch inference at midnight (collect forecast & schedule, download latest model, predict delays)
- Store predictions in Hopsworks File System
f. User Interface
- Gradio app hosted on Hugging Face Spaces
- Fetches predictions & flight schedules for user queries and visualizes results

##### Data Flow & Interactions
- Flow into Historical & Real‐Time Feature Pipelines → Hopsworks Feature Store
- From Feature Store → Training Pipeline → Hopsworks Model Registry
- From Model Registry + New Data → Inference Pipeline → Hopsworks FS
- Predictions → Gradio UI → End User

##### Architectural Patterns & Principles
- Serverless orchestration via Modal for scheduled jobs
- Separation of concerns: distinct pipelines for feature engineering, training, inference
- Use of managed MLOps platform (Hopsworks) for feature & model management
- Modular code organization under src/{feature_pipeline, training_pipeline, inference_pipeline, user_interface}

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        direction TB
        SMHI_Hist["SMHI OpenData Grid Archive (GRIB)"]:::external
        Zyla_Hist["Zyla Flight History API (JSON)"]:::external
        SMHI_RT["SMHI Forecast & MetAnalysis APIs"]:::external
        Swedavia["Swedavia Flight Info v2 API"]:::external
    end

    %% Raw Data Storage
    RawFlightHistoric["Raw Flight Historical Data"]:::storage
    RawWeatherHistoric["Raw Weather Historical Data"]:::storage

    %% Extracted Data
    ExtractFlightHistoric["Extracted Flight Historical Data"]:::storage
    ExtractWeatherHistoric["Extracted Weather Historical Data"]:::storage

    %% Processed Data
    ProcFlightHistoric["Processed Flight Historical Data"]:::storage
    ProcWeatherHistoric["Processed Weather Historical Data"]:::storage

    %% Historical Feature Pipeline
    subgraph "Historical Feature Pipeline" 
        direction TB
        CollectorFlight["Collector (Flight)"]:::ingestion
        ExtractorFlight["Extractor (Flight)"]:::ingestion
        ProcessorFlight["Processor (Flight)"]:::ingestion
        CollectorWeather["Collector (Weather)"]:::ingestion
        ExtractorWeather["Extractor & Time Shifter (Weather)"]:::ingestion
        Merger["Merger"]:::ingestion
        EDA["EDA"]:::ingestion
        Uploader["Uploader to Feature Store"]:::ingestion
    end

    %% Real-Time Feature Pipeline
    subgraph "Real-Time Feature Pipeline"
        direction TB
        RealTimeBackfill["Daily Backfill Script"]:::ingestion
    end

    %% Feature Store & Model Registry
    FeatureStore["Hopsworks Feature Store"]:::storage
    ModelRegistry["Hopsworks Model Registry"]:::storage

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        Init["Initialization"]:::training
        ModelTraining["Hyperparameter Tuning & Model Training"]:::training
        TrainingOrch["Daily Training Orchestration"]:::training
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        BatchInference["Batch Inference at Midnight"]:::inference
    end

    %% Prediction Storage
    PredictionFS["Hopsworks File System (Predictions)"]:::storage

    %% User Interface
    subgraph "User Interface"
        direction TB
        GradioUI["Gradio App (Hugging Face)"]:::ui
        EndUser(["End User"]):::ui
    end

    %% Connections
    SMHI_Hist -->|GRIB| RawWeatherHistoric
    Zyla_Hist -->|JSON| RawFlightHistoric
    RawWeatherHistoric --> ExtractorWeather
    RawFlightHistoric --> ExtractorFlight
    ExtractorWeather --> ProcWeatherHistoric
    ExtractorFlight --> ProcFlightHistoric
    ProcWeatherHistoric --> Merger
    ProcFlightHistoric --> Merger
    Merger --> EDA
    EDA --> Uploader
    Uploader --> FeatureStore

    SMHI_RT --> RealTimeBackfill
    Zyla_Hist --> RealTimeBackfill
    RealTimeBackfill --> FeatureStore

    FeatureStore --> Init
    Init --> ModelTraining
    ModelTraining --> TrainingOrch
    TrainingOrch --> ModelRegistry

    SMHI_RT --> BatchInference
    Swedavia --> BatchInference
    BatchInference --> ModelRegistry
    BatchInference --> PredictionFS

    PredictionFS --> GradioUI
    GradioUI --> EndUser

    %% Click Events
    click RawFlightHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/flight_historical_data/raw_flight_historical_data.zip"
    click RawWeatherHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/weather_historical_data/raw_weather_historical_data.csv"
    click ExtractFlightHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/flight_historical_data/extracted_flight_historical_data.json"
    click ExtractWeatherHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/weather_historical_data/extracted_weather_historical_data.csv"
    click ProcFlightHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/flight_historical_data/processed_flight_historical_data.csv"
    click ProcWeatherHistoric "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/datasets/weather_historical_data/processed_weather_historical_data.csv"
    click CollectorFlight "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_historical_flight/feature_pipeline_flightAPI_historical_collector.py"
    click ExtractorFlight "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_historical_flight/feature_pipeline_flightAPI_historical_extractor.py"
    click ProcessorFlight "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_historical_flight/feature_pipeline_flightAPI_historical_processor.py"
    click CollectorWeather "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_historical_weather/feature_pipeline_weatherAPI_historical_collector.py"
    click ExtractorWeather "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_historical_weather/feature_pipeline_weatherAPI_historical_extractor.py"
    click Merger "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_flightWeather_historical_merger.py"
    click EDA "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_flightWeather_EDA.py"
    click Uploader "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_historical/feature_pipeline_flightWeather_historical_uploader.py"
    click RealTimeBackfill "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/feature_pipeline/feature_pipeline_realtime/backfill_feature_pipeline_flightWeather_daily.py"
    click Init "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/training_pipeline/training_pipeline_flightWeather_initializer.py"
    click ModelTraining "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/training_pipeline/training_pipeline_flightWeather_model_selection_and_evaluation.py"
    click TrainingOrch "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/training_pipeline/training_pipeline_flightWeather_daily.py"
    click BatchInference "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/inference_pipeline/batch_inference_pipeline_flightDelay_daily.py"
    click GradioUI "https://github.com/lakku153/sml-project-2023-manfredi-meneghin/blob/main/src/user_interface_hugging_face/app.py"

    %% Styles
    classDef external fill:#E0F7FA,stroke:#006064,color:#006064
    classDef storage fill:#CFD8DC,stroke:#455A64,color:#455A64,shape:cylinder
    classDef ingestion fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1
    classDef training fill:#C8E6C9,stroke:#1B5E20,color:#1B5E20
    classDef inference fill:#FFE0B2,stroke:#E65100,color:#E65100
    classDef ui fill:#E1BEE7,stroke:#4A148C,color:#4A148C
    class SMHI_Hist,Zyla_Hist,SMHI_RT,Swedavia external
    class RawFlightHistoric,RawWeatherHistoric,ExtractFlightHistoric,ExtractWeatherHistoric,ProcFlightHistoric,ProcWeatherHistoric,FeatureStore,ModelRegistry,PredictionFS storage
    class CollectorFlight,ExtractorFlight,ProcessorFlight,CollectorWeather,ExtractorWeather,Merger,EDA,Uploader,RealTimeBackfill ingestion
    class Init,ModelTraining,TrainingOrch training
    class BatchInference inference
    class GradioUI,EndUser ui
```
