How lakku153/sml-project-2023-manfredi-meneghin works
Project Type and Purpose
• “DeLight” is a serverless machine‐learning pipeline for predicting flight delays at Stockholm Arlanda Airport.
• It ingests historical and real‐time data from external APIs (weather and flight), processes features, trains daily models, serves batch predictions, and offers a Gradio UI on Hugging Face.
Key Architectural Components
a. External Data Sources
• Historical: SMHI OpenData Grid Archive (GRIB), Zyla Flight History API (JSON)
• Real‐Time: SMHI Forecast & MetAnalysis APIs, Swedavia Flight Info v2 API
b. Data Ingestion & Feature Pipelines (Serverless compute via Modal / local scripts)
• Historical Feature Pipeline
– Collector (download & store raw data)
– Extractor (convert GRIB/JSON → CSV)
– Processor (normalize, timestamp adjust, rotate wind vectors)
– Merger (join weather+flight historic features)
– EDA & Uploader to Hopsworks Feature Store
• Real‐Time Feature Pipeline
– Daily backfill script: fetch yesterday’s/today’s/tomorrow’s data → process → upload to Feature Store
c. Feature Store & Model Registry
• Hopsworks Feature Store holds evolving feature groups
• Hopsworks Model Registry stores and version‐controls XGBoost models
d. Training Pipeline
• Initialization (download feature group, perform model‐specific transformations)
• Daily Training (GridSearchCV hyperparameter tuning + XGBoost training)
• Upload updated model to Model Registry
e. Inference Pipeline
• Batch inference at midnight (collect forecast & schedule, download latest model, predict delays)
• Store predictions in Hopsworks File System
f. User Interface
• Gradio app hosted on Hugging Face Spaces
• Fetches predictions & flight schedules for user queries and visualizes results
Data Flow & Interactions
• Flow into Historical & Real‐Time Feature Pipelines → Hopsworks Feature Store
• From Feature Store → Training Pipeline → Hopsworks Model Registry
• From Model Registry + New Data → Inference Pipeline → Hopsworks FS
• Predictions → Gradio UI → End User
Architectural Patterns & Principles
• Serverless orchestration via Modal for scheduled jobs
• Separation of concerns: distinct pipelines for feature engineering, training, inference
• Use of managed MLOps platform (Hopsworks) for feature & model management
• Modular code organization under src/{feature_pipeline, training_pipeline, inference_pipeline, user_interface}