How anvayb/skystream works
Project Type and Purpose
• SkyStream is a data-driven analytics and prediction platform (not a traditional multi-tier web app).
• Core functions:
• Batch data ingestion & exploration (EDA & Models notebooks)
• Machine-learning model training & serving (delay predictor)
• Interactive dashboards for visualization (Streamlit app)
File Structure
Top-level:
• “EDA & Models/” – Jupyter notebook for exploratory data analysis and model development.
• “Streamlit/” – Python code (Visualization Dashboard.py), notebooks (Probability Predictor.ipynb), PDFs of intended UI mockups.
• dataset.txt – pointer to raw datasets.
• README & project presentation PDFs.
No separate “frontend” or “backend” folders—instead, Streamlit combines front-end UI and Python code.
README for Architecture Insights
• Data sources: Combined Flights 2021/2022 CSVs (Kaggle).
• Dependencies: Python 3.8+, PySpark, Streamlit, scikit-learn (inferred).
• Runtime: Local (Flask/Streamlit on port 5000).
Main Components
a. Data Sources
• External CSV datasets (Kaggle)
• (Optionally) Real-time feed or API for weather data
b. Data Ingestion & Processing
• PySpark jobs or Pandas pipelines in notebooks
• ETL transformations, feature engineering
c. Model Training & Storage
• Jupyter environment where ML models are trained
• Model artifacts persisted (Pickle, MLflow, etc.)
d. Model Serving / Prediction Engine
• Python module (Probability Predictor.ipynb / package)
• Exposes prediction API or function calls
e. Visualization Dashboard (Streamlit App)
• Single-page UI combining:
– Flight Delay Dashboard (maps, charts)
– Delay Predictor form & output
• Hosted locally (Flask/Streamlit server on :5000)
Relationships & Data Flows
• Raw CSVs → (1) EDA & Model Training
• Feature store & model artifacts → (2) Deployed to the Prediction Engine
• User events (UI inputs) → Prediction Engine → UI update
• Periodic batch jobs → refresh datasets or model retraining
Architectural Patterns & Design Principles
• Modular separation:
• Batch (EDA & training) vs. real-time (prediction & visualization)
• Reusable ML component: training pipeline decoupled from serving code
• Single page app: Streamlit functions as both UI and lightweight backend
Final Checklist
• Components: Data Sources, ETL/Processing, Model Training, Model Store, Prediction Engine, Dashboard
• Labels: component names + key frameworks