How kobinabrandon/hourly-divvy-trip-predictor works
Main Components Overview
a) Feature Pipeline
b) Training Pipeline
c) Inference Pipeline
Detailed Component Breakdown
Feature Pipeline Section:
• Preprocessing components showing:
• Data sourcing (from data_sourcing.py)
• Preprocessing (from preprocessing.py)
• Feature engineering (from feature_engineering.py)
• Indexing components (mixed_indexer.py and rounding_indexer.py)
• Data flow to Hopsworks Feature Store
Training Pipeline Section:
• Components for:
• Model architecture selection (models.py)
• Hyperparameter tuning component (hyperparameter_tuning.py)
• Training process (training.py)
• Connection to CometML for model registry
• Model versioning and tracking
Inference Pipeline Section:
• Split into Backend and Frontend subsections:
Backend:
• Feature store API interaction
• Model registry API interaction
• Inference engine
• Backfill mechanism
Frontend:
• Main dashboard (main.py)
• Predictions view
• Monitoring dashboard
• Data visualization components
Integration Points and Data Flow
Integration points with external services:
• Hopsworks Feature Store
• CometML Model Registry
• GitHub Actions for automated backfilling
Specific Design Elements
Use the following visual elements:
Data Sources → Feature Pipeline → Feature Store → Training Pipeline → Model Registry → Inference Pipeline → Frontend
Monitoring and Logging
A cross-cutting concern layer showing:
• Monitoring components (monitoring.py)
• Performance tracking
• Error logging
Key Technologies to Highlight
• Python (core language)
• Streamlit (frontend)
• Hopsworks (feature store)
• CometML (model registry)
• Docker (containerization)
• GitHub Actions (automation)