How amulyaprasanth/hourly_amazon_stock_prediction works
Project Type & Purpose
• This is a full-stack machine-learning application that:
• Ingests historical Amazon stock data
• Runs feature-engineering and model-training pipelines
• Executes batch inference on an hourly cadence
• Exposes results via a lightweight dashboard (Streamlit)
• Uses GitHub Actions to automate pipeline runs and dashboard deployment
Key Components (Based on File Structure)
a. Data & Configuration
• External Data Source (e.g., AlphaVantage / Yahoo Finance API)
• Local/Cloud Storage for raw data & feature store (not explicit in code but implied)
• config/config.yml
b. Feature Engineering
• amazonstockprediction/feature_pipeline.py
• amazonstockprediction/backfill_features.py
c. Model Training
• amazonstockprediction/training_pipeline.py
d. Batch Inference
• amazonstockprediction/batch_inference_pipeline.py
e. Logging & Utilities
• amazonstockprediction/logger.py
• amazonstockprediction/utils.py
f. Dashboard
• dashboard/app.py
• dashboard/requirements.txt
g. Orchestration & CI/CD
• .github/workflows/train_pipeline.yml
• .github/workflows/feature_and_inference_pipeline.yml
• .github/workflows/reboot_streamlit.yml
h. Notebooks & Tests
• notebooks/*.ipynb (exploratory & demo)
• tests/test_utils.py
Architectural Patterns & Relationships
• Pipelines as Modular Blocks: Each pipeline (features → training → inference) is a self-contained Python script.
• Sequential Data Flow: Raw data → feature store → model training → model artifacts → batch inference → predictions storage → dashboard.
• Orchestration Layer: GitHub Actions schedules and triggers each pipeline; also handles dashboard (re)deployment.
• Config-Driven: Shared config.yml drives parameters for all pipelines, ensuring consistency.
• Testing: A lightweight test suite validates utilities.
Detailing Each Component
• Feature Engineering:
• backfill_features.py (initial backfill)
• feature_pipeline.py (hourly updates)
• Training Pipeline:
• training_pipeline.py (consumes latest features → outputs model artifact)
• Batch Inference:
• batch_inference_pipeline.py (loads model → generates predictions)
• Dashboard:
• app.py (Streamlit; pulls predictions and visualizes)
• Orchestration:
• GitHub Actions workflows calling Python modules on schedule or event
• Configuration:
• config.yml (single source of truth for database paths, API keys, model hyperparameters)