How paulescu/end-2-end-real-time-ml works
Main Components
• Dev Environment & Infrastructure:
• "devcontainer" (the pre-configured Docker-based development environment)
• .env configuration (for TurboML credentials)
• Data Ingestion & Feature Engineering:
• The datasets that comprise raw transaction data (as simulated through the script and dataset creation in setup_feature_pipeline.py)
• The feature engineering component (the logic encapsulated in setup_feature_pipeline.py which processes these datasets)
• Machine Learning Model Handling:
• Offline training and online deployment of the model (the setup_model.py script outlines how the model is defined and deployed)
• Model monitoring and comparison components, which are part of the TurboML dashboard and the iterative process described in the README
• Live Data Generation:
• The generate_live_data.py script that simulates a live data stream (representing real-time transaction data)
• External TurboML Platform:
• Represents the TurboML backend which handles model deployment, real-time predictions, monitoring, and connectors (data sources like Kafka or databases in a production scenario)
Relationships and Interactions Between Components
• There’s interaction between the setup_feature_pipeline.py (which creates and processes datasets) and the subsequent model training/online deployment via setup_model.py.
• Additionally, indicate the connection between the development environment (devcontainer) which hosts and runs the scripts locally, and the TurboML platform that handles the heavy lifting (deployment, monitoring, and scaling).
Architectural Patterns and Design Principles
• Modularity: Each major step (pipeline setup, model deployment, live predictions, monitoring, etc.) is handled by distinct Python scripts.
• Separation of Concerns: The project separates configuration (config.py), feature pipeline (setup_feature_pipeline.py), model deployment (setup_model.py), and live data simulation (generate_live_data.py).
• Containerization: Usage of a devcontainer ensures a reproducible development environment.
• Integration with a cloud-based service (TurboML) to manage infrastructure, serving as the backbone for real-time predictions and model monitoring.
Relevant Technologies and Tools
• Docker (via devcontainer) and VSCode (for the development environment)
• Python (all scripts and configuration files)
• TurboML platform (for model deployment, monitoring, and API endpoints)
• Makefile (to streamline running commands like “make feature-pipeline”, “make model”, and “make live-data”)
• Environment configuration (.env) to manage sensitive credentials