How ayanarshad02/hft_realtime_datapipeline works
Project Type and Purpose
• This is an MLOps framework that automates a real-time data pipeline for stock (HFT) data. It leverages ZenML to orchestrate the pipeline steps.
• The system is designed to fetch, ingest, preprocess (including handling missing values), perform feature engineering on stock data, and then store the processed data in a PostgreSQL database.
• Additionally, there is a mechanism (via a shell script) to schedule the pipeline daily using cron.
File Structure
• Top-Level Components:
• pipelines: Contains the pipeline orchestration file (run_pipeline.py) that ties together the steps.
• src: Houses the core business logic (data fetching, ingestion, handling missing values, transformation, and storing data).
• steps: Contains the ZenML step definitions corresponding to each processing stage.
• tests: Includes unit tests to ensure the correct behavior of critical functions in the src folder.
• utils: Contains configuration and logging utilities, used across the pipeline.
• setup.py and setup_daily_pipeline.sh: For packaging the application and scheduling daily pipeline runs, respectively.
• Architectural Insights:
• The use of separate directories for src and steps indicates a clear separation between implementation logic and ZenML step wrappers.
• The pipelines directory centralizes the orchestration and is key for understanding overall workflow.
• Presence of configuration files (utils/config.py) and logging (utils/logger.py) underline best practices in monitoring and managing pipeline execution.
Summary of Components
• External Data Source(s) → Fetch Data (src/fetch_data.py & steps/fetch_data_step.py)
• Ingest Data (src/ingest_data.py & steps/ingest_data_step.py)
• Handle Missing Values (src/handle_missing_value.py & steps/handle_missing_value_step.py)
• Feature Engineering (src/feature_engineering.py & steps/feature_engineering_step.py)
• Store Processed Data (src/storing_preprocessed_data.py & steps/storing_preprocessed_data_step.py) → PostgreSQL Database
• ZenML Orchestrator (pipelines/run_pipeline.py) managing the above steps.
• Cron Job Scheduler (setup_daily_pipeline.sh) triggering pipeline executions.
• Support Utilities (utils/config.py, utils/logger.py) used across the system.