# ayanarshad02/hft_realtime_datapipeline architecture

> Architecture diagram of the GitHub repository ayanarshad02/hft_realtime_datapipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayanarshad02/hft_realtime_datapipeline
- Repository: https://github.com/ayanarshad02/hft_realtime_datapipeline

## Overview

##### Project Type and Purpose
- This is an MLOps framework that automates a real-time data pipeline for stock (HFT) data. It leverages ZenML to orchestrate the pipeline steps.
- The system is designed to fetch, ingest, preprocess (including handling missing values), perform feature engineering on stock data, and then store the processed data in a PostgreSQL database.
- Additionally, there is a mechanism (via a shell script) to schedule the pipeline daily using cron.

##### File Structure
- Top-Level Components:
- pipelines: Contains the pipeline orchestration file (run_pipeline.py) that ties together the steps.
- src: Houses the core business logic (data fetching, ingestion, handling missing values, transformation, and storing data).
- steps: Contains the ZenML step definitions corresponding to each processing stage.
- tests: Includes unit tests to ensure the correct behavior of critical functions in the src folder.
- utils: Contains configuration and logging utilities, used across the pipeline.
- setup.py and setup_daily_pipeline.sh: For packaging the application and scheduling daily pipeline runs, respectively.

- Architectural Insights:
- The use of separate directories for src and steps indicates a clear separation between implementation logic and ZenML step wrappers.
- The pipelines directory centralizes the orchestration and is key for understanding overall workflow.
- Presence of configuration files (utils/config.py) and logging (utils/logger.py) underline best practices in monitoring and managing pipeline execution.

##### Summary of Components
- External Data Source(s) → Fetch Data (src/fetch_data.py & steps/fetch_data_step.py)
- Ingest Data (src/ingest_data.py & steps/ingest_data_step.py)
- Handle Missing Values (src/handle_missing_value.py & steps/handle_missing_value_step.py)
- Feature Engineering (src/feature_engineering.py & steps/feature_engineering_step.py)
- Store Processed Data (src/storing_preprocessed_data.py & steps/storing_preprocessed_data_step.py) → PostgreSQL Database
- ZenML Orchestrator (pipelines/run_pipeline.py) managing the above steps.
- Cron Job Scheduler (setup_daily_pipeline.sh) triggering pipeline executions.
- Support Utilities (utils/config.py, utils/logger.py) used across the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    Cron["Cron Job Scheduler"]:::cron
    ZenML["ZenML Orchestrator"]:::orchestrator
    Ext["Stock Data Source"]:::external

    subgraph "Fetch Data"
        F1["Fetch Data (Logic)"]:::step
        F2["Fetch Data (Step)"]:::step
        F1 --> F2
    end

    subgraph "Ingest Data"
        I1["Ingest Data (Logic)"]:::step
        I2["Ingest Data (Step)"]:::step
        I1 --> I2
    end

    subgraph "Handle Missing Values"
        M1["Handle Missing Values (Logic)"]:::step
        M2["Handle Missing Values (Step)"]:::step
        M1 --> M2
    end

    subgraph "Feature Engineering"
        FE1["Feature Engineering (Logic)"]:::step
        FE2["Feature Engineering (Step)"]:::step
        FE1 --> FE2
    end

    subgraph "Store Processed Data"
        S1["Store Processed Data (Logic)"]:::step
        S2["Store Processed Data (Step)"]:::step
        S1 --> S2
    end

    DB["PostgreSQL Database"]:::db

    subgraph "Utilities"
        U1["Configuration Utility"]:::utility
        U2["Logging Utility"]:::utility
    end

    %% Data Flow Connections
    Cron --> ZenML
    ZenML --> F2
    Ext --> F1
    F2 --> I1
    ZenML --> I2
    I2 --> M1
    ZenML --> M2
    M2 --> FE1
    ZenML --> FE2
    FE2 --> S1
    ZenML --> S2
    S2 --> DB

    %% Utility connections
    ZenML --> U1
    ZenML --> U2

    %% Click Events
    click F1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/src/fetch_data.py"
    click F2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/steps/fetch_data_step.py"
    click I1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/src/ingest_data.py"
    click I2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/steps/ingest_data_step.py"
    click M1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/src/handle_missing_value.py"
    click M2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/steps/handle_missing_value_step.py"
    click FE1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/src/feature_engineering.py"
    click FE2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/steps/feature_engineering_step.py"
    click S1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/src/storing_preprocessed_data.py"
    click S2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/steps/storing_preprocessed_data_step.py"
    click ZenML "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/pipelines/run_pipeline.py"
    click Cron "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/setup_daily_pipeline.sh"
    click U1 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/utils/config.py"
    click U2 "https://github.com/ayanarshad02/hft_realtime_datapipeline/blob/main/utils/logger.py"

    %% Styles
    classDef external fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef orchestrator fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef step fill:#A9DFBF,stroke:#333,stroke-width:2px;
    classDef utility fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef db fill:#F5B7B1,stroke:#333,stroke-width:2px;
    classDef cron fill:#F9E79F,stroke:#333,stroke-width:2px;
```
