# bmd1905/customer-purchase-prediction-ml-system architecture

> Architecture diagram of the GitHub repository bmd1905/customer-purchase-prediction-ml-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bmd1905/customer-purchase-prediction-ml-system
- Repository: https://github.com/bmd1905/customer-purchase-prediction-ml-system

## Overview

##### Project Type and Purpose
- This project is a full-scale, production-grade MLOps pipeline built around real-time e-commerce behavior data processing.
- It handles data ingestion from Kafka and PostgreSQL CDC, schema validation through Flink, transformation using Spark Streaming, feature storage (both online in Redis and offline in PostgreSQL/DWH), distributed training via Ray and MLflow integration, and modern model serving with Ray Serve and a FastAPI-based feature service. In parallel, observability tools (SigNoz, Prometheus, Grafana, Superset, etc.) and an NGINX proxy provide monitoring, alerting, and secure access.

##### Establishing Relationships and Data Flow
- Data ingestion flows from Kafka producers and CDC (originating from PostgreSQL changes) into respective Kafka topics.
- Flink’s schema validation subscribes to these topics, then re-publishes to “validated” and “invalid” topics.
- Validated data then flows into two branches: one going to the Data Lake (MinIO) and eventually the Data Warehouse via the Airflow-managed data_pipeline.
- Simultaneously, Spark Streaming jobs read from the “validated” topic, process the data, and push features to both the offline store (PostgreSQL) and online store (Redis).
- From the warehouse/offline store, data is pulled by the Airflow-managed training pipeline, which funnels data into distributed training (Ray Cluster) and logs results into MLflow.
- Post training, the model is loaded via Ray Serve to power the prediction service, while the feature service separately exposes real-time features via FastAPI.
- Observability tools (SigNoz, Grafana, Prometheus) wrap around these pipelines, monitoring metrics, logs, and even Ray Dashboard metrics.
- Integration points like model versioning (MLflow and Model Registry) and the orchestrated deployment (via Docker Compose and Airflow).

##### Architectural Patterns and Design Principles to Highlight
- Microservices Architecture: Demonstrate how each module (e.g., CDC, feature store, model registry, serving API) operates as an independently containerized service.
- Streaming and Batch Data Processing: Clearly separate the real-time streaming side (Kafka, Flink, Spark Streaming) from the batch/analytical side (Data Warehouse and offline store).
- Distributed and Scalable Training/Serving: Indicate the use of Ray for both distributed training, hyperparameter tuning, and scalable model serving.
- Containerization and Orchestration: Emphasize that all components are containerized, with individual Docker Compose files deployed across a common network.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Pipeline
    subgraph "Data Pipeline"
        direction TB
        KP("Kafka Producer"):::ingestion
        click KP "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/producer"
        CDC("CDC Service"):::ingestion
        click CDC "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/cdc"
        KI("Kafka Infrastructure"):::ingestion
        click KI "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/docker-compose.kafka.yaml"
        
        KT("Kafka Topics"):::ingestion
        
        SV("Flink Schema Validation Job"):::ingestion
        click SV "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/streaming/jobs/schema_validation_job.py"
        AI("Alert Invalid Events Job"):::ingestion
        click AI "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/streaming/jobs/alert_invalid_events_job.py"
        
        VT("Validated Topic"):::ingestion
        IT("Invalid Topic"):::ingestion
        
        M("Data Lake (MinIO)"):::ingestion
        click M "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/docker-compose.data-lake.yaml"
        SS("Spark Streaming Job"):::ingestion
        
        OF("Offline Feature Store"):::ingestion
        click OF "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/feature_stores/offline_write_batch.py"
        SQL("Feature Store SQL"):::ingestion
        click SQL "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/feature_stores/sql"
        ON("Online Feature Store"):::ingestion
        click ON "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/feature_stores/api.py"
        FS("FastAPI Feature Service"):::serving
        click FS "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/serving/online_feature_service.py"
        
        DWH("PostgreSQL DWH"):::ingestion
        click DWH "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/docker-compose.dwh.yaml"
        LDWH("Load_to_DWH Script"):::ingestion
        click LDWH "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/orchestration/dags/data_pipeline/gold/load_to_dwh.py"
        
        %% Connections for Data Pipeline
        KP --> KT
        CDC --> KT
        KI --> KT
        KT -->|"raw_events"| SV
        KT -->|"raw_events"| AI
        SV --> VT
        AI --> IT
        VT --> M
        VT --> SS
        SS --> OF
        SS --> ON
        ON --> FS
        M --> LDWH
        LDWH --> DWH
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        ADF("Airflow Orchestration"):::training
        click ADF "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/orchestration/dags"
        DWH --- ADF
        OF --- ADF
        
        RC("Ray Cluster (Distributed Training)"):::training
        click RC "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/ray"
        ADF --> RC
        
        ML("MLflow Model Registry"):::training
        click ML "https://github.com/bmd1905/customer-purchase-prediction-ml-system/tree/main/src/model_registry"
        RC --> ML
    end

    %% Serving Pipeline
    subgraph "Serving Pipeline"
        direction TB
        RS("Ray Serve"):::serving
        click RS "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/docker-compose.serving.yaml"
        ML --> RS
        
        PS("Prediction Service"):::serving
        click PS "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/serving/main.py"
        click PS "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/src/serving/prediction_service.py"
        RS --> PS
    end

    %% Observability & Infrastructure
    subgraph "Observability & Infrastructure"
        direction TB
        NGINX("NGINX Proxy Manager"):::observability
        click NGINX "https://github.com/bmd1905/customer-purchase-prediction-ml-system/blob/main/docker-compose.nginx.yaml"
        Sig("SigNoz"):::observability
        Prom("Prometheus"):::observability
        Graf("Grafana"):::observability
        Sup("Superset"):::observability
        
        ADF --- Sig
        ADF --- Prom
        ADF --- Graf
        RC --- Prom
        M --- Graf
        NGINX --- Graf
    end

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#007bff,stroke-width:2px;
    classDef training fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef serving fill:#fff3cd,stroke:#fd7e14,stroke-width:2px;
    classDef observability fill:#e2d6f3,stroke:#6f42c1,stroke-width:2px;
```
