# joshua-omolewa/stock_streaming_pipeline_project architecture

> Architecture diagram of the GitHub repository joshua-omolewa/stock_streaming_pipeline_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joshua-omolewa/stock_streaming_pipeline_project
- Repository: https://github.com/joshua-omolewa/stock_streaming_pipeline_project

## Overview

##### Main Components Identification

a) Data Source Layer:
- Alpha Vantage Stock API
- Custom Stock API (FastAPI) hosted on EC2 with NGINX
- S3 Landing Zone for raw stock data

b) Data Ingestion Layer:
- Apache NiFi container
- MySQL Database container
- Debezium CDC container

c) Stream Processing Layer:
- Amazon MSK (Kafka)
- Amazon EMR with Spark Streaming
- Apache Livy

d) Orchestration Layer:
- Apache Airflow
- AWS Lambda Functions

e) Storage & Processing Layer:
- S3 (transformed data with HUDI format)
- AWS Glue
- Amazon Athena

f) Visualization Layer:
- Power BI
- Tableau

##### Connection Flows

- Data flow paths
- Control flow paths (different line style)
- Latency or frequency indicators where applicable

##### Key Processes to Highlight

a) Ingestion Flow:
**Stock API → NiFi → MySQL → Debezium → MSK**

b) Processing Flow:
**MSK → Spark Streaming → S3 (HUDI) → Glue → Athena**

c) Orchestration Flow:
**API → Airflow → Lambda → Livy → EMR**

##### Additional Elements

- A monitoring section showing Great Expectations data quality checks
- Timing indicators for real-time aspects (30-second updates)
- Container boundaries for dockerized components

##### Technical Details

- EC2 instance groupings
- Docker container relationships
- MSK topic structure
- Security groups and network boundaries
- Data format transformations (CSV → HUDI)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Sources"
        AV[Alpha Vantage API]
        API[Custom Stock API]:::api
        S3L[(S3 Landing Zone)]:::aws
    end

    subgraph "Data Ingestion"
        NIFI[Apache NiFi]:::apache
        MYSQL[(MySQL Database)]:::database
        DEB[Debezium CDC]:::apache
    end

    subgraph "Stream Processing"
        MSK[Amazon MSK]:::aws
        EMR[EMR Spark Streaming]:::aws
        LIVY[Apache Livy]:::apache
    end

    subgraph "Orchestration"
        AIR[Apache Airflow]:::apache
        LAMBDA[AWS Lambda]:::aws
    end

    subgraph "Storage & Processing"
        S3H[(S3 HUDI Storage)]:::aws
        GLUE[AWS Glue]:::aws
        ATHENA[Amazon Athena]:::aws
    end

    subgraph "Visualization"
        PBI[Power BI]:::viz
        TAB[Tableau]:::viz
    end

    subgraph "Data Quality"
        GE[Great Expectations]:::quality
    end

    %% Data Flow Connections
    AV --> API
    API --> NIFI
    API --> S3L
    NIFI --> MYSQL
    MYSQL --> DEB
    DEB --> MSK
    MSK --> EMR
    EMR --> S3H
    S3H --> GLUE
    GLUE --> ATHENA
    ATHENA --> PBI
    ATHENA --> TAB

    %% Control Flow
    AIR --> LAMBDA
    LAMBDA --> LIVY
    LIVY --> EMR
    GE --> MYSQL
    GE --> S3H

    %% Component Mappings
    click API "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/api.py"
    click NIFI "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/Joshua-final-stocks-nifi.xml"
    click MYSQL "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/stock DML.sql"
    click EMR "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/spark-final.py"
    click LAMBDA "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/lamda_function.py"
    click AIR "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/aiflowlambda.py"
    click GE "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/tree/main/great expectation files/"
    click PBI "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/stocks.pbix"
    click TAB "https://github.com/Joshua-omolewa/Stock_streaming_pipeline_project/blob/main/joshuastock.twb"

    %% Styling
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef apache fill:#D22128,stroke:#232F3E,color:white
    classDef database fill:#326CE5,stroke:#232F3E,color:white
    classDef viz fill:#722ED1,stroke:#232F3E,color:white
    classDef api fill:#85EA2D,stroke:#232F3E,color:black
    classDef quality fill:#FF4D4F,stroke:#232F3E,color:white

    %% Legend
    subgraph Legend
        L1[AWS Services]:::aws
        L2[Apache Components]:::apache
        L3[Databases]:::database
        L4[Visualization]:::viz
        L5[APIs]:::api
        L6[Data Quality]:::quality
    end
```
