# tigisthailay/stream-processing-pipeline- architecture

> Architecture diagram of the GitHub repository tigisthailay/stream-processing-pipeline-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tigisthailay/stream-processing-pipeline-
- Repository: https://github.com/tigisthailay/stream-processing-pipeline-

## Overview

##### Project type and purpose
- This is a real-time streaming data pipeline whose goal is to ingest financial tick data from the Finnhub WebSocket API, process it with Apache Spark, and store results in Snowflake.
- Key technologies: Docker Compose, Python, Apache Kafka, Apache Spark, Snowflake.

##### File structure
- Top-level artifacts:
- Finnhub_Producer/            ← Python service that connects to Finnhub and produces messages to Kafka
- stream_processing/           ← Spark streaming job that reads from Kafka, transforms, writes to Snowflake
- docker-compose.yaml          ← orchestrates all containers and networking
- README.md/.env               ← configuration and environment variable definitions
- Inside Finnhub_Producer: Dockerfile, requirements.txt, Finnhub_Producer.py, utils.py
- Inside stream_processing: Dockerfile, requirements.txt, stream_processing.py, Spark and Snowflake connector JARs

##### Extract architecture insights from README
- Data flow: Finnhub WebSocket → Kafka topic(s) → Spark Structured Streaming → Snowflake
- External services: Snowflake data warehouse, Finnhub API
- Monitoring & logging: Kafka UI, Docker logs

##### Main components
a. Finnhub Producer Container
- Connects via WebSocket to Finnhub API
- Publishes JSON messages into Kafka “ingest” topic
- Tech: Python, websockets library, kafka-python
b. Apache Kafka Cluster (Docker service)
- Single-broker setup via Compose
- Stores raw tick data in a topic
- Exposes bootstrap server to producers and consumers
c. Kafka UI (optional monitoring service on 8083)
d. Spark Streaming Container
- Uses spark-submit to run stream_processing.py
- Reads from Kafka ingest topic, applies transformations
- Writes enriched records to Kafka sink topic or directly to Snowflake
- Tech: Spark Structured Streaming, spark-sql-kafka, spark-snowflake connectors
e. Snowflake Data Warehouse (external service)
- Receives upserts from Spark via JDBC connector
- Houses the processed financial data for analytics and BI

##### Relationships and data flows
- Optionally: Spark Streaming job → Kafka sink topic (for downstream consumers)

##### Architectural patterns and design principles
- Containerized micro-services for isolation (each major component in its own Docker container)
- Decoupling via asynchronous messaging (Kafka acts as the buffer between producer and processor)
- Single responsibility: Producer only ingests, Spark only transforms, Snowflake only stores

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Docker Compose"
        direction TB
        Producer["Finnhub Producer Container"]:::ingestion
        Kafka["Apache Kafka Broker\n(Port 9092)"]:::ingestion
        UI["Kafka UI\n(Port 8083)"]:::monitoring
        Spark["Spark Streaming Container"]:::processing
    end

    ExternalFinnhub["Finnhub API"]:::external
    ExternalSnowflake["Snowflake Data Warehouse"]:::external
    Orchestration["docker-compose.yaml"]:::docs
    Documentation["README.md/.env"]:::docs
    Screenshots["screenshots/"]:::docs

    ExternalFinnhub -->|"ws://finnhub.io"| Producer
    Producer -->|"JSON tick data"| Kafka
    Kafka -->|"JSON tick data"| Spark
    Spark -->|"Snowflake JDBC"| ExternalSnowflake
    UI -.->|"monitors topics"| Kafka

    Producer --> Orchestration
    Kafka --> Orchestration
    UI --> Orchestration
    Spark --> Orchestration
    Documentation --> Orchestration
    Screenshots --> Orchestration

    click Producer "https://github.com/tigisthailay/stream-processing-pipeline-/tree/main/Finnhub_Producer/"
    click Spark "https://github.com/tigisthailay/stream-processing-pipeline-/tree/main/stream_processing/"
    click Orchestration "https://github.com/tigisthailay/stream-processing-pipeline-/blob/main/docker-compose.yaml"
    click Documentation "https://github.com/tigisthailay/stream-processing-pipeline-/blob/main/README.md"
    click Documentation "https://github.com/tigisthailay/stream-processing-pipeline-/blob/main/.env"
    click Screenshots "https://github.com/tigisthailay/stream-processing-pipeline-/tree/main/screenshots/"

    classDef ingestion fill:#D6E9C6,stroke:#3C763D,color:#3C763D
    classDef processing fill:#D9EDF7,stroke:#31708F,color:#31708F
    classDef storage fill:#FCF8E3,stroke:#8A6D3B,color:#8A6D3B
    classDef monitoring fill:#F2DEDE,stroke:#A94442,color:#A94442,stroke-dasharray: 5 5
    classDef external fill:#ECECEC,stroke:#666,color:#333,stroke-dasharray: 2 2
    classDef docs fill:#FFF2CC,stroke:#8A6D3B,color:#8A6D3B,stroke-dasharray: 2 2
```
