# minora2k4/stock-etl-pipeline architecture

> Architecture diagram of the GitHub repository minora2k4/stock-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minora2k4/stock-etl-pipeline
- Repository: https://github.com/minora2k4/stock-etl-pipeline

## Overview

##### Project type and purpose (from README + tree)
This is a **containerized, end-to-end real-time streaming “Lakehouse” pipeline** for market data (crypto + US equities):

**Finnhub WebSocket → Kafka → Spark Structured Streaming → MinIO (S3) Parquet → DuckDB SQL → Streamlit dashboard**

It’s not a monolith; it’s a **multi-service data platform** wired together via Docker Compose and streaming/storage interfaces.

Top-level signals that define architecture boundaries:

- **`kafka/`**: producer-side ingestion microservice
- `producer.py` = Finnhub WebSocket client + Kafka publisher
- `config.py`, `utils.py` = configuration/retry/serialization glue
- **`spark/`**: streaming ETL processor
- `processor.py` = Spark Structured Streaming job reading Kafka, transforming, writing Parquet to S3 (MinIO)
- **`duckdb/`**: analytics + BI application
- `analytics.py` = DuckDB connection + S3/MinIO configuration + SQL queries
- `data_visualization.py` = Streamlit UI + Plotly + auto-refresh

Also: each service has its own **Dockerfile** and **requirements.txt** → indicates clean service boundaries and independently buildable images.

Shows *runtime services*, network boundaries, and data flows.

Shows the internal modules in `kafka/`, `spark/`, and `duckdb/` and how they interact.

This maps directly to the file tree and avoids ambiguity.

##### Finnhub → kafka-producer
- Protocol: WebSocket
- Payload: JSON trades

##### kafka-producer → Kafka topic (`stock-ticks`)
- Protocol: Kafka producer API
- Payload: serialized JSON (note if raw or wrapped)

##### Spark Structured Streaming (spark-job) ← Kafka
- Protocol: Kafka consumer connector
- Data: JSON events

##### spark-job → MinIO
- Protocol: S3A (Spark `s3a://...`)
- Format: Parquet
- Path: `warehouse/processed_trades`
- Partitioning: `date`, `symbol`

##### DuckDB → MinIO
- Protocol: S3 (DuckDB httpfs / S3 config)
- Reads: Parquet files

##### Streamlit UI → DuckDB
- Calls into `analytics.py` query functions
- Output: aggregated results / time series for Plotly
- UI: auto-refresh

- `.env` → kafka-producer (token, tickers, kafka server/topic)
- `.env` → spark-job (kafka bootstrap, topic, S3 endpoint/keys)
- `.env` → duckdb/streamlit (MinIO endpoint/keys, paths)

##### Architectural patterns/design principles actually used here
Call these out explicitly in annotations/legends:

- **Decoupled streaming via Kafka** (producer/consumer independence, buffering, replay potential)
- **Lakehouse approach**: raw-ish events transformed into **columnar Parquet** in object storage; queried directly without a separate warehouse engine
- **Separation of Concerns in BI service** (`analytics.py` vs `data_visualization.py`)
- **Microservice-ish containers**: ingestion, processing, storage, analytics, UI all independently deployable
- **Near real-time ETL**: Spark micro-batching (5s trigger)

##### “detail callouts” as side notes
- Kafka topic name: `stock-ticks`
- Spark trigger: `5s`
- Output location: `s3a://warehouse/processed_trades`
- Partition keys: `date`, `symbol`
- Dashboard endpoint: `localhost:8501`
- MinIO console endpoint: `:9001`

Ports and depends_on as small annotations.
2. For each service, map it to its directory (`kafka/`, `spark/`, `duckdb/`) and list the entrypoint script(s).
- `kafka/producer.py` uses `config.py` + `utils.py` → Kafka
- `spark/processor.py` stages: read Kafka → parse JSON → enrich/compute → partition → write Parquet
- `duckdb/analytics.py` provides query API consumed by `data_visualization.py`

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Diagram 1: Container / Runtime Architecture (Docker Compose)
  %% =========================

  subgraph "External Systems"
    FINNHUB(("Finnhub\nWebSocket API")):::external
  end

  subgraph "Docker Compose Network"
    direction TB

    subgraph "Ingestion"
      direction TB
      KP["kafka-producer\n(Python: Finnhub WS → Kafka)"]:::ingest
      KP_CALL["Modules:\nproducer.py\nconfig.py\nutils.py"]:::code
    end

    subgraph "Streaming / Messaging"
      direction TB
      ZK["Zookeeper"]:::stream
      KB["Kafka Broker\nTopic: stock-ticks"]:::stream
    end

    subgraph "Processing (Spark Cluster)"
      direction TB
      SM["Spark Master"]:::compute
      SW["Spark Worker(s)"]:::compute
      SJ["spark-job\n(Spark Structured Streaming ETL)\nTrigger: 5s"]:::compute
      SJ_CALL["Module:\nprocessor.py"]:::code
    end

    subgraph "Storage (Lakehouse)"
      direction TB
      MINIO[("MinIO\n(S3-compatible Object Storage)\nConsole: :9001")]:::storage
      WH[(Bucket: warehouse\nDataset: processed_trades\nPartitions: date, symbol)]:::storageLite
      MSET["minio-setup\n(init job: create bucket/policy)"]:::init
    end

    subgraph "Analytics & BI"
      direction TB
      DADB["duckdb-analytics + streamlit\n(DuckDB SQL over Parquet + UI)\nDashboard: :8501"]:::bi
      BI_CALL["Modules:\nanalytics.py\ndata_visualization.py"]:::code
    end

    ENV[".env\n(tokens, tickers, brokers, S3 creds/endpoints)"]:::config
  end

  %% Primary data path (vertical)
  FINNHUB -->|"WebSocket(JSON_trades)"| KP
  KP -->|"Kafka_producer(JSON_events)"| KB
  KB -->|"Kafka_consumer_connector(JSON_events)"| SJ
  SJ -->|"S3A(Parquet)\ns3a://warehouse/processed_trades"| MINIO
  MINIO -->|"stores"| WH
  DADB -->|"S3(httpfs)\nread_Parquet"| MINIO

  %% Support/service relationships
  KB -->|"coordination"| ZK
  SM -->|"schedules"| SW
  SJ -->|"runs_on"| SM

  MSET -->|"init_bucket/policy"| MINIO

  %% Config flows
  ENV -.->|"FINNHUB_API_TOKEN\nFINNHUB_TICKERS\nKAFKA_TOPIC"| KP
  ENV -.->|"KAFKA_BOOTSTRAP\nKAFKA_TOPIC\nS3_ENDPOINT/KEY/SECRET"| SJ
  ENV -.->|"S3_ENDPOINT/KEY/SECRET\nwarehouse_path"| DADB

  %% Notes / callouts
  NOTE1["Notes:\n- Kafka buffers bursts/backpressure\n- Lakehouse: Parquet on object storage\n- DuckDB queries files directly\n- Streamlit auto-refresh dashboard"]:::note
  DADB -.->|"renders(Plotly)\nauto_refresh"| NOTE1

  %% =========================
  %% Diagram 2: Spark Structured Streaming ETL (inside spark-job)
  %% =========================
  subgraph "Spark ETL Flow (spark/processor.py)"
    direction TB
    S1["ReadStream\nKafka(topic=stock-ticks)"]:::compute
    S2["Parse JSON\n(schema)"]:::compute
    S3["Enrich/Compute\n(total_trade_value)"]:::compute
    S4["Partition\nby date, symbol"]:::compute
    S5["WriteStream\nParquet to S3A\nTrigger: 5s"]:::compute
  end

  KB -->|"source"| S1
  S1 -->|"events"| S2 -->|"rows"| S3 -->|"rows"| S4 -->|"batches"| S5
  S5 -->|"s3a://warehouse/processed_trades"| MINIO

  %% =========================
  %% Diagram 3: BI / Analytics Internals (DuckDB + Streamlit)
  %% =========================
  subgraph "BI Internals (duckdb/)"
    direction TB
    UI["Streamlit UI\n(data_visualization.py)\n- auto-refresh\n- Plotly charts"]:::bi
    QL["Query Layer\n(analytics.py)\n- S3/MinIO config\n- SQL queries"]:::bi
    DDB[(DuckDB Engine\nin-process)]:::db
  end

  UI -->|"calls(query_functions)"| QL
  QL -->|"executes(SQL)"| DDB
  DDB -->|"scan Parquet over S3"| MINIO

  %% =========================
  %% Click Events (component_mapping)
  %% =========================
  click DC "https://github.com/minora2k4/stock-etl-pipeline/blob/main/docker-compose.yml"
  click RD "https://github.com/minora2k4/stock-etl-pipeline/blob/main/README.md"

  click KP "https://github.com/minora2k4/stock-etl-pipeline/tree/main/kafka/"
  click KP_CALL "https://github.com/minora2k4/stock-etl-pipeline/tree/main/kafka/"
  click KPP "https://github.com/minora2k4/stock-etl-pipeline/blob/main/kafka/producer.py"
  click KPC "https://github.com/minora2k4/stock-etl-pipeline/blob/main/kafka/config.py"
  click KPU "https://github.com/minora2k4/stock-etl-pipeline/blob/main/kafka/utils.py"
  click KPD "https://github.com/minora2k4/stock-etl-pipeline/tree/main/kafka/Dockerfile"
  click KPR "https://github.com/minora2k4/stock-etl-pipeline/blob/main/kafka/requirements.txt"

  click SJ "https://github.com/minora2k4/stock-etl-pipeline/tree/main/spark/"
  click SJ_CALL "https://github.com/minora2k4/stock-etl-pipeline/blob/main/spark/processor.py"
  click SPD "https://github.com/minora2k4/stock-etl-pipeline/tree/main/spark/Dockerfile"
  click SPR "https://github.com/minora2k4/stock-etl-pipeline/blob/main/spark/requirements.txt"

  click DADB "https://github.com/minora2k4/stock-etl-pipeline/tree/main/duckdb/"
  click BI_CALL "https://github.com/minora2k4/stock-etl-pipeline/tree/main/duckdb/"
  click DQA "https://github.com/minora2k4/stock-etl-pipeline/blob/main/duckdb/analytics.py"
  click DV "https://github.com/minora2k4/stock-etl-pipeline/blob/main/duckdb/data_visualization.py"
  click DDD "https://github.com/minora2k4/stock-etl-pipeline/tree/main/duckdb/Dockerfile"
  click DDR "https://github.com/minora2k4/stock-etl-pipeline/blob/main/duckdb/requirements.txt"

  click ROOTP "https://github.com/minora2k4/stock-etl-pipeline/blob/main/processor.py"

  %% Hidden nodes to attach click events to specific files (without showing paths)
  DC["Docker Compose file"]:::hidden
  RD["Project README"]:::hidden
  KPP["producer.py"]:::hidden
  KPC["config.py"]:::hidden
  KPU["utils.py"]:::hidden
  KPD["Dockerfile"]:::hidden
  KPR["requirements.txt"]:::hidden
  SPD["Dockerfile"]:::hidden
  SPR["requirements.txt"]:::hidden
  DQA["analytics.py"]:::hidden
  DV["data_visualization.py"]:::hidden
  DDD["Dockerfile"]:::hidden
  DDR["requirements.txt"]:::hidden
  ROOTP["root processor.py"]:::hidden

  %% =========================
  %% Styles (color is required)
  %% =========================
  classDef external fill:#E5E7EB,stroke:#6B7280,color:#111827,stroke-width:1.5px
  classDef stream fill:#FB923C,stroke:#9A3412,color:#111827,stroke-width:1.5px
  classDef ingest fill:#FDE68A,stroke:#B45309,color:#111827,stroke-width:1.5px
  classDef compute fill:#93C5FD,stroke:#1D4ED8,color:#0B1220,stroke-width:1.5px
  classDef storage fill:#86EFAC,stroke:#166534,color:#052e16,stroke-width:1.5px
  classDef storageLite fill:#BBF7D0,stroke:#166534,color:#052e16,stroke-width:1.2px,stroke-dasharray: 4 3
  classDef bi fill:#D8B4FE,stroke:#6D28D9,color:#1F102A,stroke-width:1.5px
  classDef db fill:#C4B5FD,stroke:#4C1D95,color:#1F102A,stroke-width:1.5px
  classDef init fill:#A7F3D0,stroke:#065F46,color:#052e16,stroke-width:1.5px,stroke-dasharray: 6 3
  classDef config fill:#FBCFE8,stroke:#9D174D,color:#1F102A,stroke-width:1.5px,stroke-dasharray: 4 3
  classDef note fill:#FEF3C7,stroke:#92400E,color:#111827,stroke-width:1.2px
  classDef code fill:#F3F4F6,stroke:#9CA3AF,color:#111827,stroke-width:1px,stroke-dasharray: 3 3
  classDef hidden fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

  %% Keep hidden nodes out of layout
  DC --- RD
  KPP --- KPC
  KPU --- KPD
  KPR --- SPD
  SPR --- DQA
  DV --- DDD
  DDR --- ROOTP
```
