How tigisthailay/stream-processing-pipeline- works
Project type and purpose
• This is a real-time streaming data pipeline whose goal is to ingest financial tick data from the Finnhub WebSocket API, process it with Apache Spark, and store results in Snowflake.
• Key technologies: Docker Compose, Python, Apache Kafka, Apache Spark, Snowflake.
File structure
• Top-level artifacts:
• Finnhub_Producer/ ← Python service that connects to Finnhub and produces messages to Kafka
• stream_processing/ ← Spark streaming job that reads from Kafka, transforms, writes to Snowflake
• docker-compose.yaml ← orchestrates all containers and networking
• README.md/.env ← configuration and environment variable definitions
• Inside Finnhub_Producer: Dockerfile, requirements.txt, Finnhub_Producer.py, utils.py
• Inside stream_processing: Dockerfile, requirements.txt, stream_processing.py, Spark and Snowflake connector JARs
Extract architecture insights from README
• Data flow: Finnhub WebSocket → Kafka topic(s) → Spark Structured Streaming → Snowflake
• External services: Snowflake data warehouse, Finnhub API
• Monitoring & logging: Kafka UI, Docker logs
Main components
a. Finnhub Producer Container
• Connects via WebSocket to Finnhub API
• Publishes JSON messages into Kafka “ingest” topic
• Tech: Python, websockets library, kafka-python
b. Apache Kafka Cluster (Docker service)
• Single-broker setup via Compose
• Stores raw tick data in a topic
• Exposes bootstrap server to producers and consumers
c. Kafka UI (optional monitoring service on 8083)
d. Spark Streaming Container
• Uses spark-submit to run stream_processing.py
• Reads from Kafka ingest topic, applies transformations
• Writes enriched records to Kafka sink topic or directly to Snowflake
• Tech: Spark Structured Streaming, spark-sql-kafka, spark-snowflake connectors
e. Snowflake Data Warehouse (external service)
• Receives upserts from Spark via JDBC connector
• Houses the processed financial data for analytics and BI
Relationships and data flows
• Optionally: Spark Streaming job → Kafka sink topic (for downstream consumers)
Architectural patterns and design principles
• Containerized micro-services for isolation (each major component in its own Docker container)
• Decoupling via asynchronous messaging (Kafka acts as the buffer between producer and processor)
• Single responsibility: Producer only ingests, Spark only transforms, Snowflake only stores