How minora2k4/stock-etl-pipeline works
Project type and purpose (from README + tree)
This is a containerized, end-to-end real-time streaming “Lakehouse” pipeline for market data (crypto + US equities):
Finnhub WebSocket → Kafka → Spark Structured Streaming → MinIO (S3) Parquet → DuckDB SQL → Streamlit dashboard
It’s not a monolith; it’s a multi-service data platform wired together via Docker Compose and streaming/storage interfaces.
Top-level signals that define architecture boundaries:
• kafka/: producer-side ingestion microservice
• producer.py = Finnhub WebSocket client + Kafka publisher
• config.py, utils.py = configuration/retry/serialization glue
• spark/: streaming ETL processor
• processor.py = Spark Structured Streaming job reading Kafka, transforming, writing Parquet to S3 (MinIO)
• duckdb/: analytics + BI application
• analytics.py = DuckDB connection + S3/MinIO configuration + SQL queries
• data_visualization.py = Streamlit UI + Plotly + auto-refresh
Also: each service has its own Dockerfile and requirements.txt → indicates clean service boundaries and independently buildable images.
Shows *runtime services*, network boundaries, and data flows.
Shows the internal modules in kafka/, spark/, and duckdb/ and how they interact.
This maps directly to the file tree and avoids ambiguity.
Finnhub → kafka-producer
• Protocol: WebSocket
• Payload: JSON trades
kafka-producer → Kafka topic (stock-ticks)
• Protocol: Kafka producer API
• Payload: serialized JSON (note if raw or wrapped)
Spark Structured Streaming (spark-job) ← Kafka
• Protocol: Kafka consumer connector
• Data: JSON events
spark-job → MinIO
• Protocol: S3A (Spark s3a://...)
• Format: Parquet
• Path: warehouse/processed_trades
• Partitioning: date, symbol
DuckDB → MinIO
• Protocol: S3 (DuckDB httpfs / S3 config)
• Reads: Parquet files
Streamlit UI → DuckDB
• Calls into analytics.py query functions
• Output: aggregated results / time series for Plotly
• UI: auto-refresh
• .env → kafka-producer (token, tickers, kafka server/topic)
• .env → spark-job (kafka bootstrap, topic, S3 endpoint/keys)
• .env → duckdb/streamlit (MinIO endpoint/keys, paths)
Architectural patterns/design principles actually used here
Call these out explicitly in annotations/legends:
• Decoupled streaming via Kafka (producer/consumer independence, buffering, replay potential)
• Lakehouse approach: raw-ish events transformed into columnar Parquet in object storage; queried directly without a separate warehouse engine
• Separation of Concerns in BI service (analytics.py vs data_visualization.py)
• Microservice-ish containers: ingestion, processing, storage, analytics, UI all independently deployable
• Near real-time ETL: Spark micro-batching (5s trigger)
“detail callouts” as side notes
• Kafka topic name: stock-ticks
• Spark trigger: 5s
• Output location: s3a://warehouse/processed_trades
• Partition keys: date, symbol
• Dashboard endpoint: localhost:8501
• MinIO console endpoint: :9001
Ports and depends_on as small annotations.
2. For each service, map it to its directory (kafka/, spark/, duckdb/) and list the entrypoint script(s).
• kafka/producer.py uses config.py + utils.py → Kafka
• spark/processor.py stages: read Kafka → parse JSON → enrich/compute → partition → write Parquet
• duckdb/analytics.py provides query API consumed by data_visualization.py