How phnguynmkhoi/streaming_data_pipeline works
Main Components
• Data Source: At the base is a Postgres database with change data capture enabled via a Debezium connector (referenced in the file “debezium-connector-postgres.json”).
• Ingestion Layer: Kafka serves as the backbone for data ingestion – it captures the events produced by Debezium.
• Processing Layer: Apache Spark is used to process and transform the streaming data from Kafka. This layer is triggered by scripts such as “spark_extract_data_script.sh” and uses additional Python scripts found in the “src/” directory (e.g., “main.py” for simulating transactions).
• Storage Layers:
• MinIO for archiving processed data (as described in the README and referenced in “apps/extract_data.py”).
• Apache Pinot for real-time analytics – with several Pinot configuration files (schemas and table definitions) under the “pinot” directory.
• Supporting/Orchestration Components: Additional scripts (within “script/”), Docker Compose configuration (“docker-compose.yml”), and container setups (e.g., “Dockerfile.spark”) orchestrate deployment and execution.
Understand Relationships and Data Flow
• Data flows into Kafka, serving as an intermediary message broker.
• Apache Spark reads from Kafka, performs necessary transformations, and may push transformed data back into Kafka for further processing or onward to storage.
• The processed data then:
• Is stored in MinIO for archival purposes.
• Is ingested by Apache Pinot to be made accessible for real-time query and analysis.
• Additional supporting scripts manage schema and table creation in Pinot (“create_all_schema.sh”, “create_all_table.sh”).
Architectural Patterns and Design Principles
• The architecture follows a streaming pipeline approach with decoupled components.
• There is a separation of concerns between the ingestion, processing, and storage layers, which is ideal for scalability and fault tolerance.
• The project uses containerization (Docker/Docker Compose) to manage dependencies and deploy the pipeline components.
• Scripts and configuration files split responsibilities into discrete tasks (data generation, schema/table management, data transformation, etc.).
Relevant Technologies
• Debezium for CDC (change data capture) from Postgres.
• Kafka for real-time messaging and data ingestion.
• Apache Spark as the distributed processing engine.
• MinIO as the object storage solution for archival.
• Apache Pinot as the realtime analytical data warehouse.
• Docker & Docker Compose for deployment orchestration.
• Python scripts (in “src” and “apps”) performing simulation and extraction tasks.
Elements
• “Postgres Database with Debezium” (source).
• “Kafka Broker” (ingestion).
• “MinIO Data Lake” (archival storage).
• “Apache Pinot” (realtime analytics and data modeling) – show separate schema/table definitions.
• From Postgres → Debezium → Kafka.
• From Kafka → Spark.
• From Spark → [MinIO and/or Kafka] → Pinot.
• Labels for containerization layers (Docker Compose) if relevant, to stress that this is a containerized environment.
• You might also note external dependencies like “aws-java-sdk” and “hadoop-aws” mentioned in the README, which are needed for MinIO integration.