How pritomdas/real-time-streaming-data-pipeline-and-dashboard works
Project Type & Purpose
• This is a full-stack real-time data pipeline project that integrates a simulation of server status data, multiple streaming data pipelines (one implemented in Scala and another in PySpark), and a dashboard built with Django/Flexmonster for visualization.
• The system utilizes technologies like Apache Kafka, Apache Spark (both Scala and PySpark based), Hadoop, ZooKeeper, PostgreSQL, and Docker for containerizing and running clusters.
File Structure
• There are three main parts:
• “data_center_server_live_status”: Contains Python scripts to simulate live server status data (and a Kafka consumer demo).
• “datamaking_real_time_data_pipeline (PySpark)” and “datamaking_streaming_data_pipeline (Scala)”: They include streaming pipeline implementations. The Scala project has the typical Scala/Java project directories (src, project, libraries, target) and uses build.sbt.
• “server_status_monitoring”: A Django project (with dashboard, models, views, templates, manage.py) that consumes data and shows a live dashboard.
• The presence of Docker setup scripts (mentioned in README) and dependencies in the Scala project (via external libraries) indicate that the system is deployed on a Docker environment incorporating multiple clusters.
Important Components to Highlight
a. Data Sources & Ingestion:
• Data Simulators: The Python simulator in “data_center_server_live_status” generates live server status events.
• Kafka Producer: The simulation data likely gets pushed into Kafka topics (as indicated by the Kafka consumer and Docker setup steps).
b. Messaging & Streaming:
• Kafka Broker: Use one or more Docker containers acting as the Kafka broker with its associated ZooKeeper container.
• Streaming Pipelines: Two distinct streaming modules are provided:
• Scala-based Spark Structured Streaming pipeline from “datamaking_streaming_data_pipeline (Scala)”
• PySpark-based streaming pipeline in “datamaking_real_time_data_pipeline (PySpark)”
These pipelines subscribe to Kafka topics, process the streaming data via Spark (possibly on an Apache Hadoop cluster), and then persist results.
c. Storage and Processing:
• Hadoop Cluster: Acts as scalable storage and processing framework; clusters are built via Docker.
• Apache Spark: The streaming computations (both Scala and PySpark) are executed here.
• PostgreSQL Database: The processed results (server events) are stored in a PostgreSQL database for use by the dashboard.
d. User Interface & Visualization:
• Django Web Application: Located in the “server_status_monitoring” folder, it handles dashboard routing, data fetch, and displays real-time visualizations using Flexmonster.
e. Docker & Cluster Management:
• Docker Containers: Overall, the infrastructure (Kafka, ZooKeeper, Hadoop, Spark, PostgreSQL) is managed and deployed within Docker.
Relationships and Data Flow
• Data Flow starts with the Python-based event simulator that sends server status events into Kafka.
• Kafka, managed in a Docker-based cluster along with ZooKeeper, acts as a distributed message queue.
• The streaming pipelines (Scala and PySpark) consume this data from Kafka. They process data in real-time using Spark and (optionally) store intermediate results into the Hadoop environment when needed.
• Post-processed data is then inserted into a PostgreSQL database.
• The Django dashboard application queries PostgreSQL and uses Flexmonster for data visualization to show a live dashboard.
Architectural Patterns & Design Principles
• Microservices/Containerization: Each major component (data simulator, Kafka, Spark Streaming, PostgreSQL, Django dashboard) is decoupled and possibly run as separate Docker containers.
• Real-Time Streaming: Use of Kafka and Spark Streaming indicates an event-driven, real-time processing architecture.
• Separation of Concerns: The data ingestion, processing, storage, and visualization are clearly separated.