How lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles works
Project Type and Purpose
• This is a full-stack, containerized real-time data pipeline for monitoring autonomous vehicles.
• It simulates vehicle telemetry and sensor data, streams it through Kafka, processes with Flink, stores in TimescaleDB, and visualizes via Streamlit.
File Structure
• Top-level files:
• Dockerfile, docker-compose.yml – container and orchestration config
• requirements.txt – Python dependencies
• Service directories:
• kafka/producer.py – data generation and Kafka producer
• Flink/flink_job.py – Flink job for stream processing
• TimescaleDB/tables.sql – SQL schema for time-series tables
• Streamlit/dashboard.py – real-time dashboard code
README Insights
• Pipeline stages described and already illustrated by pipeline.png.
• Lists technologies and flow: data generation → Kafka → Flink → TimescaleDB → Streamlit.
Main Components & Interactions
a. Components
• Data Generator (kafka/producer.py) – Python scripts generating telemetry & sensor events
• Kafka Broker – two topics (vehicle_telemetry, sensor_data)
• Flink Cluster (flink_job.py) – consumes topics, applies alert logic
• TimescaleDB (PostgreSQL) – stores processed events & alerts
• Streamlit App (dashboard.py) – real-time UI
• Docker & docker-compose – containerize and network these services
b. Data Flow
1. Python producer → Kafka topics
2. Flink job subscribes to topics → processes records → writes to TimescaleDB
3. Streamlit reads from TimescaleDB to render metrics & alerts
c. Architectural Patterns
• Event-driven streaming pipeline
• Microservices (each service in its own container)
• Time-series storage optimized for high-write throughput
d. Key Technologies
• Apache Kafka for pub/sub messaging
• Apache Flink for stateful stream processing
• TimescaleDB/PostgreSQL for durable TSDB storage
• Streamlit for interactive dashboards
• Docker for service isolation and deployment
Detailed Steps for the Engineer
4. Kafka broker with two topic symbols inside (vehicle_telemetry, sensor_data).
5. Kafka to Flink (job manager + task managers), annotate “Flink job: flink_job.py”.