# lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles architecture

> Architecture diagram of the GitHub repository lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles
- Repository: https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles

## Overview

##### Project Type and Purpose
- This is a full-stack, containerized real-time data pipeline for monitoring autonomous vehicles.
- It simulates vehicle telemetry and sensor data, streams it through Kafka, processes with Flink, stores in TimescaleDB, and visualizes via Streamlit.

##### File Structure
- Top-level files:
- Dockerfile, docker-compose.yml – container and orchestration config
- requirements.txt – Python dependencies
- Service directories:
- kafka/producer.py – data generation and Kafka producer
- Flink/flink_job.py – Flink job for stream processing
- TimescaleDB/tables.sql – SQL schema for time-series tables
- Streamlit/dashboard.py – real-time dashboard code

##### README Insights
- Pipeline stages described and already illustrated by `pipeline.png`.
- Lists technologies and flow: data generation → Kafka → Flink → TimescaleDB → Streamlit.

##### Main Components & Interactions
a. Components
- Data Generator (kafka/producer.py) – Python scripts generating telemetry & sensor events
- Kafka Broker – two topics (`vehicle_telemetry`, `sensor_data`)
- Flink Cluster (flink_job.py) – consumes topics, applies alert logic
- TimescaleDB (PostgreSQL) – stores processed events & alerts
- Streamlit App (dashboard.py) – real-time UI
- Docker & docker-compose – containerize and network these services
b. Data Flow
1. Python producer → Kafka topics
2. Flink job subscribes to topics → processes records → writes to TimescaleDB
3. Streamlit reads from TimescaleDB to render metrics & alerts
c. Architectural Patterns
- Event-driven streaming pipeline
- Microservices (each service in its own container)
- Time-series storage optimized for high-write throughput
d. Key Technologies
- Apache Kafka for pub/sub messaging
- Apache Flink for stateful stream processing
- TimescaleDB/PostgreSQL for durable TSDB storage
- Streamlit for interactive dashboards
- Docker for service isolation and deployment

##### Detailed Steps for the Engineer
4. Kafka broker with two topic symbols inside (`vehicle_telemetry`, `sensor_data`).
5. Kafka to Flink (job manager + task managers), annotate “Flink job: flink_job.py”.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Docker Compose Network"
        direction TB
        Producer["Data Generator\n(kafka/producer.py)"]:::kafka
        Kafka["Kafka Broker\nTopics: vehicle_telemetry, sensor_data\nPort: 9092"]:::kafka
        Flink["Flink Cluster\n(Job Manager & TaskManager)"]:::flink
        TimescaleDB["TimescaleDB\n(SQL Schema: tables.sql)\nPort: 5432"]:::storage
        Streamlit["Streamlit App\n(port: 8501)"]:::ui
        DockerfileNode["Dockerfile"]:::orchestration
        Requirements["requirements.txt"]:::orchestration
    end

    Browser["User Browser"]:::external

    Producer -->|"produce to vehicle_telemetry & sensor_data"| Kafka
    Kafka -->|"stream topics"| Flink
    Flink -->|"write processed events"| TimescaleDB
    TimescaleDB -->|"read metrics & alerts"| Streamlit
    Streamlit -->|"view dashboard"| Browser

    click Producer "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/kafka/producer.py"
    click Orchestration "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/docker-compose.yml"
    click Flink "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/Flink/flink_job.py"
    click TimescaleDB "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/TimescaleDB/tables.sql"
    click Streamlit "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/Streamlit/dashboard.py"
    click DockerfileNode "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/tree/main/Dockerfile"
    click Requirements "https://github.com/lokeshreddy-18/real-time-data-pipeline-for-monitoring-autonomous-vehicles/blob/main/requirements.txt"

    classDef kafka fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef flink fill:#90EE90,stroke:#000,stroke-width:1px
    classDef storage fill:#FFD580,stroke:#000,stroke-width:1px
    classDef ui fill:#D8BFD8,stroke:#000,stroke-width:1px
    classDef orchestration fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#000,stroke-width:1px
```
