How nirodhauwanpriya/fleet-tracking-platform works
1. What this repository is
- A distributed telemetry pipeline for real-time fleet tracking.
- It is split into four runtime components: a Python simulator, a Go ingestion gateway, a Go stream processor/UI server, and a Go storage worker.
- Kafka is the central event bus; PostgreSQL is the durable sink.
2. Core data model and contract
- The system revolves around a telemetry.proto gRPC/Protobuf contract in gateway/proto.
- Generated Go and Python bindings (telemetry.pb.go, telemetry_grpc.pb.go, telemetry_pb2.py, telemetry_pb2_grpc.py) show that the message schema is shared across producer and consumer sides.
- This contract is the boundary between the simulator and the gateway.
3. Edge ingestion boundary: simulator → gateway
- simulator/sim.py acts as an IoT fleet emitter, sending vehicle telemetry over gRPC.
- gateway/main.go is the ingestion entrypoint; it accepts binary gRPC traffic and converts it into asynchronous stream events.
- The gateway is the system’s decoupling point between low-latency edge ingestion and downstream processing.
4. Event backbone: Kafka
- The README identifies Kafka topic raw_positions as the central stream.
- The topic is partitioned, and the gateway distributes messages using vehicle-based hashing so related events stay ordered by vehicle_id.
- This enables independent scaling of consumers and separates ingestion from processing/storage latency.
5. Stream processing / CEP subsystem: processor
- processor/main.go is the real-time analytics and control-plane service.
- It consumes from Kafka as its own consumer group, which means it can evolve independently from storage.
- This component is the “live” view layer: it likely maintains in-memory state for fleet tracking and computes operational insights before serving them to users.
6. User-facing visualization boundary
- The processor also serves processor/index.html, which is the browser dashboard.
- The README describes a Leaflet.js-based map and WebSocket updates, so the processor bridges stream state to the UI in near real time.
- This is the only directly user-facing runtime in the architecture.
7. Persistence subsystem: storage-worker
- storage-worker/main.go is the durable sink consumer.
- It reads the same Kafka topic via a separate consumer group, so persistence is isolated from live analytics.
- Its job is to transform stream records into PostgreSQL rows for history, audit, and reporting.
8. Durable store: PostgreSQL
- PostgreSQL is used for permanent telemetry history.
- The README indicates a fleet_telemetry database accessed with application credentials, implying the storage worker owns the write path.
- This separates operational stream traffic from queryable long-term storage.
9. Infrastructure and local orchestration
- docker-compose.yml provisions the supporting infrastructure: Kafka, PostgreSQL, and Kafdrop for Kafka inspection.
- This makes the pipeline runnable locally as a multi-service system rather than a monolith.
- The repository is therefore organized around runtime boundaries more than shared application code.
10. Build/tooling boundaries
- The Go services are independent modules (gateway, processor, storage-worker each have their own go.mod), which reinforces service-level deployment.
- gateway also includes Buf configuration (buf.yaml, buf.gen.yaml), indicating Protobuf code generation is part of the build workflow.
- The Python simulator is separate from the Go runtime, emphasizing polyglot integration through the gRPC contract.
11. Primary end-to-end data flow
- Telemetry originates in the simulator.
- It enters the gateway over gRPC, is serialized into Kafka events, then fans out to two consumers:
- processor for real-time tracking/UI,
- storage-worker for PostgreSQL persistence.
- This is the main architecture diagram axis: edge ingestion → event bus → parallel downstream consumers.
12. Architectural takeaway
- The main design pattern is event-driven fan-out with clear separation of concerns.
- Ingestion, live analytics, and persistence are intentionally decoupled so each can scale or fail independently.
- For an architecture diagram, the key boundaries are: simulator, gRPC gateway, Kafka topic/consumer groups, processor/UI, storage worker, and PostgreSQL.
Main entry points: telemetry.proto Telemetry proto · docker-compose.yml Kafka · gateway/main.go Gateway · processor/main.go Processor
Components 16
Contract
Shared telemetry schema and generated bindings
- Py bindings · Python client bindings generated from the telemetry contract
simulator/telemetry_pb2.py - Py gRPC · Python gRPC service bindings for the simulator side
simulator/telemetry_pb2_grpc.py - Telemetry proto · Shared gRPC/Protobuf schema for telemetry exchange
gateway/proto/telemetry.proto - Go bindings · Go bindings generated from the telemetry contract
gateway/telemetry.pb.go - Go gRPC · Go gRPC service bindings for the gateway
gateway/telemetry_grpc.pb.go - Buf config · Protobuf build configuration for the gateway service
gateway/buf.yaml - Buf gen · Protobuf generation settings for Go bindings
gateway/buf.gen.yaml
Runtimes
Core services that ingest, process, visualize, and persist telemetry
- Simulator · Synthesizes fleet telemetry and sends it over gRPC
simulator/sim.py - Gateway · Accepts gRPC telemetry and publishes normalized events to Kafka
gateway/main.go - Processor · Consumes telemetry for live tracking and near-real-time control logic
processor/main.go - Dashboard · Browser map and live updates served by the processor
processor/index.html - Storage worker · Consumes telemetry and writes durable records to PostgreSQL
storage-worker/main.go
Infrastructure
Supporting event bus and durable storage
- Kafka · Central topic backbone for raw telemetry fan-out
docker-compose.yml - raw_positions · Partitioned telemetry stream keyed to preserve vehicle ordering
- PostgreSQL · Persistent fleet telemetry database
docker-compose.yml - Compose stack · Brings up Kafka, PostgreSQL, and inspection tooling locally
docker-compose.yml
Connections 19
- Simulator → Py gRPC: uses · Simulator calls the generated Python gRPC client surface
- Simulator → Telemetry proto: implements · Simulator produces messages defined by the shared telemetry schema
- Telemetry proto → Go bindings: generates · Telemetry schema generates Go message bindings
- Telemetry proto → Go gRPC: generates · Telemetry schema generates Go service bindings
- Telemetry proto → Py bindings: generates · Telemetry schema generates Python message bindings
- Telemetry proto → Py gRPC: generates · Telemetry schema generates Python service bindings
- Buf config → Telemetry proto: configures · Buf configuration governs contract compilation
- Buf gen → Go bindings: generates · Buf generation produces Go protobuf artifacts
- Gateway → Go gRPC: uses · Gateway uses generated Go gRPC server bindings
- Gateway → Kafka: publishes · Gateway serializes inbound telemetry into Kafka events
- Kafka → raw_positions: hosts · Kafka carries the raw telemetry topic
- Gateway → raw_positions: writes · Gateway produces partitioned raw position events
- raw_positions → Processor: consumed by · Processor reads the topic in its own consumer group
- raw_positions → Storage worker: consumed by · Storage worker reads the topic independently for persistence
- Processor → Dashboard: serves · Processor serves the browser dashboard and live updates
- Processor → Kafka: subscribes · Processor consumes telemetry from Kafka for live tracking
- Storage worker → PostgreSQL: persists · Storage worker writes telemetry history to PostgreSQL
- Compose stack → Kafka: provisions · Compose brings up Kafka for local runs
- Compose stack → PostgreSQL: provisions · Compose brings up PostgreSQL for local runs