How udaradev/fleet-lambda-platform works
Fleet Lambda Platform simulates ride-hailing telemetry and expenses, producing near-real-time fleet insights and authoritative daily profitability reports. Telemetry is published to Kafka and consumed independently by a speed stream for live serving and a raw stream for immutable archival. Airflow triggers batch reconciliation of archived trips with expense files; reconciliation validates and aggregates inputs, then updates daily serving data and publishes integrity-manifested reports. The API exposes live metrics, alerts, health, and reports. The README documents monitoring and data-quality workflows beyond the sampled source; those branches are included at the documented level, without assuming their internal task wiring.
Main entry points: reconcile.py Batch reconciliation · speed_job.py Speed stream · data_quality_dag.py Quality workflow · main.py FastAPI service
Components 22
Simulation and ingestion
- Telemetry producer
simulators/producer_stream.py - Expense producer
simulators/producer_batch.py - Kafka events
- Expense CSV
Lambda streaming
- Speed stream
streaming/speed_job.py - Raw stream
streaming/raw_job.py - Event validation
streaming/validation.py - Serving sink
streaming/sink.py - Staged merge
streaming/staged_sink.py - MinIO archive
Batch and quality
- Archive verification
common/archive.py - Daily workflow
airflow/dags/daily_profitability_dag.py - Batch reconciliation
batch/reconcile.py - Trip aggregation
batch/spark_reconcile.py - Business rules
common/domain.py - Quality workflow
airflow/dags/data_quality_dag.py
Serving and reporting
- PostgreSQL serving
- Report exports
- FastAPI service
api/main.py
Monitoring
- Prometheus
- Grafana
Other components
- Business user
Connections 26
- Telemetry producer → Kafka events: publishes events
- Expense producer → Expense CSV: writes expenses
- Kafka events → Speed stream: consumed by
- Kafka events → Raw stream: consumed by
- Speed stream → Event validation: validates events
- Speed stream → Serving sink: writes serving data
- Speed stream → Staged merge: merges batches
- Serving sink → PostgreSQL serving: updates live tables
- Staged merge → PostgreSQL serving: merges staged data
- Raw stream → MinIO archive: archives Parquet
- Daily workflow → Batch reconciliation: runs pipeline
- Batch reconciliation → Expense CSV: reads expenses
- Batch reconciliation → Archive verification: verifies archive
- Archive verification → MinIO archive: checks objects
- Batch reconciliation → Trip aggregation: aggregates trips
- Trip aggregation → MinIO archive: reads Parquet
- Batch reconciliation → Business rules: applies rules
- Batch reconciliation → PostgreSQL serving: stores daily results
- Batch reconciliation → Report exports: publishes exports
- Quality workflow → PostgreSQL serving: checks data quality
- Quality workflow → Archive verification: checks integrity
- Quality workflow → Report exports: checks manifests
- FastAPI service → PostgreSQL serving: queries serving data
- FastAPI service → Business user: serves metrics and reports
- Prometheus → FastAPI service: scrapes metrics
- Grafana → Prometheus: queries metrics