How annis-souames/flights-metrics works
File structure
• Top-level files:
• producer.py & consumer.py – entry points for local/testing runs of the producer/consumer
• Pipfile / requirements.txt – dependencies (e.g., kafka-python, boto3)
• config/ – JSON templates for AWS (Lambda IAM, credentials), env (geofence), Kafka (Upstash)
• api/ – modules to fetch flight data from FlightRadar and OpenSky
• stream/ – Streams-specific implementations:
– RecordsProducer.py – publishes flight records to Kafka
– S3Consumer.py – reads from Kafka, writes to S3, runs Athena “ALTER TABLE ADD PARTITION” queries
• Deployment scripts: package-app.sh & package-deps.sh for bundling code + dependencies into Lambda-compatible zip packages.
Extract architecture insights from README
• Two AWS Lambda functions:
1. FetchLambda (runs every 15 min): calls api/flightradar.py & api/opensky.py → pushes messages to Kafka.
2. ArchiveLambda (runs hourly): pulls from Kafka topic → writes JSON to S3 (partitioned by date/hour) → issues Athena partition commands directly.
• Kafka broker on Upstash (free tier).
• Data stored in S3 with a 3-day lifecycle policy.
• Athena for SQL querying S3 data.
• Metabase on Amazon Lightsail (Docker) querying Athena.
Main components & their interactions
a. Flight Fetch Lambda (AWS Lambda)
• Invokes api/.py modules → produces messages → Kafka Cluster (Upstash)
b. Kafka Cluster
• Durable message buffer / streaming layer
c. Archive Lambda (AWS Lambda)
• Consumes topic → writes records → S3 Bucket (partitioned)
• Executes Athena DDL for new partitions
d. S3 Bucket
• Data storage + lifecycle policy (3 days retention)
e. AWS Athena
• Serverless query engine on S3 data
f. Metabase (Lightsail Docker)
• BI dashboard → queries Athena → renders metrics
g. (Optional) Glue – noted in README as replaced by direct Athena DDL in S3Consumer
Architectural patterns & design principles
• Event-driven streaming via Kafka
• Serverless compute: stateless AWS Lambda for both ingestion and archiving
• Data partitioning in S3 (time-based) for query efficiency
• Cost optimization: direct Athena partitioning vs. Glue, 3-day retention policy
• Separation of concerns: ingestion, storage, query, visualization