How amoako419/real-time_trip_processing works
Project type and purpose
• This is a serverless, event-driven data processing and analytics pipeline on AWS.
• Its purpose is to ingest streaming trip‐start and trip‐end events, store them in DynamoDB, match and merge start/end pairs into completed trips, and compute daily KPIs via an AWS Glue job.
Main components
a. Amazon Kinesis Data Stream (“TripEventsStream”) – entry point for trip events
b. Lambda Function 1 (scripts/lambda1.py) – reads from Kinesis, cleans/validates, writes RAW# items to DynamoDB
c. DynamoDB table (“TripEventsTable”) – single table for both RAW# and COMPLETED# items; Streams enabled
d. Lambda Function 2 (scripts/lambda2.py) – triggered by DynamoDB Stream, matches RAW# start/end, writes COMPLETED# items
e. AWS Glue job (scripts/glue_scripts.py) – periodically scans COMPLETED# items, computes KPIs, writes results to S3
f. Amazon S3 bucket – stores JSON/CSV KPI output (data/kpi_results)
g. (Optional) AWS Step Functions (scripts/step_functions.json) – orchestrates or monitors end-to-end workflow
Relationships & data flow
• Kinesis → Lambda1 → DynamoDB (RAW# items)
• DynamoDB Streams → Lambda2 → DynamoDB (COMPLETED# items)
• Scheduled/manual trigger → AWS Glue → DynamoDB scan → S3
• (Optionally) Step Functions orchestration around Lambdas and Glue
Architectural patterns & principles
• Serverless & event-driven design
• Single-table DynamoDB pattern with composite sort keys (RAW#{type}#{ts} and COMPLETED#{dropoff})
• Lambda micro-functions for discrete processing steps
• Decoupling via Streams and managed services
• Stateless compute, scalable storage, on-demand analytics
Essential details to capture
• AWS Kinesis Data Stream for ingestion
• Two discrete Lambda functions and their triggers
• DynamoDB single-table schema, Streams integration
• Glue job for KPI aggregation and S3 output
• (Optional) Step Functions workflow