# amoako419/real-time_trip_processing architecture

> Architecture diagram of the GitHub repository amoako419/real-time_trip_processing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amoako419/real-time_trip_processing
- Repository: https://github.com/amoako419/real-time_trip_processing

## Overview

##### Project type and purpose
- This is a serverless, event-driven data processing and analytics pipeline on AWS.
- Its purpose is to ingest streaming trip‐start and trip‐end events, store them in DynamoDB, match and merge start/end pairs into completed trips, and compute daily KPIs via an AWS Glue job.

##### Main components
a. Amazon Kinesis Data Stream (“TripEventsStream”) – entry point for trip events
b. Lambda Function 1 (scripts/lambda1.py) – reads from Kinesis, cleans/validates, writes RAW# items to DynamoDB
c. DynamoDB table (“TripEventsTable”) – single table for both RAW# and COMPLETED# items; Streams enabled
d. Lambda Function 2 (scripts/lambda2.py) – triggered by DynamoDB Stream, matches RAW# start/end, writes COMPLETED# items
e. AWS Glue job (scripts/glue_scripts.py) – periodically scans COMPLETED# items, computes KPIs, writes results to S3
f. Amazon S3 bucket – stores JSON/CSV KPI output (data/kpi_results)
g. (Optional) AWS Step Functions (scripts/step_functions.json) – orchestrates or monitors end-to-end workflow

##### Relationships & data flow
- Kinesis → Lambda1 → DynamoDB (RAW# items)
- DynamoDB Streams → Lambda2 → DynamoDB (COMPLETED# items)
- Scheduled/manual trigger → AWS Glue → DynamoDB scan → S3
- (Optionally) Step Functions orchestration around Lambdas and Glue

##### Architectural patterns & principles
- Serverless & event-driven design
- Single-table DynamoDB pattern with composite sort keys (`RAW#{type}#{ts}` and `COMPLETED#{dropoff}`)
- Lambda micro-functions for discrete processing steps
- Decoupling via Streams and managed services
- Stateless compute, scalable storage, on-demand analytics

##### Essential details to capture
- AWS Kinesis Data Stream for ingestion
- Two discrete Lambda functions and their triggers
- DynamoDB single-table schema, Streams integration
- Glue job for KPI aggregation and S3 output
- (Optional) Step Functions workflow

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    subgraph "Ingestion" 
        Kinesis["TripEventsStream<br/>(Kinesis Data Stream)"]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing"
        Lambda1["Lambda1<br/>(clean/validate RAW#)"]:::processing
        Lambda2["Lambda2<br/>(match RAW#/COMPLETED#)"]:::processing
        Scheduler["Scheduled Trigger"]:::processing
    end

    %% Storage Layer
    subgraph "Storage"
        DynamoDB["TripEventsTable<br/>(DynamoDB Single Table)"]:::storage
    end

    %% Analytics Layer
    subgraph "Analytics"
        Glue["Glue Job<br/>(compute daily KPIs)"]:::analytics
        S3["S3 Bucket<br/>(KPI Output)"]:::analytics
    end

    %% Optional Orchestration
    subgraph "AWS Step Functions" 
        StepFuncs["Step Functions<br/>(Workflow)"]:::processing
    end

    %% Data Flow
    Kinesis -->|"PutRecords"| Lambda1
    Lambda1 -->|"PutItem RAW#"| DynamoDB
    DynamoDB -->|"DynamoDB Stream event"| Lambda2
    Lambda2 -->|"PutItem COMPLETED#"| DynamoDB

    Scheduler -->|"Schedule trigger"| Glue
    Glue -->|"Scan & Compute"| DynamoDB
    Glue -->|"Write KPI JSON"| S3

    %% Orchestration connections
    StepFuncs --> Lambda1
    StepFuncs --> Lambda2
    StepFuncs --> Glue

    %% Click Events
    click Kinesis "https://github.com/amoako419/real-time_trip_processing/blob/main/scripts/to_kinesis.py"
    click Lambda1 "https://github.com/amoako419/real-time_trip_processing/blob/main/scripts/lambda1.py"
    click Lambda2 "https://github.com/amoako419/real-time_trip_processing/blob/main/scripts/lambda2.py"
    click Glue "https://github.com/amoako419/real-time_trip_processing/blob/main/scripts/glue_scripts.py"
    click S3 "https://github.com/amoako419/real-time_trip_processing/tree/main/data/kpi_results/"
    click StepFuncs "https://github.com/amoako419/real-time_trip_processing/blob/main/scripts/step_functions.json"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef analytics fill:#9370DB,stroke:#333,stroke-width:1px
```
