# annis-souames/flights-metrics architecture

> Architecture diagram of the GitHub repository annis-souames/flights-metrics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/annis-souames/flights-metrics
- Repository: https://github.com/annis-souames/flights-metrics

## Overview

##### File structure
- Top-level files:
- producer.py & consumer.py – entry points for local/testing runs of the producer/consumer
- Pipfile / requirements.txt – dependencies (e.g., kafka-python, boto3)
- config/ – JSON templates for AWS (Lambda IAM, credentials), env (geofence), Kafka (Upstash)
- api/ – modules to fetch flight data from FlightRadar and OpenSky
- stream/ – Streams-specific implementations:
– RecordsProducer.py – publishes flight records to Kafka
– S3Consumer.py – reads from Kafka, writes to S3, runs Athena “ALTER TABLE ADD PARTITION” queries
- Deployment scripts: package-app.sh & package-deps.sh for bundling code + dependencies into Lambda-compatible zip packages.

##### Extract architecture insights from README
- Two AWS Lambda functions:
1. **FetchLambda** (runs every 15 min): calls api/flightradar.py & api/opensky.py → pushes messages to Kafka.
2. **ArchiveLambda** (runs hourly): pulls from Kafka topic → writes JSON to S3 (partitioned by date/hour) → issues Athena partition commands directly.
- Kafka broker on Upstash (free tier).
- Data stored in S3 with a 3-day lifecycle policy.
- Athena for SQL querying S3 data.
- Metabase on Amazon Lightsail (Docker) querying Athena.

##### Main components & their interactions
a. **Flight Fetch Lambda** (AWS Lambda)
- Invokes api/.py modules → produces messages → **Kafka Cluster** (Upstash)
b. **Kafka Cluster**
- Durable message buffer / streaming layer
c. **Archive Lambda** (AWS Lambda)
- Consumes topic → writes records → **S3 Bucket** (partitioned)
- Executes Athena DDL for new partitions
d. **S3 Bucket**
- Data storage + lifecycle policy (3 days retention)
e. **AWS Athena**
- Serverless query engine on S3 data
f. **Metabase** (Lightsail Docker)
- BI dashboard → queries Athena → renders metrics
g. **(Optional) Glue** – noted in README as replaced by direct Athena DDL in S3Consumer

##### Architectural patterns & design principles
- **Event-driven streaming** via Kafka
- **Serverless compute**: stateless AWS Lambda for both ingestion and archiving
- **Data partitioning** in S3 (time-based) for query efficiency
- **Cost optimization**: direct Athena partitioning vs. Glue, 3-day retention policy
- **Separation of concerns**: ingestion, storage, query, visualization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Services"
        Kafka["Kafka Cluster (Upstash)"]:::messaging
        Metabase["Metabase (Lightsail Docker)"]:::analytics
    end

    %% AWS Cloud
    subgraph "AWS Cloud"
        subgraph "Monitoring/Triggers"
            CW15["CloudWatch Event\n(15min)"]:::trigger
            CW60["CloudWatch Event\n(1hr)"]:::trigger
        end
        subgraph "Compute"
            FL["FetchLambda\n(AWS Lambda)"]:::compute
            AL["ArchiveLambda\n(AWS Lambda)"]:::compute
        end
        subgraph "Storage & Analytics"
            S3["S3 Bucket\n(partitioned)"]:::storage
            Athena["Athena"]:::analytics
        end
        IAM["IAM Roles & Policies"]:::meta
        Lifecycle["Lifecycle Policy\n(3-day)"]:::meta
    end

    %% Code Modules & Artifacts
    subgraph "Code & Config"
        producer["producer.py"]:::code
        flightradar["api/flightradar.py"]:::code
        opensky["api/opensky.py"]:::code
        state["api/state.py"]:::code
        recordsProducer["stream/RecordsProducer.py"]:::code
        consumercode["consumer.py"]:::code
        s3consumer["stream/S3Consumer.py"]:::code
        configKafka["config/kafka.example.json"]:::config
        configEnv["config/env.example.json"]:::config
        configAws["config/aws.example.json"]:::config
        configPy["config.py"]:::config
        packageApp["package-app.sh"]:::deploy
        packageDeps["package-deps.sh"]:::deploy
        pipfile["Pipfile / Pipfile.lock"]:::config
        reqs["requirements.txt"]:::config
        readme["readme.md"]:::docs
        docs["docs/"]:::docs
        logsMain["logs/main.log"]:::docs
        logsAll["logs/logs.log"]:::docs
    end

    %% Connections
    CW15 --> FL
    CW60 --> AL

    FL -->|"publishes"| Kafka
    AL -->|"consumes"| Kafka

    AL -->|"writes JSON"| S3
    AL -->|"DDL partitions"| Athena
    S3 -->|"data for queries"| Athena
    Athena -->|"queries data"| Metabase

    FL --> producer
    producer --> flightradar
    producer --> opensky
    producer --> state
    producer --> recordsProducer

    AL --> consumercode
    consumercode --> s3consumer

    FL --> IAM
    AL --> IAM
    S3 --> Lifecycle

    packageApp --> producer
    packageApp --> consumercode
    packageDeps --> producer
    packageDeps --> consumercode

    %% Click Events
    click producer "https://github.com/annis-souames/flights-metrics/blob/main/producer.py"
    click flightradar "https://github.com/annis-souames/flights-metrics/blob/main/api/flightradar.py"
    click opensky "https://github.com/annis-souames/flights-metrics/blob/main/api/opensky.py"
    click state "https://github.com/annis-souames/flights-metrics/blob/main/api/state.py"
    click recordsProducer "https://github.com/annis-souames/flights-metrics/blob/main/stream/RecordsProducer.py"
    click configKafka "https://github.com/annis-souames/flights-metrics/blob/main/config/kafka.example.json"
    click configEnv "https://github.com/annis-souames/flights-metrics/blob/main/config/env.example.json"
    click consumercode "https://github.com/annis-souames/flights-metrics/blob/main/consumer.py"
    click s3consumer "https://github.com/annis-souames/flights-metrics/blob/main/stream/S3Consumer.py"
    click configAws "https://github.com/annis-souames/flights-metrics/blob/main/config/aws.example.json"
    click configPy "https://github.com/annis-souames/flights-metrics/blob/main/config.py"
    click packageApp "https://github.com/annis-souames/flights-metrics/blob/main/package-app.sh"
    click packageDeps "https://github.com/annis-souames/flights-metrics/blob/main/package-deps.sh"
    click pipfile "https://github.com/annis-souames/flights-metrics/tree/main/Pipfile"
    click reqs "https://github.com/annis-souames/flights-metrics/blob/main/requirements.txt"
    click readme "https://github.com/annis-souames/flights-metrics/blob/main/readme.md"
    click docs "https://github.com/annis-souames/flights-metrics/tree/main/docs/"
    click logsMain "https://github.com/annis-souames/flights-metrics/blob/main/logs/main.log"
    click logsAll "https://github.com/annis-souames/flights-metrics/blob/main/logs/logs.log"

    %% Styles
    classDef compute fill:#D0E8FF,stroke:#0366d6,color:#000
    classDef messaging fill:#FFE5B4,stroke:#FF8C00,color:#000
    classDef storage fill:#E0FFD0,stroke:#228B22,color:#000
    classDef analytics fill:#E5D0FF,stroke:#800080,color:#000
    classDef trigger fill:#FFD7D7,stroke:#FF0000,color:#000
    classDef meta fill:#F0F0F0,stroke:#A9A9A9,color:#000
    classDef code fill:#FFF0E0,stroke:#CD853F,color:#000
    classDef config fill:#F5F5DC,stroke:#DAA520,color:#000
    classDef deploy fill:#E0FFFF,stroke:#20B2AA,color:#000
    classDef docs fill:#F0E68C,stroke:#B8860B,color:#000
```
