# kdb04/tweeeeedbt architecture

> Architecture diagram of the GitHub repository kdb04/tweeeeedbt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kdb04/tweeeeedbt
- Repository: https://github.com/kdb04/tweeeeedbt

## Overview

##### Project type and purpose
- This is a real-time data streaming pipeline (“TweeeeDBT”) built in Python that uses Apache Kafka for ingestion, Apache Spark for stream processing, and PostgreSQL as the persistent store.
- Its purpose is to simulate tweet data production, process/transform it in real time, and persist various views (geolocation, team mentions, verified-user windows, etc.) into a database.

##### File structure
- Top-level scripts:
- producer.py – generates/sends raw tweet events into Kafka
- stream_processing.py – a Spark Structured Streaming job that reads raw events from one or more Kafka topics, transforms/enriches them, and writes results back to other Kafka topics
- batch_processing.py – an offline Spark job for batch analytics (e.g., historical aggregations)
- consumer_<…>.py – separate Python consumers for each processed topic (geolocation, team mentions, user verification streams, windowed verified user counts) that read from Kafka and write into PostgreSQL
- Supporting files:
- docker-compose.yaml – spins up Zookeeper and Kafka containers
- reset-kafka.sh – resets Kafka topics between runs
- DB/ – contains schema.sql (PostgreSQL DDL) and dbcheck.py (sanity-check script)
- requirements.txt – Python dependencies
- No front-end; purely back-end streaming micro-services.

##### README for additional insights
- Shows how to launch Zookeeper/Kafka via Docker Compose
- Spark standalone set-up instructions (local Spark)
- PostgreSQL installation, user creation, schema application
- Run order:
1. reset-kafka.sh
2. spark-submit stream_processing.py
3. python3 consumer_*.py (each in its own terminal)
4. python3 producer.py

##### Main components and their relationships
a. Producer service
- publishes raw events to “tweets_raw” Kafka topic
b. Kafka cluster (Zookeeper + Kafka brokers)
- transient messaging bus between producers, stream processors, and consumers
c. Spark Streaming job
- consumes from “tweets_raw”
- applies transformations/enrichments
- writes multiple processed streams to distinct Kafka topics (e.g., “tweets_geo”, “tweets_teammentions”, “tweets_userverify”, “tweets_verified_user_windowed”)
d. Consumer services
- one micro-service per topic
- consumes enriched events and writes into respective tables in PostgreSQL
e. PostgreSQL database
- persistent store for downstream analytics or dashboards

##### Architectural patterns and design principles
- Separation of concerns: ingestion (producer), processing (Spark), storage (consumers + DB)
- Micro-service approach for each downstream view via separate consumer scripts
- Use of a message-broker to decouple producers, processors, and consumers
- Idempotent, stateless stream processing in Spark

## Diagram (Mermaid)

```mermaid
graph TD
    %% Producer
    Producer["Producer"]:::producer
    click Producer "https://github.com/kdb04/tweeeeedbt/blob/main/producer.py"

    %% Development Environment
    subgraph "Development Environment"
        direction TB
        %% Kafka Cluster
        subgraph "Kafka Cluster" 
            direction TB
            Zookeeper["Zookeeper"]:::infra
            KafkaBroker["Kafka Broker"]:::infra
        end
        click Zookeeper "https://github.com/kdb04/tweeeeedbt/blob/main/docker-compose.yaml"
        click KafkaBroker "https://github.com/kdb04/tweeeeedbt/blob/main/docker-compose.yaml"

        %% Topics
        TweetsRaw["tweets_raw"]:::topic
        TweetsGeo["tweets_geo"]:::topic
        TweetsTeam["tweets_teammentions"]:::topic
        TweetsUserVerify["tweets_userverify"]:::topic
        TweetsWindow["tweets_verified_user_windowed"]:::topic

        %% Spark Cluster
        subgraph "Spark Streaming Cluster"
            direction TB
            SparkStreaming["Spark Streaming"]:::stream
            BatchJob["Batch Analytics"]:::batch
        end
        click SparkStreaming "https://github.com/kdb04/tweeeeedbt/blob/main/stream_processing.py"
        click BatchJob "https://github.com/kdb04/tweeeeedbt/blob/main/batch_processing.py"

        %% Support Scripts
        ResetScript["reset-kafka.sh"]:::script
        click ResetScript "https://github.com/kdb04/tweeeeedbt/blob/main/reset-kafka.sh"
    end

    %% Consumers
    subgraph "Consumer Services"
        direction TB
        ConsumerGeo["ConsumerGeo"]:::consumer
        ConsumerTeam["ConsumerTeam"]:::consumer
        ConsumerVerify["ConsumerVerify"]:::consumer
        ConsumerWindow["ConsumerWindow"]:::consumer
    end
    click ConsumerGeo "https://github.com/kdb04/tweeeeedbt/blob/main/consumer_geolocation.py"
    click ConsumerTeam "https://github.com/kdb04/tweeeeedbt/blob/main/consumer_teammentions.py"
    click ConsumerVerify "https://github.com/kdb04/tweeeeedbt/blob/main/consumer_userverify.py"
    click ConsumerWindow "https://github.com/kdb04/tweeeeedbt/blob/main/consumer_verifieduserwindowed.py"

    %% PostgreSQL Database
    subgraph "PostgreSQL DB"
        direction TB
        Postgres["PostgreSQL"]:::db
        Schema["schema.sql"]:::script
        Dbcheck["dbcheck.py"]:::script
    end
    click Schema "https://github.com/kdb04/tweeeeedbt/blob/main/DB/schema.sql"
    click Dbcheck "https://github.com/kdb04/tweeeeedbt/blob/main/DB/dbcheck.py"

    %% Dependencies file
    Requirements["requirements.txt"]:::script
    click Requirements "https://github.com/kdb04/tweeeeedbt/blob/main/requirements.txt"

    %% Flow
    Producer -->|publishes to| TweetsRaw
    TweetsRaw -->|input to| SparkStreaming

    SparkStreaming -->|writes| TweetsGeo
    SparkStreaming -->|writes| TweetsTeam
    SparkStreaming -->|writes| TweetsUserVerify
    SparkStreaming -->|writes| TweetsWindow

    TweetsGeo -->|consumed by| ConsumerGeo
    TweetsTeam -->|consumed by| ConsumerTeam
    TweetsUserVerify -->|consumed by| ConsumerVerify
    TweetsWindow -->|consumed by| ConsumerWindow

    ConsumerGeo -->|writes to| Postgres
    ConsumerTeam -->|writes to| Postgres
    ConsumerVerify -->|writes to| Postgres
    ConsumerWindow -->|writes to| Postgres

    %% Infrastructure
    Zookeeper --> KafkaBroker
    ResetScript -.->|resets topics| KafkaBroker

    %% Batch job note
    BatchJob -.->|reads from| KafkaBroker
    BatchJob -.->|writes historical results to DB| Postgres

    %% Classes
    classDef producer fill:#a2f5a2,stroke:#000,stroke-width:1px;
    classDef infra fill:#f2f2f2,stroke:#666,stroke-width:1px;
    classDef topic fill:#f9d579,stroke:#d58c00,stroke-width:1px,stroke-dasharray: 5 5;
    classDef stream fill:#a0d8f1,stroke:#005f7f,stroke-width:1px;
    classDef batch fill:#c0c0c0,stroke:#333,stroke-width:1px;
    classDef consumer fill:#d1a3f1,stroke:#5e2a7e,stroke-width:1px;
    classDef db fill:#d3d3d3,stroke:#666,stroke-width:1px;
    classDef script fill:#fff3c4,stroke:#cc9c00,stroke-width:1px;
```
