# agerejoseph1/real-time-stock-analytics-pipeline architecture

> Architecture diagram of the GitHub repository agerejoseph1/real-time-stock-analytics-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/agerejoseph1/real-time-stock-analytics-pipeline
- Repository: https://github.com/agerejoseph1/real-time-stock-analytics-pipeline

## Overview

##### Main Components
- Data Producer: Represented by the “data-producer” directory, this service fetches stock data from an external API. It is containerized (has its own Dockerfile) and has a dedicated requirements file.
- Data Consumer: Represented by the “data-consumer” directory, this component receives data from Kafka, processes it, and writes to a PostgreSQL database. It too is containerized.
- Message Broker (Kafka): As noted in the README, Kafka is the middleware used to stream the data between the producer and consumer. Don’t forget to include Zookeeper which is used for service coordination.
- PostgreSQL Database: The final storage for the processed data, to which the data consumer writes.

##### Relationships and Interactions
- The Data Producer fetches data from an external API and sends messages into Kafka.
- Kafka (guided by Zookeeper) queues and streams these messages.
- The Data Consumer listens for these messages from Kafka and, upon receiving them, processes and stores them in a PostgreSQL database.
- Docker Compose (via the “docker-compose.yml” file) orchestrates all of these services, ensuring that each component runs as its own container and communicates over a defined network.

##### Architectural Patterns and Design Principles
- Containerization: All major components are containerized using Docker and managed with Docker Compose.
- Data Streaming Pipeline: The producer-consumer model with Kafka as a mediator is key, along with external service integration (external API and PostgreSQL).
- Separation of Concerns: Each service (data producer, Kafka, data consumer, and database) handles a distinct task within the pipeline.

##### Technologies, Frameworks, and Libraries
- Docker and Docker Compose: For container orchestration and service management.
- Apache Kafka (and associated Zookeeper): For real-time data streaming.
- PostgreSQL: For data persistence.
- Python: Likely used for implementing data fetching and processing logic (as seen from the .py files).

- At the top or on one side, show the external API from which the Data Producer fetches data.
- In a central cluster, include Kafka and Zookeeper together, treating them as a core message broker component.
- On the consumer side, show the Data Consumer’s interaction with the PostgreSQL database.
- Optionally, include an “infrastructure” layer showing that Docker Compose spans all these containers.

## Diagram (Mermaid)

```mermaid
flowchart TD
    ExternalAPI["External API"]:::external
    DataProducer["Data Producer"]:::service
    KafkaZookeeper["Kafka & Zookeeper"]:::service
    DataConsumer["Data Consumer"]:::service
    PostgreSQL["PostgreSQL Database"]:::database
    Orchestration["Docker Compose"]:::orchestration

    ExternalAPI -->|"fetches"| DataProducer
    DataProducer -->|"sends"| KafkaZookeeper
    KafkaZookeeper -->|"streams"| DataConsumer
    DataConsumer -->|"stores"| PostgreSQL

    Orchestration ---|"orchestrates"| DataProducer
    Orchestration ---|"orchestrates"| KafkaZookeeper
    Orchestration ---|"orchestrates"| DataConsumer
    Orchestration ---|"orchestrates"| PostgreSQL

    click DataProducer "https://github.com/agerejoseph1/real-time-stock-analytics-pipeline/tree/main/data-producer/"
    click DataConsumer "https://github.com/agerejoseph1/real-time-stock-analytics-pipeline/tree/main/data-consumer/"
    click KafkaZookeeper "https://github.com/agerejoseph1/real-time-stock-analytics-pipeline/blob/main/docker-compose.yml"
    click PostgreSQL "https://github.com/agerejoseph1/real-time-stock-analytics-pipeline/blob/main/docker-compose.yml"
    click Orchestration "https://github.com/agerejoseph1/real-time-stock-analytics-pipeline/blob/main/docker-compose.yml"

    classDef external fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef service fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef database fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#FFB6C1,stroke:#333,stroke-width:2px;
```
