# darshilparmar/stock-market-kafka-data-engineering-project architecture

> Architecture diagram of the GitHub repository darshilparmar/stock-market-kafka-data-engineering-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darshilparmar/stock-market-kafka-data-engineering-project
- Repository: https://github.com/darshilparmar/stock-market-kafka-data-engineering-project

## Overview

##### Project Type & Purpose
- This project is an end-to-end data engineering pipeline focused on handling real-time stock market data. It leverages Apache Kafka for streaming, AWS services (such as S3, Athena, Glue, EC2) for storage, processing, and cataloging, and Python for implementation.
- The README clearly outlines that the project ingests stock market data with Kafka and further processes it via AWS tools.

##### File Structure Analysis
- The file tree shows two Jupyter notebooks (KafkaConsumer.ipynb and KafkaProducer.ipynb) which indicate separate roles in the pipeline: one for consuming Kafka streams and one for producing messages.
- presence of command_kafka.txt suggests a script or command line instructions for Kafka operations.
- The indexProcessed.csv file is likely the dataset used within the project.

##### Identifying Main Components
a. Data Producers:
- Kafka Producer (as implemented in KafkaProducer.ipynb)
b. Message Broker:
- Apache Kafka as the central component that handles real-time data streaming.
c. Data Consumers:
- Kafka Consumer (as implemented in KafkaConsumer.ipynb)
d. AWS Ecosystem Components:
- S3: For data storage.
- Athena: For query execution on the stored data.
- Glue Crawler & Glue Catalog: For data discovery and metadata management.
- EC2: Likely for executing parts of the processing or running additional services.
e. Data Source:
- The stock market dataset (indexProcessed.csv) that is either ingested or processed to simulate streaming data.
f. Auxiliary/Operational Elements:
- Command scripts (command_kafka.txt) that may manage or automate Kafka configurations.

##### Relationships & Interactions
- Data flows from the Data Source through the Kafka Producer into the Kafka topic.
- The Kafka Consumer then processes this streamed data.
- Processed data is then sent to AWS S3 for storage.
- AWS Glue (Crawler and Catalog) picks up the new data for cataloging.
- Athena queries operate on the cataloged data.
- EC2 instances might be hosting some of these processes or managing AWS service interactions.

##### Architectural Patterns & Technologies
- Core components like Apache Kafka and AWS components along with Python-based processing.
- Mention the use of Jupyter notebooks which serve as the development/testing interface for the Kafka components.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        DS["Data Source (indexProcessed.csv)"]:::ingestion
        Prod["Kafka Producer (Notebook)"]:::ingestion
    end
    DS -->|"streams data to"| Prod

    %% Messaging Layer
    subgraph "Messaging Layer"
        K["Apache Kafka"]:::messaging
    end
    Prod -->|"publishes to"| K

    %% Processing/Consumption Layer
    subgraph "Processing/Consumption Layer"
        Cons["Kafka Consumer (Notebook)"]:::processing
    end
    K -->|"delivers messages to"| Cons

    %% Storage & Data Management Layer
    subgraph "Storage & Data Management Layer"
        S3["AWS S3"]:::storage
        Glue["AWS Glue (Crawler & Catalog)"]:::aws
    end
    Cons -->|"stores processed data in"| S3
    S3 -->|"cataloged by"| Glue

    %% Query & Analytics Layer
    subgraph "Query & Analytics Layer"
        Athena["AWS Athena"]:::query
    end
    Glue -->|"queried by"| Athena

    %% Deployment/Orchestration Layer
    subgraph "Deployment/Orchestration Layer"
        EC2["AWS EC2"]:::orchestration
    end
    %% Indicating EC2 hosts processing components
    EC2 -->|"hosts"| Cons

    %% Operational Elements
    subgraph "Operational Elements"
        CMD["Kafka Commands (command_kafka.txt)"]:::operational
    end
    CMD -->|"configures"| K

    %% Documentation
    subgraph "Documentation"
        Arch["Architecture Overview (README.md)"]:::operational
    end
    Arch ---|"design reference"| Prod

    %% Styles
    classDef ingestion fill:#f9e79f,stroke:#000,stroke-width:2px;
    classDef messaging fill:#aed6f1,stroke:#000,stroke-width:2px;
    classDef processing fill:#a9dfbf,stroke:#000,stroke-width:2px;
    classDef storage fill:#fadbd8,stroke:#000,stroke-width:2px;
    classDef query fill:#d2b4de,stroke:#000,stroke-width:2px;
    classDef orchestration fill:#f5cba7,stroke:#000,stroke-width:2px;
    classDef operational fill:#ebecee,stroke:#000,stroke-width:2px;
    classDef aws fill:#d6dbdf,stroke:#000,stroke-width:2px;

    %% Click Events
    click Prod "https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/KafkaProducer.ipynb"
    click Cons "https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/KafkaConsumer.ipynb"
    click DS "https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/indexProcessed.csv"
    click CMD "https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/command_kafka.txt"
    click Arch "https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/README.md"
```
