# dhruveww/stock-market-real-time-analysis-with-kafka-and-aws architecture

> Architecture diagram of the GitHub repository dhruveww/stock-market-real-time-analysis-with-kafka-and-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws
- Repository: https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws

## Overview

This repository documents a real-time stock-data pipeline: a Python producer publishes simulated data to Kafka, a consumer writes it to S3, and Glue cataloging enables Athena queries and optional visualization. The notebooks and query file are present, but their contents were not provided, so component behavior and links below rely on README statements rather than inspected code; the CSV is documented as a simulated dataset, but its runtime use is unspecified.

Main entry points: [`kafkaConsumer.ipynb`](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaConsumer.ipynb) (Data consumer), [`kafkaProducer.ipynb`](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaProducer.ipynb) (Data producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_streaming["Streaming"]
  node_producer["Data producer"]
  node_kafka["Apache Kafka"]
  node_consumer["Data consumer"]
  node_ec2["AWS EC2"]
end

subgraph group_lake["Data Lake"]
  node_s3[("AWS S3")]
  node_crawler["Glue crawler"]
  node_catalog[("Glue catalog")]
end

subgraph group_analytics["Analytics"]
  node_athena["AWS Athena"]
  node_queries["SQL queries<br/>[kafka queries.txt]"]
  node_quicksight["QuickSight"]
end

node_dataset["Stock dataset<br/>[indexProcessed.csv]"]

node_producer -->|"publishes data"| node_kafka
node_consumer -->|"consumes data"| node_kafka
node_consumer -->|"stores CSV"| node_s3
node_crawler -->|"crawls data"| node_s3
node_crawler -->|"builds metadata"| node_catalog
node_athena -->|"queries cataloged data"| node_catalog
node_athena -->|"queries stored data"| node_s3
node_athena -.->|"enables visualization"| node_quicksight
node_ec2 -->|"hosts Kafka"| node_kafka

click node_producer "https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaProducer.ipynb"
click node_consumer "https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaConsumer.ipynb"
click node_queries "https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafka%20queries.txt"
click node_dataset "https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/indexProcessed.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka,node_consumer,node_ec2 toneBlue
class node_s3,node_crawler,node_catalog toneAmber
class node_athena,node_queries,node_quicksight toneMint
class node_dataset toneTeal
```

## Components

### Streaming

- [Data producer](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaProducer.ipynb): `kafkaProducer.ipynb`
- **Apache Kafka**
- [Data consumer](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafkaConsumer.ipynb): `kafkaConsumer.ipynb`
- **AWS EC2**

### Data Lake

- **AWS S3**
- **Glue crawler**
- **Glue catalog**

### Analytics

- **AWS Athena**
- [SQL queries](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/kafka%20queries.txt): `kafka queries.txt`
- **QuickSight**

### Other components

- [Stock dataset](https://github.com/dhruveww/stock-market-real-time-analysis-with-kafka-and-aws/blob/main/indexProcessed.csv): `indexProcessed.csv`

## Connections

- Data producer → Apache Kafka: publishes data (evidence: `README.md`)
- Data consumer → Apache Kafka: consumes data (evidence: `README.md`)
- Data consumer → AWS S3: stores CSV (evidence: `README.md`)
- Glue crawler → AWS S3: crawls data (evidence: `README.md`)
- Glue crawler → Glue catalog: builds metadata (evidence: `README.md`)
- AWS Athena → Glue catalog: queries cataloged data (evidence: `README.md`)
- AWS Athena → AWS S3: queries stored data (evidence: `README.md`)
- AWS Athena → QuickSight: enables visualization (evidence: `README.md`)
- AWS EC2 → Apache Kafka: hosts Kafka (evidence: `README.md`)
