# s7yby02/kafka_to_opensearch architecture

> Architecture diagram of the GitHub repository s7yby02/kafka_to_opensearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/s7yby02/kafka_to_opensearch
- Repository: https://github.com/s7yby02/kafka_to_opensearch

## Overview

##### Project type and purpose
- This is a lightweight, containerized real-time data pipeline. It streams Wikimedia recent‐change events into Kafka, then consumes and indexes them into OpenSearch, with visualization in OpenSearch Dashboards. It’s neither a traditional web front-end/back-end app nor a microservices suite—rather, it’s a streaming ETL pipeline.

##### File structure
- Top-level files:
- docker-compose.yml – orchestration of containers (Zookeeper, Kafka broker, OpenSearch, OpenSearch Dashboards)
- kafkaProducer.py – Python script that pulls from Wikimedia EventStream API and pushes to the Kafka topic
- kafkaConsumer.py – Python script that reads from the Kafka topic and indexes into OpenSearch
- requirements.txt – Python dependencies
- README.md – high-level overview and instructions
- No “src” or “services” folders, but we clearly have two pipeline components and a container orchestration file.

##### Extract insights from the README
- There’s an existing architecture image (./architecture.png), which likely shows the end-to-end flow.
- Key dependencies: Docker Compose for multi-container setup, Python 3.x for producer/consumer, Apache Kafka, OpenSearch, OpenSearch Dashboards.

1. Wikimedia EventStream API (external HTTP stream)
2. Docker Compose network overlay (namespace grouping all containers)
3. Zookeeper container (Kafka coordination)
4. Kafka broker container (message broker)
5. Kafka Producer (kafkaProducer.py)
6. Kafka Consumer (kafkaConsumer.py)
7. OpenSearch container (data store / index)
8. OpenSearch Dashboards container (UI for visualization)
b. Relationships & data flows:
- Wikimedia EventStream API → event stream → Producer
- Producer → publishes messages → Kafka broker (topic “wikimedia”)
- Consumer → subscribes & reads → Kafka broker
- Consumer → indexes documents → OpenSearch
- OpenSearch Dashboards → queries → OpenSearch
c. Architectural patterns & principles:
- Pub/Sub streaming pipeline
- Single-responsibility: Producer for ingestion, Consumer for indexing
- Containerization and orchestration via Docker Compose
d. Technologies & frameworks:
- Python scripts (use the requirements.txt to highlight libraries: kafka-python, requests, opensearch-py)
- Apache Kafka and Zookeeper
- OpenSearch (Elasticsearch fork) with OpenSearch Dashboards
- Docker & Docker Compose

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Service
    Wikimedia["Wikimedia EventStream API"]:::external

    %% Compute Scripts
    Producer["Kafka Producer\n(kafkaProducer.py)"]:::compute
    Consumer["Kafka Consumer\n(kafkaConsumer.py)"]:::compute

    %% Docker Compose Network
    subgraph "Docker Compose Network"
        direction TB
        ZK["Zookeeper"]:::message
        Kafka["Kafka Broker\n(topic: wikimedia)"]:::message
        OS["OpenSearch\n(index: wikimedia)"]:::datastore
        Dash["OpenSearch Dashboards"]:::datastore
    end

    %% Data Flows
    Wikimedia -->|"event stream"| Producer
    Producer -->|"publish JSON events"| Kafka
    ZK -->|"coordination"| Kafka
    Consumer -->|"subscribe/read"| Kafka
    Consumer -->|"index documents"| OS
    Dash -->|"query REST API"| OS

    %% Click Events
    click Producer "https://github.com/s7yby02/kafka_to_opensearch/blob/main/kafkaProducer.py"
    click Consumer "https://github.com/s7yby02/kafka_to_opensearch/blob/main/kafkaConsumer.py"
    click ZK "https://github.com/s7yby02/kafka_to_opensearch/blob/main/docker-compose.yml"
    click Kafka "https://github.com/s7yby02/kafka_to_opensearch/blob/main/docker-compose.yml"
    click OS "https://github.com/s7yby02/kafka_to_opensearch/blob/main/docker-compose.yml"
    click Dash "https://github.com/s7yby02/kafka_to_opensearch/blob/main/docker-compose.yml"

    %% Styles
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef message fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef compute fill:#99ccff,stroke:#333,stroke-width:1px
    classDef datastore fill:#ccffcc,stroke:#333,stroke-width:1px

    %% Legend
    subgraph "Legend"
        direction TB
        E["External Service"]:::external
        M["Messaging Layer"]:::message
        C["Compute Scripts"]:::compute
        D["Data Store & UI"]:::datastore
    end
```
