How s7yby02/kafka_to_opensearch works
Project type and purpose
• This is a lightweight, containerized real-time data pipeline. It streams Wikimedia recent‐change events into Kafka, then consumes and indexes them into OpenSearch, with visualization in OpenSearch Dashboards. It’s neither a traditional web front-end/back-end app nor a microservices suite—rather, it’s a streaming ETL pipeline.
File structure
• Top-level files:
• docker-compose.yml – orchestration of containers (Zookeeper, Kafka broker, OpenSearch, OpenSearch Dashboards)
• kafkaProducer.py – Python script that pulls from Wikimedia EventStream API and pushes to the Kafka topic
• kafkaConsumer.py – Python script that reads from the Kafka topic and indexes into OpenSearch
• requirements.txt – Python dependencies
• README.md – high-level overview and instructions
• No “src” or “services” folders, but we clearly have two pipeline components and a container orchestration file.
Extract insights from the README
• There’s an existing architecture image (./architecture.png), which likely shows the end-to-end flow.
• Key dependencies: Docker Compose for multi-container setup, Python 3.x for producer/consumer, Apache Kafka, OpenSearch, OpenSearch Dashboards.
1. Wikimedia EventStream API (external HTTP stream)
2. Docker Compose network overlay (namespace grouping all containers)
3. Zookeeper container (Kafka coordination)
4. Kafka broker container (message broker)
5. Kafka Producer (kafkaProducer.py)
6. Kafka Consumer (kafkaConsumer.py)
7. OpenSearch container (data store / index)
8. OpenSearch Dashboards container (UI for visualization)
b. Relationships & data flows:
• Wikimedia EventStream API → event stream → Producer
• Producer → publishes messages → Kafka broker (topic “wikimedia”)
• Consumer → subscribes & reads → Kafka broker
• Consumer → indexes documents → OpenSearch
• OpenSearch Dashboards → queries → OpenSearch
c. Architectural patterns & principles:
• Pub/Sub streaming pipeline
• Single-responsibility: Producer for ingestion, Consumer for indexing
• Containerization and orchestration via Docker Compose
d. Technologies & frameworks:
• Python scripts (use the requirements.txt to highlight libraries: kafka-python, requests, opensearch-py)
• Apache Kafka and Zookeeper
• OpenSearch (Elasticsearch fork) with OpenSearch Dashboards
• Docker & Docker Compose