How damklis/dataengineeringproject works
Project Type and Purpose
• This is a containerized, end-to-end data engineering pipeline with:
• Data ingestion (Airflow DAGs for scraping RSS feeds, proxy pool management)
• Streaming (Apache Kafka topics + Kafka Connect + Debezium)
• Storage and indexing (MongoDB, Elasticsearch, MinIO)
• Data access (Django Rest Framework API)
Main Components
a. Orchestration & Ingestion
• Airflow Scheduler & Worker
• Custom Python operators/modules:
• proxypool_operator
• rss_news_operator
b. Message Bus & Connectors
• Apache Kafka cluster
• Topic A (“raw_rss_news”)
• Topic B (“mongo_changes”)
• Kafka Connect workers with connectors:
• MongoDB Sink connector
• Debezium MongoDB Source connector
• Elasticsearch Sink connector
• S3/MinIO Sink connector
c. Storage
• MongoDB replica set
• Elasticsearch cluster
• MinIO object store (S3-compatible)
d. API Service
• Django + Django REST Framework
• uWSGI + Nginx proxy
• Modules for news, search, users
e. Supporting Services
• Redis (if used by Airflow or Celery)
• Zookeeper (if Kafka uses it)
Data Flow and Relationships
1) “Airflow → Kafka Topic A” (rss_news_producer publishes messages)
2) “Kafka Topic A → MongoDB” (MongoDB Sink upserts on _id)
3) “MongoDB → Kafka Topic B” (Debezium emits change events)
4) “Kafka Topic B → Elasticsearch” (Elasticsearch Sink upserts index)
5) “Kafka Topic B → MinIO” (S3 Sink persists JSON blobs)
6) “API → Elasticsearch” (search queries)
7) “API → MongoDB” (read/write news or user data if needed)
Architectural Patterns & Design Principles
• Micro-batch ingestion via Airflow (periodic)
• Event streaming and CQRS:
• Topic A for raw ingestion
• Topic B for state changes and read-model updates
• Containerization & ‑as-code: Dockerfiles in each component + docker-compose
• Idempotent upserts in MongoDB/Elastic to guarantee consistency
• Separation of concerns: ingestion, streaming, storage, serving
Technologies and Frameworks
• Python 3.8, Airflow (custom operators, DAGs)
• Apache Kafka, Kafka Connect, Debezium
• MongoDB, Elasticsearch, MinIO (S3 API)
• Django, Django REST Framework, uWSGI, Nginx
• Docker & docker-compose, shell scripts (manage.sh, run_tests.sh)
• Ingestion (e.g., green)
• Streaming (e.g., yellow)
• Storage (e.g., blue)
• API/Serving (e.g., orange)
• Persistence layers with “upsert on _id” or “change data capture”
Final Checklist
• [ ] Notes on scheduling frequency (Airflow), connector types, and upsert semantics
• [ ] Boundary outlines for Docker-compose services vs. external services (e.g., MinIO, Kafka cluster)