# damklis/dataengineeringproject architecture

> Architecture diagram of the GitHub repository damklis/dataengineeringproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damklis/dataengineeringproject
- Repository: https://github.com/damklis/dataengineeringproject

## Overview

##### Project Type and Purpose
- This is a containerized, end-to-end data engineering pipeline with:
- Data ingestion (Airflow DAGs for scraping RSS feeds, proxy pool management)
- Streaming (Apache Kafka topics + Kafka Connect + Debezium)
- Storage and indexing (MongoDB, Elasticsearch, MinIO)
- Data access (Django Rest Framework API)

##### Main Components
a. Orchestration & Ingestion
- Airflow Scheduler & Worker
- Custom Python operators/modules:
- proxypool_operator
- rss_news_operator
b. Message Bus & Connectors
- Apache Kafka cluster
- Topic A (“raw_rss_news”)
- Topic B (“mongo_changes”)
- Kafka Connect workers with connectors:
- MongoDB Sink connector
- Debezium MongoDB Source connector
- Elasticsearch Sink connector
- S3/MinIO Sink connector
c. Storage
- MongoDB replica set
- Elasticsearch cluster
- MinIO object store (S3-compatible)
d. API Service
- Django + Django REST Framework
- uWSGI + Nginx proxy
- Modules for news, search, users
e. Supporting Services
- Redis (if used by Airflow or Celery)
- Zookeeper (if Kafka uses it)

##### Data Flow and Relationships
1) “Airflow → Kafka Topic A” (rss_news_producer publishes messages)
2) “Kafka Topic A → MongoDB” (MongoDB Sink upserts on _id)
3) “MongoDB → Kafka Topic B” (Debezium emits change events)
4) “Kafka Topic B → Elasticsearch” (Elasticsearch Sink upserts index)
5) “Kafka Topic B → MinIO” (S3 Sink persists JSON blobs)
6) “API → Elasticsearch” (search queries)
7) “API → MongoDB” (read/write news or user data if needed)

##### Architectural Patterns & Design Principles
- Micro-batch ingestion via Airflow (periodic)
- Event streaming and CQRS:
- Topic A for raw ingestion
- Topic B for state changes and read-model updates
- Containerization & ‑as-code: Dockerfiles in each component + docker-compose
- Idempotent upserts in MongoDB/Elastic to guarantee consistency
- Separation of concerns: ingestion, streaming, storage, serving

##### Technologies and Frameworks
- Python 3.8, Airflow (custom operators, DAGs)
- Apache Kafka, Kafka Connect, Debezium
- MongoDB, Elasticsearch, MinIO (S3 API)
- Django, Django REST Framework, uWSGI, Nginx
- Docker & docker-compose, shell scripts (manage.sh, run_tests.sh)

- Ingestion (e.g., green)
- Streaming (e.g., yellow)
- Storage (e.g., blue)
- API/Serving (e.g., orange)
- Persistence layers with “upsert on _id” or “change data capture”

##### Final Checklist
- [ ] Notes on scheduling frequency (Airflow), connector types, and upsert semantics
- [ ] Boundary outlines for Docker-compose services vs. external services (e.g., MinIO, Kafka cluster)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration & Ingestion Layer
    subgraph "Orchestration & Ingestion"
        direction TB
        Airflow["Airflow Scheduler & Worker"]:::ingestion
        ProxyPoolOp["proxypool_operator"]:::ingestion
        RSSNewsOp["rss_news_operator"]:::ingestion
    end

    %% Message Bus & Connectors Layer
    subgraph "Message Bus & Connectors"
        direction TB
        KafkaCluster["Apache Kafka Cluster"]:::streaming
        TopicA["Topic A\n(raw_rss_news)"]:::streaming
        TopicB["Topic B\n(mongo_changes)"]:::streaming
        subgraph "Kafka Connect Workers"
            direction TB
            MongoSink["MongoDB Sink Connector"]:::streaming
            DebeziumSource["Debezium MongoDB Source"]:::streaming
            ESSink["Elasticsearch Sink Connector"]:::streaming
            S3Sink["S3/MinIO Sink Connector"]:::streaming
        end
    end

    %% Storage Layer
    subgraph "Storage"
        direction TB
        MongoDB["MongoDB Replica Set\n(upsert on _id)"]:::storage
        Elasticsearch["Elasticsearch Cluster\n(upsert on _id)"]:::storage
        MinIO["MinIO Object Store\n(persist JSON blobs)"]:::storage
    end

    %% API Service Layer
    subgraph "API Service"
        direction TB
        DjangoAPI["Django REST Framework API"]:::serving
        subgraph "API Modules"
            direction TB
            NewsModule["news module"]:::serving
            SearchModule["search module"]:::serving
            UsersModule["users module"]:::serving
        end
        Proxy["uWSGI + Nginx Proxy"]:::serving
    end

    %% Supporting Services
    subgraph "Supporting Services"
        direction TB
        Redis["Redis"]:::support
    end

    %% Data Flows
    Airflow -->|"publishes RSS"| TopicA
    TopicA -->|"upsert on _id"| MongoSink
    MongoSink --> MongoDB
    MongoDB -->|"CDC events"| DebeziumSource
    DebeziumSource --> TopicB
    TopicB -->|"upsert on _id"| ESSink
    ESSink --> Elasticsearch
    TopicB -->|"persist JSON"| S3Sink
    S3Sink --> MinIO
    DjangoAPI -->|"search queries"| Elasticsearch
    DjangoAPI -->|"read/write"| MongoDB

    %% Click Events for Mapping
    click Airflow "https://github.com/damklis/dataengineeringproject/tree/master/airflow/Dockerfile"
    click ProxyPoolOp "https://github.com/damklis/dataengineeringproject/blob/master/airflow/dags/custom_operators/proxypool_operator.py"
    click RSSNewsOp "https://github.com/damklis/dataengineeringproject/blob/master/airflow/dags/custom_operators/rss_news_operator.py"
    click KafkaCluster "https://github.com/damklis/dataengineeringproject/tree/master/kafka/Dockerfile"
    click TopicA "https://github.com/damklis/dataengineeringproject/blob/master/kafka/create_default_topic.sh"
    click MongoSink "https://github.com/damklis/dataengineeringproject/blob/master/connect/connectors/mongo-sink.json"
    click DebeziumSource "https://github.com/damklis/dataengineeringproject/blob/master/connect/connectors/mongo-dbz-source.json"
    click ESSink "https://github.com/damklis/dataengineeringproject/blob/master/connect/connectors/elasticsearch-sink.json"
    click S3Sink "https://github.com/damklis/dataengineeringproject/blob/master/connect/connectors/s3-minio-sink.json"
    click MongoDB "https://github.com/damklis/dataengineeringproject/blob/master/mongo/init.sh"
    click Elasticsearch "https://github.com/damklis/dataengineeringproject/tree/master/elastic/Dockerfile"
    click MinIO "https://github.com/damklis/dataengineeringproject/tree/master/minio/Dockerfile"
    click DjangoAPI "https://github.com/damklis/dataengineeringproject/tree/master/api/Dockerfile"
    click NewsModule "https://github.com/damklis/dataengineeringproject/blob/master/api/app/news/models.py"
    click SearchModule "https://github.com/damklis/dataengineeringproject/blob/master/api/app/search/documents.py"
    click UsersModule "https://github.com/damklis/dataengineeringproject/blob/master/api/app/users/models.py"
    click Proxy "https://github.com/damklis/dataengineeringproject/tree/master/api/proxy/Dockerfile"
    click Redis "https://github.com/damklis/dataengineeringproject/blob/master/redis/redis.conf"
    click Airflow "https://github.com/damklis/dataengineeringproject/blob/master/airflow/dags/rss_news_dag.py"

    %% Styles
    classDef ingestion fill:#a2d5c6,stroke:#077b8a,color:#000
    classDef streaming fill:#ffe066,stroke:#ff9f1c,color:#000
    classDef storage fill:#89c2d9,stroke:#1e6091,color:#000
    classDef serving fill:#ffadad,stroke:#d7263d,color:#000
    classDef support fill:#e0e0e0,stroke:#6e6e6e,color:#000
```
