# qazsedc13/airflow_kafka architecture

> Architecture diagram of the GitHub repository qazsedc13/airflow_kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qazsedc13/airflow_kafka
- Repository: https://github.com/qazsedc13/airflow_kafka

## Overview

##### Project type and purpose
- This is an Apache Airflow–based data‐orchestration project that integrates with Apache Kafka.
- Its purpose is to define and execute directed‐acyclic‐graph (DAG) pipelines which produce, consume, and route JSON messages through Kafka topics, triggering downstream workflows.

##### File structure
- Top‐level directories/files:
- .astro/ (Astro CLI configuration & integrity tests)
- dags/ (all pipeline definitions in Python)
- src/ (custom Python modules, if any)
- tests/ (unit tests for DAG integrity and examples)
- Dockerfile, docker-compose.override.yml (container definitions)
- airflow_settings.yaml (Airflow config overrides)
- requirements.txt & packages.txt (Python and system deps)
- Patterns & components:
- Dev container support (.devcontainer/) for consistent local development
- Multiple example and production DAGs highlighting Kafka produce/consume tasks
- Integration tests to validate DAG integrity and message flows

##### Extract insights from README
- The project is run locally via the Astro CLI, which spins up Airflow components in Docker containers.
- It orchestrates a “produce_consume_treats” DAG that randomly routes messages to a “listen_for_mood” task, which may trigger “walking_your_pet” downstream.
- Kafka is the central messaging bus; Airflow tasks act as producers/consumers.

##### Main components and how to represent them
a. Airflow runtime (inside Docker via Astro CLI)
- Scheduler
- Webserver
- Worker(s) or CeleryExecutor (if used)
- Metadata database (e.g., Postgres)
b. Apache Kafka cluster
- Broker(s)
- Topics (e.g., “treats_topic”, “mood_topic”)
c. DAG definitions & tasks
- Producer tasks (job_kafka_produce_json.py, produce_consume_treats)
- Consumer tasks (job_kafka_read_json3/4, listen_to_the_stream)
- Conditional branching (listen_for_mood → walking_your_pet)
d. External dependencies
- Astro CLI (local dev orchestration)
- Docker Engine
- System packages (packages.txt)

##### Relationships & interactions
- Airflow Scheduler triggers DAGs on a cron or manual run.
- Producer tasks send JSON messages to Kafka topics over the network.
- Consumer tasks subscribe to those topics and pull messages.
- Conditional tasks inspect message payload and invoke downstream DAGs.
- Airflow’s metadata DB tracks state; Web UI displays DAG graphs.

##### Architectural patterns & technologies
- Event‑driven pipeline pattern using message queue (Kafka) as backbone
- Modular DAGs for separation of concerns
- Containerized Airflow deployment via Astro CLI + Docker
- Pytest‑based DAG integrity and functionality tests

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Deployment Environment"
        CLI["Astro CLI"]:::external
        Docker["Docker Engine"]:::external
        CLI --> Docker
    end

    subgraph "Airflow Runtime"
        Scheduler["Airflow Scheduler (Python)"]:::airflow
        Webserver["Airflow Webserver (Python)"]:::airflow
        Worker["Airflow Worker (Python)"]:::airflow
        DB["Metadata DB (Postgres)"]:::db
        Scheduler --> Worker
        Scheduler --> Webserver
        Scheduler --> DB
    end

    subgraph "Kafka Cluster"
        Broker["Kafka Broker"]:::kafka
        subgraph "Topics"
            Treats["treats_topic"]:::kafka
            Mood["mood_topic"]:::kafka
        end
        Broker --> Treats
        Broker --> Mood
    end

    Scheduler -->|"Triggers DAG Runs"| DAG1

    subgraph "produce_consume_treats DAG"
        DAG1["produce_consume_treats"]:::dag
        Prod["job_kafka_produce_json"]:::task
        Cons3["job_kafka_read_json3"]:::task
        Cons4["job_kafka_read_json4"]:::task
        Listen["listen_to_the_stream"]:::task
        Walk["walking_your_pet"]:::task
        DAG1 --> Prod
        DAG1 --> Cons3
        DAG1 --> Cons4
        DAG1 --> Listen
        Prod -->|"Produce JSON"| Treats
        Treats -->|"Consume JSON"| Cons3
        Treats -->|"Consume JSON"| Cons4
        Listen -->|"Branch on mood"| Walk
        Mood -->|"Trigger Listen"| Listen
    end

    subgraph "Other DAGs"
        DAG2["produce_consume1"]:::dag
        DAG3["produce_consume2"]:::dag
        Hello["example_dag_hello_kafka"]:::dag
        Example["exampledag"]:::dag
        SysCheck["system_packages_check"]:::dag
    end

    %% Click Events
    click Dockerfile "https://github.com/qazsedc13/airflow_kafka/tree/main/Dockerfile"
    click Docker "https://github.com/qazsedc13/airflow_kafka/blob/main/docker-compose.override.yml"
    click airflow_settings.yaml "https://github.com/qazsedc13/airflow_kafka/blob/main/airflow_settings.yaml"
    click .astro/ "https://github.com/qazsedc13/airflow_kafka/tree/main/.astro/"
    click .astro/config.yaml "https://github.com/qazsedc13/airflow_kafka/blob/main/.astro/config.yaml"
    click .devcontainer/devcontainer.json "https://github.com/qazsedc13/airflow_kafka/blob/main/.devcontainer/devcontainer.json"
    click Prod "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/job_kafka_produce_json.py"
    click Cons3 "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/job_kafka_read_json3.py"
    click Cons4 "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/job_kafka_read_json4.py"
    click Listen "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/listen_to_the_stream.py"
    click DAG2 "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/produce_consume1.py"
    click DAG3 "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/produce_consume2.py"
    click DAG1 "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/produce_consume_treats.py"
    click Walk "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/walking_your_pet.py"
    click Hello "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/example_dag_hello_kafka.py"
    click Example "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/exampledag.py"
    click SysCheck "https://github.com/qazsedc13/airflow_kafka/blob/main/dags/system_packages_check.py"
    click requirements.txt "https://github.com/qazsedc13/airflow_kafka/blob/main/requirements.txt"
    click packages.txt "https://github.com/qazsedc13/airflow_kafka/blob/main/packages.txt"
    click test1 "https://github.com/qazsedc13/airflow_kafka/blob/main/.astro/test_dag_integrity_default.py"
    click test2 "https://github.com/qazsedc13/airflow_kafka/blob/main/tests/dags/test_dag_integrity.py"
    click test3 "https://github.com/qazsedc13/airflow_kafka/blob/main/tests/dags/test_dag_example.py"

    %% Styles
    classDef airflow fill:#AED8E6,stroke:#1E90FF,stroke-width:2px
    classDef kafka fill:#FFD8A8,stroke:#FF8C00,stroke-width:2px
    classDef db fill:#C8E6C9,stroke:#388E3C,stroke-width:2px
    classDef dag fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px
    classDef task fill:#BBDEFB,stroke:#1976D2,stroke-width:2px
    classDef external fill:#F5F5F5,stroke:#9E9E9E,stroke-width:2px
```
