How qazsedc13/airflow_kafka works
Project type and purpose
• This is an Apache Airflow–based data‐orchestration project that integrates with Apache Kafka.
• Its purpose is to define and execute directed‐acyclic‐graph (DAG) pipelines which produce, consume, and route JSON messages through Kafka topics, triggering downstream workflows.
File structure
• Top‐level directories/files:
• .astro/ (Astro CLI configuration & integrity tests)
• dags/ (all pipeline definitions in Python)
• src/ (custom Python modules, if any)
• tests/ (unit tests for DAG integrity and examples)
• Dockerfile, docker-compose.override.yml (container definitions)
• airflow_settings.yaml (Airflow config overrides)
• requirements.txt & packages.txt (Python and system deps)
• Patterns & components:
• Dev container support (.devcontainer/) for consistent local development
• Multiple example and production DAGs highlighting Kafka produce/consume tasks
• Integration tests to validate DAG integrity and message flows
Extract insights from README
• The project is run locally via the Astro CLI, which spins up Airflow components in Docker containers.
• It orchestrates a “produce_consume_treats” DAG that randomly routes messages to a “listen_for_mood” task, which may trigger “walking_your_pet” downstream.
• Kafka is the central messaging bus; Airflow tasks act as producers/consumers.
Main components and how to represent them
a. Airflow runtime (inside Docker via Astro CLI)
• Scheduler
• Webserver
• Worker(s) or CeleryExecutor (if used)
• Metadata database (e.g., Postgres)
b. Apache Kafka cluster
• Broker(s)
• Topics (e.g., “treats_topic”, “mood_topic”)
c. DAG definitions & tasks
• Producer tasks (job_kafka_produce_json.py, produce_consume_treats)
• Consumer tasks (job_kafka_read_json3/4, listen_to_the_stream)
• Conditional branching (listen_for_mood → walking_your_pet)
d. External dependencies
• Astro CLI (local dev orchestration)
• Docker Engine
• System packages (packages.txt)
Relationships & interactions
• Airflow Scheduler triggers DAGs on a cron or manual run.
• Producer tasks send JSON messages to Kafka topics over the network.
• Consumer tasks subscribe to those topics and pull messages.
• Conditional tasks inspect message payload and invoke downstream DAGs.
• Airflow’s metadata DB tracks state; Web UI displays DAG graphs.
Architectural patterns & technologies
• Event‑driven pipeline pattern using message queue (Kafka) as backbone
• Modular DAGs for separation of concerns
• Containerized Airflow deployment via Astro CLI + Docker
• Pytest‑based DAG integrity and functionality tests