# ankitdand/youtube-etl-pipeline-using-airflow architecture

> Architecture diagram of the GitHub repository ankitdand/youtube-etl-pipeline-using-airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ankitdand/youtube-etl-pipeline-using-airflow
- Repository: https://github.com/ankitdand/youtube-etl-pipeline-using-airflow

## Overview

##### Main Components
- Airflow Containers: Note that when you run ‘astro dev start’, four key Docker containers are spun up, which comprise:
- Postgres (the Metadata Database)
- Webserver (provides the UI and user interactions)
- Scheduler (monitors and triggers task execution based on DAG definitions)
- Triggerer (handles deferred tasks and dynamic triggers)
- Project Source Code: The 'dags' folder contains your Airflow DAG Python files (e.g., etl_youtube.py) that define workflows. Also take note of the tests directory for validation, and configuration files (like Dockerfile, .astro/config.yaml, and requirements.txt / packages.txt).

##### Out Component Relationships
- The Webserver communicates with the Postgres database for storing and retrieving Airflow metadata.
- The Scheduler orchestrates the execution of DAGs by interacting with the Webserver and the Triggerer.
- The Triggerer is used to offload deferred/dynamic tasks and works closely with the Scheduler.
- From the project structure, detail where the DAGs reside and how they feed into the Airflow system components.
- Role of configuration files (Dockerfile, docker-compose.yml) that tie these containers together.

##### Architectural Patterns
- Containerized architecture using Docker, which encapsulates services in isolated containers.
- Separation of concerns between the Airflow components and the core business logic defined in the DAGs.
- How the use of configuration files (e.g., .astro/config.yaml) suggests a modular and configuration-driven approach, as well as the integration testing strategy visible through the tests directory.

##### Relevant Technologies
- Docker & docker-compose: For container orchestration.
- Python & Airflow DAGs (using TaskFlow API): For pipeline definitions.
- Postgres: As the metadata database.
- Astronomer CLI: For managing local development and deployments.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Container Orchestration
    subgraph "Container Orchestration"
        DC["Docker Compose"]:::container
        WS["Airflow Webserver"]:::container
        SC["Airflow Scheduler"]:::container
        TG["Airflow Triggerer"]:::container
        PG["Postgres Metadata DB"]:::container
    end

    %% Project Source Code
    subgraph "Project Source Code"
        DAG["Airflow DAGs (ETL Scripts)"]:::project
        TEST["Tests (DAG Validation)"]:::project
        subgraph "Configuration Files"
            DF["Dockerfile"]:::config
            ACFG[".astro/config.yaml"]:::config
            PKG["packages.txt"]:::config
            REQ["requirements.txt"]:::config
        end
    end

    %% Container Interactions
    WS -->|"metadata_RW"| PG
    PG -->|"metadata_RW"| WS
    SC -->|"trigger_tasks"| TG
    SC -->|"compiles_DAGs"| DAG
    TEST -->|"validates"| DAG

    %% Configuration associations
    DC -->|"configures"| DF
    DC -->|"configures"| ACFG
    DC -->|"configures"| PKG
    DC -->|"configures"| REQ

    %% Click Events
    click DC "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/docker-compose.yml"
    click WS "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/docker-compose.yml"
    click SC "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/docker-compose.yml"
    click TG "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/docker-compose.yml"
    click PG "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/docker-compose.yml"
    click DAG "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/dags/etl_youtube.py"
    click TEST "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/tests/dags/test_dag_example.py"
    click DF "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/tree/main/Dockerfile"
    click ACFG "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/.astro/config.yaml"
    click PKG "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/packages.txt"
    click REQ "https://github.com/ankitdand/youtube-etl-pipeline-using-airflow/blob/main/requirements.txt"

    %% Styles
    classDef container fill:#FFE6AA,stroke:#FF9900,stroke-width:2px;
    classDef project fill:#D0E8F2,stroke:#006699,stroke-width:2px;
    classDef config fill:#CDEADF,stroke:#009966,stroke-width:2px;
```
