# astronomer/dag-factory architecture

> Architecture diagram of the GitHub repository astronomer/dag-factory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/astronomer/dag-factory
- Repository: https://github.com/astronomer/dag-factory

## Overview

##### Project Type and Purpose
- This is a Python library (“dag-factory”) that lets users declaratively define Apache Airflow DAGs via YAML/JSON configuration files.
- It provides a factory-style API + Airflow plugin to load, validate, and instantiate DAG objects at runtime, minimizing boilerplate Python code.

##### Core Architectural Components
a. Configuration Layer
- User‑provided YAML/JSON files describing DAG structure, tasks, dependencies, default_args, datasets, callbacks, etc.
b. Parsing & Validation
- `dagfactory.parsers` parses config files into in‑memory data structures.
- `dagfactory.exceptions` raises domain‑specific errors on invalid configs.
c. DAG Construction (Factory)
- `dagfactory.dagbuilder` and `dagfactory.dagfactory` apply the Factory pattern to convert parsed configs into Airflow `DAG` and task/operator objects.
- Leverages `dagfactory.constants`, `dagfactory.settings` (env vars & defaults), and `dagfactory.utils`.
d. Plugin Registration
- `dagfactory.plugin` contains the Airflow plugin entrypoint. On Airflow startup, it scans a configured directory of config files, invokes the Factory, and “injects” the resulting DAGs into `globals()`.
e. Runtime Listeners & Telemetry
- `dagfactory.listeners.runtime_event` implements an observer: hooks into DAG/task lifecycle events.
- `dagfactory.telemetry` collects usage/operational data, ships to an analytics endpoint.
f. External Services & Dependencies
- Apache Airflow core (Scheduler, Webserver, Metadata DB)
- YAML/JSON parsing libs
- Optional HTTP hooks/operators (e.g., HTTP sensor tasks)
- Telemetry endpoint (external analytics service)

##### Component Relationships & Data Flow
1. At Airflow startup: Plugin component triggers
2. Configuration files → Parser → validated config objects
3. Validated config → DAG Builder / Factory → Python `DAG` + `BaseOperator` instances
4. DAG objects → exposed to Airflow scheduler/webserver
5. During DAG run:
- Listeners subscribe to events (on_success, on_failure) → Telemetry collects and exports metrics
6. Exceptions bubble up to fail DAG construction if config errors occur

##### Architectural Patterns & Design Principles
- Factory Pattern: encapsulate DAG construction logic
- Plugin Architecture: clean integration with Airflow’s plugin interface
- Observer Pattern: runtime listeners for event handling/telemetry
- Separation of Concerns: parsing, building, plugin registration, telemetry, utilities are modular

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Configuration Layer"
        ConfigYAML[("Config YAML/JSON")]:::config
        TestsFixtures[("tests/fixtures/")]:::config
    end

    subgraph "Parsing & Validation"
        Parser["dagfactory.parsers"]:::corelib
        Exceptions["dagfactory.exceptions"]:::corelib
    end

    subgraph "DAG Construction (Factory)"
        DAGBuilder["dagfactory.dagbuilder"]:::corelib
        DAGFactory["dagfactory.dagfactory"]:::corelib
        Constants["dagfactory.constants"]:::corelib
        Settings["dagfactory.settings"]:::corelib
        Utils["dagfactory.utils"]:::corelib
    end

    subgraph "Plugin Registration"
        Plugin["dagfactory.plugin"]:::corelib
    end

    subgraph "Runtime Listeners"
        Listeners["dagfactory.listeners.runtime_event"]:::corelib
    end

    TelemetryModule["dagfactory.telemetry"]:::corelib

    subgraph "Airflow Runtime"
        Scheduler["Scheduler"]:::airflow
        Webserver["Webserver"]:::airflow
        MetadataDB[("Metadata DB")]:::airflow
    end

    TelemetryEndpoint("External Telemetry Service"):::external

    ConfigYAML -->|reads| Parser
    TestsFixtures -->|reads| Parser
    Parser -->|validates| DAGBuilder
    Parser -.->|throws| Exceptions
    DAGBuilder -->|"build_dag()"| DAGFactory
    DAGFactory -->|uses| Constants
    DAGFactory -->|uses| Settings
    DAGFactory -->|uses| Utils
    DAGFactory -->|outputs DAGs| Plugin
    Plugin -->|injects DAGs into| Scheduler
    Plugin -->|injects DAGs into| Webserver
    Scheduler -->|events| Listeners
    Webserver -->|events| Listeners
    Listeners -->|collects| TelemetryModule
    TelemetryModule -->|sends metrics| TelemetryEndpoint

    click ConfigYAML "https://github.com/astronomer/dag-factory/tree/main/dev/dags/"
    click TestsFixtures "https://github.com/astronomer/dag-factory/tree/main/tests/fixtures/"
    click Parser "https://github.com/astronomer/dag-factory/blob/main/dagfactory/parsers.py"
    click Exceptions "https://github.com/astronomer/dag-factory/blob/main/dagfactory/exceptions.py"
    click DAGBuilder "https://github.com/astronomer/dag-factory/blob/main/dagfactory/dagbuilder.py"
    click DAGFactory "https://github.com/astronomer/dag-factory/blob/main/dagfactory/dagfactory.py"
    click Constants "https://github.com/astronomer/dag-factory/blob/main/dagfactory/constants.py"
    click Settings "https://github.com/astronomer/dag-factory/blob/main/dagfactory/settings.py"
    click Utils "https://github.com/astronomer/dag-factory/blob/main/dagfactory/utils.py"
    click Plugin "https://github.com/astronomer/dag-factory/blob/main/dagfactory/plugin/__init__.py"
    click Listeners "https://github.com/astronomer/dag-factory/blob/main/dagfactory/listeners/runtime_event.py"
    click TelemetryModule "https://github.com/astronomer/dag-factory/blob/main/dagfactory/telemetry.py"

    classDef config fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef corelib fill:#add8e6,stroke:#333,stroke-width:1px
    classDef airflow fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef external fill:#ffa500,stroke:#333,stroke-width:1px
```
