How astronomer/dag-factory works
Project Type and Purpose
• This is a Python library (“dag-factory”) that lets users declaratively define Apache Airflow DAGs via YAML/JSON configuration files.
• It provides a factory-style API + Airflow plugin to load, validate, and instantiate DAG objects at runtime, minimizing boilerplate Python code.
Core Architectural Components
a. Configuration Layer
• User‑provided YAML/JSON files describing DAG structure, tasks, dependencies, default_args, datasets, callbacks, etc.
b. Parsing & Validation
• dagfactory.parsers parses config files into in‑memory data structures.
• dagfactory.exceptions raises domain‑specific errors on invalid configs.
c. DAG Construction (Factory)
• dagfactory.dagbuilder and dagfactory.dagfactory apply the Factory pattern to convert parsed configs into Airflow DAG and task/operator objects.
• Leverages dagfactory.constants, dagfactory.settings (env vars & defaults), and dagfactory.utils.
d. Plugin Registration
• dagfactory.plugin contains the Airflow plugin entrypoint. On Airflow startup, it scans a configured directory of config files, invokes the Factory, and “injects” the resulting DAGs into globals().
e. Runtime Listeners & Telemetry
• dagfactory.listeners.runtime_event implements an observer: hooks into DAG/task lifecycle events.
• dagfactory.telemetry collects usage/operational data, ships to an analytics endpoint.
f. External Services & Dependencies
• Apache Airflow core (Scheduler, Webserver, Metadata DB)
• YAML/JSON parsing libs
• Optional HTTP hooks/operators (e.g., HTTP sensor tasks)
• Telemetry endpoint (external analytics service)
Component Relationships & Data Flow
1. At Airflow startup: Plugin component triggers
2. Configuration files → Parser → validated config objects
3. Validated config → DAG Builder / Factory → Python DAG + BaseOperator instances
4. DAG objects → exposed to Airflow scheduler/webserver
5. During DAG run:
• Listeners subscribe to events (on_success, on_failure) → Telemetry collects and exports metrics
6. Exceptions bubble up to fail DAG construction if config errors occur
Architectural Patterns & Design Principles
• Factory Pattern: encapsulate DAG construction logic
• Plugin Architecture: clean integration with Airflow’s plugin interface
• Observer Pattern: runtime listeners for event handling/telemetry
• Separation of Concerns: parsing, building, plugin registration, telemetry, utilities are modular