How ryanchao2012/airfly works
Project type and purpose
• “airfly” is an open-source Python CLI tool/library, not a full-stack app.
• It auto-generates Airflow DAG definitions (dag.py) from user-defined task classes.
File structure
• Top-level directories:
• src/airfly/cli → CLI entry and argument handling
• src/airfly/model → Task interface (AirFly class)
• src/airfly/utils, _vendor → Introspection and vendored code
• examples/ → Sample task modules demonstrating usage
• tests/ → Unit tests validating each layer
• Config & build: Makefile, pyproject.toml, GitHub workflows.
README for additional insights
• Describes how tasks declare operator type, params, and upstream/downstream.
• Shows the command-line options and example generated code.
• Highlights extensibility (duck-typing custom task classes).
Main components of the system
• CLI Layer (src/airfly/cli): parses arguments, loads modules into sys.path
• Task Discovery (model + utils): introspects Python packages, finds task classes
• Dependency Resolver (model/v1): walks upstream/downstream to build a DAG graph
• Code Generator: emits Python code with imports, DAG context, TaskGroups, operators, and >> links
• External Interfaces:
• User’s modules (via --modname, --path)
• Airflow operator libraries (built-in and custom)
• Output Sink: the generated dag.py file (stdout or redirected output)
Relationships and interactions
• User invokes airfly CLI → CLI Layer
• CLI loads target module(s) → Discovery
• Discovery yields task objects → Resolver builds graph
• Resolver hands graph → Code Generator
• Code Generator references Airflow classes → emits code → Output
Architectural patterns and principles
• Layered modular design for clear separation of concerns
• Duck typing for flexible task class support
• Graph traversal (DFS/BFS) to resolve dependencies
• Code templating/generation rather than runtime execution