How shunichironomura/pdag works
Main Components
• Core Domain Model (pdag/_core): This is the heart of the system. It contains modules like collection.py, model.py, parameter.py, reference.py, and relationship.py. These files model the nodes, parameters, and their interconnections within the DAG.
• Execution Engine (pdag/_exec): This layer transforms the core model into an executable form. Components here (core.py, model.py, ref_resolver.py, to_exec_model.py, utils.py) manage the evaluation and execution of the DAG.
• Experiment Runner (pdag/_experiment): Comprising modules such as cases.py, results.py, and runner.py, this component is used to run experimental scenarios against the defined DAG models.
• Export Functionality (pdag/_export): The export modules, especially dot.py, facilitate converting the DAG into a visual representation (Graphviz DOT format).
• Notation Module (pdag/_notation): Through _decorators.py, export.py, and model.py files, this part provides a more user-friendly DSL for defining DAG structures. It translates higher-level notations into the underlying core model.
• Command-Line Interface (pdag/_cli): This component (found in _cli.py) is responsible for exposing functionality to end-users via command-line operations.
• Utility Libraries (pdag/utils): Contains useful helper functions (like _ast_utils.py, _dict_utils.py, _init_args_recorder.py, _multidef.py, _topological_sort.py) which are shared across the various layers.
• Examples and Tests: Located in src/pdag/examples and tests respectively, these serve to both demonstrate usage and validate the system architecture.
Relationships and Interactions
• The Notation Module feeds user-defined configurations into the Core Domain Model. Think of it as a translator that converts decorators and syntactic sugar into the underlying data structures.
• The Core Domain Model outputs a DAG (data and relationships) which then feeds into the Execution Engine. This engine uses the core’s model to perform operations (e.g., parameter resolution, dependency management) necessary for actual execution.
• The Experiment Runner interacts with the Execution Engine and sometimes directly with the Core to simulate different scenarios and record results.
• The CLI exposes these functionalities to users, serving as the entry point. It can trigger operations in any of the components (building the model, executing it, running experiments, or exporting).
• Utility Libraries support all layers by abstracting common tasks such as AST manipulations, dictionary operations, and topological sorting.
Architectural Patterns and Design Principles
• Layered Architecture: Separate responsibilities into distinct modules (notation, core model, execution, experiments, export, and CLI). This separation improves maintainability and testing.
• Modularity: Each directory under src/pdag (such as _core, _exec, _notation, etc.) encapsulates logically grouped functionality, enabling independent evolution.
• Clear Data Flows: The transformation path from user input (via notation and CLI) → DAG model creation (core) → DAG execution (exec) → experimentation and visualization (experiment and export) should be clearly represented.
Final Reminders
• Integration points (for example, the examples and tests folders demonstrate how the architecture is verified and used).
• Don’t hesitate to further decompose components if needed – for instance, showing internals of the Execution Engine by highlighting its submodules if that detail is valuable to the audience.
• Remember that the ultimate goal is to accurately capture the architectural elements and data flow in as much detail as practical without oversimplification.