How msinamsina/ai-graph works
Project type and purpose
• AI-Graph is a Python library (not an end-user application) that lets clients build and execute data-processing pipelines using the Chain of Responsibility pattern.
• Its core goal is to provide a modular, type-safe framework of “steps” (transformations) and “pipelines” (step chains), including built-in support for looping (ForEachStep) and progress tracking.
File structure
• ai_graph/
– pipeline/
• base.py → defines Pipeline class (manages step chain, process entry point).
– step/
• base.py → BasePipelineStep interface (defines set_next, process, _process_step).
• foreach.py → ForEachStep (looping sub-pipeline).
• video/basic.py → example/video domain step.
• tests/… mirror the ai_graph package structure for unit tests.
• docs/, examples/… code samples and documentation.
• config files (.flake8, pyproject.toml, mypy.ini, etc.) support linting, formatting, CI, and packaging.
README for additional insights
• Describes the “Pipeline Steps” and “Pipelines” abstractions, with code snippets showing how clients import Pipeline, BasePipelineStep, and ForEachStep.
• Emphasizes extensibility (clients can subclass BasePipelineStep), progress bars via tqdm, 100% pytest coverage, and modern packaging with uv.
• No external microservices or databases—this is a self-contained library.
• Client code (user’s scripts or application)
• ai_graph.pipeline.Pipeline (pipeline orchestrator)
• ai_graph.step.BasePipelineStep (abstract step interface)
• Built-in Steps (AddKeyStep, DelKeyStep, VideoStep, ForEachStep)
• ForEachStep sub-pipeline (reuses BasePipelineStep chain)
• Progress bar integration (tqdm)
• External runtime environment (Python 3.8+, pytest, mypy)
• Client code → instantiates Pipeline → calls process(data) → enters Pipeline.
• Pipeline → invokes process() on first BasePipelineStep → each step calls next.process().
• ForEachStep → splits into sub-pipeline instances for each item/iteration → collects sub-pipeline outputs → merges into results_key.
• Steps → modify the data dict and pass it along.
• Pipeline/Steps → optionally invoke tqdm progress bar.
c. Architectural patterns & principles:
• Chain of Responsibility (each step holds a reference to the next).
• Open/Closed Principle (clients extend BasePipelineStep to add new behaviors).
• Single Responsibility (each Step does one transformation).
d. Key technologies/frameworks:
• Python 3.8+
• pytest (testing)
• mypy (type checking)
• tqdm (progress bars)
• uv (packaging/CLI integrations)
Detail without over-fragmentation
• Every core package and class.
• Both sequential flow (chain of steps) and iterative flow (ForEachStep sub-pipelines).
• Exclude low-level implementation files (like .cfg files) from the main architecture view—note them as “Developer Tooling” if needed.