# msinamsina/ai-graph architecture

> Architecture diagram of the GitHub repository msinamsina/ai-graph, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/msinamsina/ai-graph
- Repository: https://github.com/msinamsina/ai-graph

## Overview

##### Project type and purpose
- AI-Graph is a Python library (not an end-user application) that lets clients build and execute data-processing pipelines using the Chain of Responsibility pattern.
- Its core goal is to provide a modular, type-safe framework of “steps” (transformations) and “pipelines” (step chains), including built-in support for looping (ForEachStep) and progress tracking.

##### File structure
- ai_graph/
– pipeline/
- base.py → defines Pipeline class (manages step chain, process entry point).
– step/
- base.py → BasePipelineStep interface (defines set_next, process, _process_step).
- foreach.py → ForEachStep (looping sub-pipeline).
- video/basic.py → example/video domain step.
- tests/… mirror the ai_graph package structure for unit tests.
- docs/, examples/… code samples and documentation.
- config files (.flake8, pyproject.toml, mypy.ini, etc.) support linting, formatting, CI, and packaging.

##### README for additional insights
- Describes the “Pipeline Steps” and “Pipelines” abstractions, with code snippets showing how clients import Pipeline, BasePipelineStep, and ForEachStep.
- Emphasizes extensibility (clients can subclass BasePipelineStep), progress bars via tqdm, 100% pytest coverage, and modern packaging with uv.
- No external microservices or databases—this is a self-contained library.

- Client code (user’s scripts or application)
- ai_graph.pipeline.Pipeline (pipeline orchestrator)
- ai_graph.step.BasePipelineStep (abstract step interface)
- Built-in Steps (AddKeyStep, DelKeyStep, VideoStep, ForEachStep)
- ForEachStep sub-pipeline (reuses BasePipelineStep chain)
- Progress bar integration (tqdm)
- External runtime environment (Python 3.8+, pytest, mypy)
- Client code → instantiates Pipeline → calls process(data) → enters Pipeline.
- Pipeline → invokes process() on first BasePipelineStep → each step calls next.process().
- ForEachStep → splits into sub-pipeline instances for each item/iteration → collects sub-pipeline outputs → merges into results_key.
- Steps → modify the data dict and pass it along.
- Pipeline/Steps → optionally invoke tqdm progress bar.
c. Architectural patterns & principles:
- Chain of Responsibility (each step holds a reference to the next).
- Open/Closed Principle (clients extend BasePipelineStep to add new behaviors).
- Single Responsibility (each Step does one transformation).
d. Key technologies/frameworks:
- Python 3.8+
- pytest (testing)
- mypy (type checking)
- tqdm (progress bars)
- uv (packaging/CLI integrations)

##### Detail without over-fragmentation
- Every core package and class.
- Both sequential flow (chain of steps) and iterative flow (ForEachStep sub-pipelines).
- Exclude low-level implementation files (like .cfg files) from the main architecture view—note them as “Developer Tooling” if needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Code Layer
    subgraph "User Code" 
        Client["Client Code"]:::external
        CustomStep["Custom Step\n(subclass of BasePipelineStep)"]:::custom
    end

    %% Core Engine Layer
    subgraph "Core Engine" 
        Pipeline["Pipeline\n.pipeline.process()"]:::core
        BaseStep["BasePipelineStep\n._process_step()"]:::core
    end

    %% Built-in Steps Layer
    subgraph "Built-in Steps" 
        AddKey["AddKeyStep"]:::builtin
        DelKey["DelKeyStep"]:::builtin
        VideoStep["VideoBasicStep"]:::builtin
        ForEach["ForEachStep\n(looping)"]:::builtin
    end

    %% Progress Integration
    Progress["tqdm Progress Bar"]:::external

    %% Developer Tooling Layer
    subgraph "Developer Tooling"
        Tests["Unit Tests"]:::tooling
        Docs["Documentation (RST)"]:::tooling
        Configs["Config Files\n(.flake8, mypy.ini, etc)"]:::tooling
    end

    %% Tests breakdown
    subgraph "Tests"
        PipelineTest["Pipeline Tests"]:::tooling
        StepTest["BasePipelineStep Tests"]:::tooling
        ForEachTest["ForEachStep Tests"]:::tooling
        VideoTest["VideoStep Tests"]:::tooling
    end

    %% Relationships
    Client -->|"instantiates"| Pipeline
    Pipeline -->|"calls process() on"| BaseStep
    BaseStep -->|"chains to"| AddKey
    AddKey -->|"next"| DelKey
    DelKey -->|"next"| VideoStep
    VideoStep -->|"next"| ForEach
    ForEach -->|"iterates and spawns sub-pipeline"| BaseStep
    ForEach -->|"aggregates results"| BaseStep
    BaseStep -->|"invokes"| Progress
    CustomStep -->|"implements"| BaseStep

    %% Developer Tooling relationships
    Pipeline --> Tests
    BaseStep --> Tests
    ForEach --> Tests
    VideoStep --> Tests
    Tests -->|tests located in| PipelineTest
    Tests -->|tests located in| StepTest
    Tests -->|tests located in| ForEachTest
    Tests -->|tests located in| VideoTest
    Docs -->|documents core API| Pipeline
    Docs -->|documents core API| BaseStep
    Docs -->|documents steps| ForEach
    Configs -.->|lint/type/ci| Pipeline
    Configs -.->|lint/type/ci| BaseStep
    Configs -.->|lint/type/ci| ForEach

    %% Click Events
    click Pipeline "https://github.com/msinamsina/ai-graph/blob/main/ai_graph/pipeline/base.py"
    click BaseStep "https://github.com/msinamsina/ai-graph/blob/main/ai_graph/step/base.py"
    click ForEach "https://github.com/msinamsina/ai-graph/blob/main/ai_graph/step/foreach.py"
    click VideoStep "https://github.com/msinamsina/ai-graph/blob/main/ai_graph/step/video/basic.py"
    click PipelineTest "https://github.com/msinamsina/ai-graph/blob/main/tests/pipeline/test_base.py"
    click StepTest "https://github.com/msinamsina/ai-graph/blob/main/tests/step/test_base.py"
    click ForEachTest "https://github.com/msinamsina/ai-graph/blob/main/tests/step/test_foreach.py"
    click VideoTest "https://github.com/msinamsina/ai-graph/blob/main/tests/step/test_video/test_basic.py"
    click CustomStep "https://github.com/msinamsina/ai-graph/blob/main/examples/ai_graph_example.ipynb"
    click Docs "https://github.com/msinamsina/ai-graph/blob/main/docs/index.rst"

    %% Styles
    classDef core fill:#f9f,stroke:#333,stroke-width:1px
    classDef builtin fill:#9ff,stroke:#333,stroke-width:1px
    classDef custom fill:#ff9,stroke:#333,stroke-width:1px
    classDef external fill:#ccc,stroke:#333,stroke-width:1px
    classDef tooling fill:#eef,stroke:#333,stroke-width:1px
```
