# netx-lab/ayo architecture

> Architecture diagram of the GitHub repository netx-lab/ayo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netx-lab/ayo
- Repository: https://github.com/netx-lab/ayo

## Overview

##### Project type and purpose
- Ayo is a library/framework (open‐source tool) for fine‐grained orchestration and end-to-end optimization of LLM-based workflows, with local inference engines instead of remote APIs.
- Key goals: represent user-defined workflows as DAGs of “task primitives,” apply optimization passes (parallelism, pipelining, pruning), then schedule & execute across engines (LLM, vector DB, aggregators, rerankers).

##### File structure
- Top-level package “Ayo” contains subpackages:
- app.py: entry point (APP.init, engine registration, query submission)
- configs/: engine configuration definitions
- dags/: DAG and Node abstractions
- engines/: implementations for different engine types (LLM, vector_db, aggregator, embedder, reranker)
- modules/: high-level user modules (Indexing, Searching, Reranking, etc.)
- opt_pass/: optimization passes (dependency pruning, stage decomposition, prefill split, decoding pipelining) and pass manager
- schedulers/: two‐level scheduling (graph_scheduler and per‐engine engine_scheduler)
- queries/: query representation and state tracking
- vis/: DAG visualization helpers
- utils.py, logger.py
- examples/: concrete example scripts showing both unoptimized and optimized DAGs, with PNG visuals.

##### README for insights
- Ayo orchestrates workflows with Node primitives in a DAG.
- It automatically applies passes (PruningDependencyPass, StageDecompositionPass, PrefillingSplitPass, LLMDecodingPipeliningPass).
- Two‐level scheduling: a GraphScheduler distributes task primitives to EngineSchedulers; EngineSchedulers manage execution on local resources.
- External dependencies: local PG/Postgres with pgvector, modified vLLM, Ray, etc.

##### Main components and their interactions
a. User/API layer
- APP (app.py)
- Entrypoint to register engines, define DAGs, apply optimization passes, and submit queries.
b. Configuration layer
- EngineConfig and model_config classes under configs/
- Defines resources, parallelism, latency profiles.
c. Workflow definition layer
- Node, DAG classes (dags/)
- Modules-to-primitives transformer (modules/ + modules_to_primitives)
- Query builder (queries/).
d. Optimization layer
- OptPass Manager (opt_pass/pass_manager.py)
- Individual Passes (pruning_dependency, stage_decomposition, prefilling_split, decoding_pipelining).
e. Scheduling layer
- GraphScheduler (schedulers/graph_scheduler.py)
- EngineScheduler (schedulers/engine_scheduler.py)
- Orchestrates assignment of Node tasks to engines.
f. Execution engines
- LLM engine (engines/llm.py)
- Vector DB engine (engines/vector_db.py)
- Embedder, Aggregator, Reranker, etc.
- Each engine has a BaseEngine interface.
g. Persistence & External Services
- Local PostgreSQL + pgvector (for vector storage)
- Modified vLLM runtime for LLM inference
- Optionally Ray for distributed execution.
h. Visualization & Logging
- vis_graph (vis/)
- logger.py
- Provides DAG visual outputs, logging for debugging.

##### Diagramming recommendations
- Rectangles for core components (App Layer, Config Layer, Workflow Layer, Opt Layer, Scheduling Layer, Engines).
- Clouds for external runtimes (vLLM).
1. User script → APP
2. APP → register engines (EngineConfig)
3. APP → build DAG (Node/Module → primitives)
4. DAG → OptPassManager → Optimized DAG
5. Optimized DAG → GraphScheduler → EngineScheduler
6. EngineScheduler → Engine instances → external services (DB, vLLM)
7. Execution results flow back up to Query → User.
c. Annotations
- Parallel execution regions introduced by specific passes (e.g., Stage Decomp).
- Feedback loops if scheduling dynamically adapts at runtime.
- Call out external dependencies (Postgres, pgvector, vLLM).

##### Essential architectural elements to capture
- Two‐level scheduling (Graph vs. Engine)
- Pluggable optimization passes and their insertion point
- Engine registry/configuration mechanism
- Separation of high-level modules vs. low-level primitives
- Integration points with external systems (storage, inference runtime)

Summary
- Then refine each layer into its key classes/modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Script"]:::external
    App["APP (app.py)"]:::orchestration
    User -->|"invokes"| App
    click App "https://github.com/netx-lab/ayo/blob/main/Ayo/app.py"

    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        EngineConfig["EngineConfig (configs/config.py)"]:::config
        ModelConfig["ModelConfig (configs/model_config.py)"]:::config
    end
    App -->|"register_engines"| EngineConfig
    App -->|"load_model_configs"| ModelConfig
    click EngineConfig "https://github.com/netx-lab/ayo/blob/main/Ayo/configs/config.py"
    click ModelConfig "https://github.com/netx-lab/ayo/blob/main/Ayo/configs/model_config.py"

    %% Workflow Definition Layer
    subgraph "Workflow Definition Layer"
        direction TB
        DAG["DAG (dags/dag.py)"]:::workflow
        Node["Node (dags/node.py)"]:::workflow
        NodeCommons["NodeCommons (dags/node_commons.py)"]:::workflow
        ModuleToPrim["ModuleToPrimitives (modules/mod_to_prim.py)"]:::workflow
        subgraph "User Modules"
            direction TB
            BaseModule["BaseModule (modules/base_module.py)"]:::workflow
            Embedding["Embedding (modules/embedding.py)"]:::workflow
            Indexing["Indexing (modules/indexing.py)"]:::workflow
            LLMSynth["LLMSynth (modules/llm_syhthesizing.py)"]:::workflow
            PromptTemplate["PromptTemplate (modules/prompt_template.py)"]:::workflow
            QueryExpanding["QueryExpanding (modules/query_expanding.py)"]:::workflow
            Reranking["Reranking (modules/reranking.py)"]:::workflow
            Searching["Searching (modules/searching.py)"]:::workflow
        end
        Query["Query (queries/query.py)"]:::workflow
        QueryState["QueryState (queries/query_state.py)"]:::workflow
    end
    App -->|"build_DAG"| DAG
    DAG --> Node
    DAG --> NodeCommons
    DAG --> ModuleToPrim
    ModuleToPrim --> BaseModule
    ModuleToPrim --> Embedding
    ModuleToPrim --> Indexing
    ModuleToPrim --> LLMSynth
    ModuleToPrim --> PromptTemplate
    ModuleToPrim --> QueryExpanding
    ModuleToPrim --> Reranking
    ModuleToPrim --> Searching
    App -->|"init_query"| Query
    Query --> QueryState
    click DAG "https://github.com/netx-lab/ayo/blob/main/Ayo/dags/dag.py"
    click Node "https://github.com/netx-lab/ayo/blob/main/Ayo/dags/node.py"
    click NodeCommons "https://github.com/netx-lab/ayo/blob/main/Ayo/dags/node_commons.py"
    click ModuleToPrim "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/mod_to_prim.py"
    click BaseModule "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/base_module.py"
    click Embedding "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/embedding.py"
    click Indexing "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/indexing.py"
    click LLMSynth "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/llm_syhthesizing.py"
    click PromptTemplate "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/prompt_template.py"
    click QueryExpanding "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/query_expanding.py"
    click Reranking "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/reranking.py"
    click Searching "https://github.com/netx-lab/ayo/blob/main/Ayo/modules/searching.py"
    click Query "https://github.com/netx-lab/ayo/blob/main/Ayo/queries/query.py"
    click QueryState "https://github.com/netx-lab/ayo/blob/main/Ayo/queries/query_state.py"

    %% Optimization Layer
    subgraph "Optimization Layer"
        direction TB
        PassManager["OptPassManager (opt_pass/pass_manager.py)"]:::optimization
        Pruning["PruningDependencyPass (opt_pass/pruning_dependency.py)"]:::optimization
        StageDecomp["StageDecompositionPass (opt_pass/stage_decomposition.py)"]:::optimization
        Prefilling["PrefillingSplitPass (opt_pass/prefilling_split.py)"]:::optimization
        Decoding["LLMDecodingPipeliningPass (opt_pass/decoding_pipeling.py)"]:::optimization
        BasePass["BasePass (opt_pass/base_pass.py)"]:::optimization
    end
    DAG -->|"optimize"| PassManager
    PassManager --> Pruning
    PassManager --> StageDecomp
    PassManager --> Prefilling
    PassManager --> Decoding
    Pruning -->|"pruned_DAG"| PassManager
    StageDecomp -->|"stages"| PassManager
    Prefilling -->|"splits"| PassManager
    Decoding -->|"pipeline"| PassManager
    PassManager -->|"optimized_DAG"| GraphScheduler
    click PassManager "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/pass_manager.py"
    click Pruning "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/pruning_dependency.py"
    click StageDecomp "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/stage_decomposition.py"
    click Prefilling "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/prefilling_split.py"
    click Decoding "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/decoding_pipeling.py"
    click BasePass "https://github.com/netx-lab/ayo/blob/main/Ayo/opt_pass/base_pass.py"

    %% Scheduling Layer
    subgraph "Scheduling Layer"
        direction TB
        GraphScheduler["GraphScheduler (schedulers/graph_scheduler.py)"]:::scheduling
        EngineScheduler["EngineScheduler (schedulers/engine_scheduler.py)"]:::scheduling
    end
    GraphScheduler -->|"assign_tasks"| EngineScheduler
    click GraphScheduler "https://github.com/netx-lab/ayo/blob/main/Ayo/schedulers/graph_scheduler.py"
    click EngineScheduler "https://github.com/netx-lab/ayo/blob/main/Ayo/schedulers/engine_scheduler.py"

    %% Execution Engines
    subgraph "Execution Engines"
        direction TB
        BaseEngine["BaseEngine (engines/base_engine.py)"]:::engines
        LLmEngine["LLMEngine (engines/llm.py)"]:::engines
        VectorDBEngine["VectorDBEngine (engines/vector_db.py)"]:::engines
        Embedder["Embedder (engines/embedder.py)"]:::engines
        Aggregator["Aggregator (engines/aggregator.py)"]:::engines
        Reranker["Reranker (engines/reranker.py)"]:::engines
        EngineTypes["EngineTypes (engines/engine_types.py)"]:::engines
        PayloadTrans["PayloadTransformers (engines/payload_transformers.py)"]:::engines
    end
    EngineScheduler --> BaseEngine
    BaseEngine --> LLmEngine
    BaseEngine --> VectorDBEngine
    BaseEngine --> Embedder
    BaseEngine --> Aggregator
    BaseEngine --> Reranker
    BaseEngine --> EngineTypes
    BaseEngine --> PayloadTrans
    click BaseEngine "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/base_engine.py"
    click LLmEngine "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/llm.py"
    click VectorDBEngine "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/vector_db.py"
    click Embedder "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/embedder.py"
    click Aggregator "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/aggregator.py"
    click Reranker "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/reranker.py"
    click EngineTypes "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/engine_types.py"
    click PayloadTrans "https://github.com/netx-lab/ayo/blob/main/Ayo/engines/payload_transformers.py"

    %% External Services
    Postgres[(Postgres + pgvector)]:::external
    vLLM{{vLLM Runtime}}:::external
    Ray((Ray)):::external
    LLmEngine --> vLLM
    VectorDBEngine --> Postgres
    EngineScheduler --> Ray

    %% Results Feedback
    LLmEngine -->|"results"| Query
    VectorDBEngine -->|"results"| Query
    Reranker -->|"results"| Query
    Query -->|"response"| User

    %% Styles
    classDef orchestration fill:#cfe2f3,stroke:#6d9eeb,color:#0b5394
    classDef config fill:#e2efda,stroke:#6aa84f,color:#38761d
    classDef workflow fill:#fff2cc,stroke:#f1c232,color:#b45f06
    classDef optimization fill:#d9d2e9,stroke:#8e7cc3,color:#3c2f5e
    classDef scheduling fill:#fce5cd,stroke:#e69138,color:#85200c
    classDef engines fill:#ead1dc,stroke:#c27ba0,color:#741b47
    classDef external fill:#d0e0e3,stroke:#76a5af,color:#134f5c
```
