How netx-lab/ayo works
Project type and purpose
• Ayo is a library/framework (open‐source tool) for fine‐grained orchestration and end-to-end optimization of LLM-based workflows, with local inference engines instead of remote APIs.
• Key goals: represent user-defined workflows as DAGs of “task primitives,” apply optimization passes (parallelism, pipelining, pruning), then schedule & execute across engines (LLM, vector DB, aggregators, rerankers).
File structure
• Top-level package “Ayo” contains subpackages:
• app.py: entry point (APP.init, engine registration, query submission)
• configs/: engine configuration definitions
• dags/: DAG and Node abstractions
• engines/: implementations for different engine types (LLM, vector_db, aggregator, embedder, reranker)
• modules/: high-level user modules (Indexing, Searching, Reranking, etc.)
• opt_pass/: optimization passes (dependency pruning, stage decomposition, prefill split, decoding pipelining) and pass manager
• schedulers/: two‐level scheduling (graph_scheduler and per‐engine engine_scheduler)
• queries/: query representation and state tracking
• vis/: DAG visualization helpers
• utils.py, logger.py
• examples/: concrete example scripts showing both unoptimized and optimized DAGs, with PNG visuals.
README for insights
• Ayo orchestrates workflows with Node primitives in a DAG.
• It automatically applies passes (PruningDependencyPass, StageDecompositionPass, PrefillingSplitPass, LLMDecodingPipeliningPass).
• Two‐level scheduling: a GraphScheduler distributes task primitives to EngineSchedulers; EngineSchedulers manage execution on local resources.
• External dependencies: local PG/Postgres with pgvector, modified vLLM, Ray, etc.
Main components and their interactions
a. User/API layer
• APP (app.py)
• Entrypoint to register engines, define DAGs, apply optimization passes, and submit queries.
b. Configuration layer
• EngineConfig and model_config classes under configs/
• Defines resources, parallelism, latency profiles.
c. Workflow definition layer
• Node, DAG classes (dags/)
• Modules-to-primitives transformer (modules/ + modules_to_primitives)
• Query builder (queries/).
d. Optimization layer
• OptPass Manager (opt_pass/pass_manager.py)
• Individual Passes (pruning_dependency, stage_decomposition, prefilling_split, decoding_pipelining).
e. Scheduling layer
• GraphScheduler (schedulers/graph_scheduler.py)
• EngineScheduler (schedulers/engine_scheduler.py)
• Orchestrates assignment of Node tasks to engines.
f. Execution engines
• LLM engine (engines/llm.py)
• Vector DB engine (engines/vector_db.py)
• Embedder, Aggregator, Reranker, etc.
• Each engine has a BaseEngine interface.
g. Persistence & External Services
• Local PostgreSQL + pgvector (for vector storage)
• Modified vLLM runtime for LLM inference
• Optionally Ray for distributed execution.
h. Visualization & Logging
• vis_graph (vis/)
• logger.py
• Provides DAG visual outputs, logging for debugging.
Diagramming recommendations
• Rectangles for core components (App Layer, Config Layer, Workflow Layer, Opt Layer, Scheduling Layer, Engines).
• Clouds for external runtimes (vLLM).
1. User script → APP
2. APP → register engines (EngineConfig)
3. APP → build DAG (Node/Module → primitives)
4. DAG → OptPassManager → Optimized DAG
5. Optimized DAG → GraphScheduler → EngineScheduler
6. EngineScheduler → Engine instances → external services (DB, vLLM)
7. Execution results flow back up to Query → User.
c. Annotations
• Parallel execution regions introduced by specific passes (e.g., Stage Decomp).
• Feedback loops if scheduling dynamically adapts at runtime.
• Call out external dependencies (Postgres, pgvector, vLLM).
Essential architectural elements to capture
• Two‐level scheduling (Graph vs. Engine)
• Pluggable optimization passes and their insertion point
• Engine registry/configuration mechanism
• Separation of high-level modules vs. low-level primitives
• Integration points with external systems (storage, inference runtime)
Summary
• Then refine each layer into its key classes/modules.