# nvidia/nemo-agent-toolkit architecture

> Architecture diagram of the GitHub repository nvidia/nemo-agent-toolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/nemo-agent-toolkit
- Repository: https://github.com/nvidia/nemo-agent-toolkit

## Overview

##### Project Type and Purpose
- Type: A modular, framework-agnostic Python toolkit for building, running, profiling, observing, and evaluating “agent” workflows via a CLI and multiple frontend plugins.
- Purpose: Provide a unifying API and CLI (“nat” commands) that orchestrates agents, tools, LLM providers, retrievers, memory stores, observability, and evaluation systems.

##### File Structure (key directories)
- src/nat/cli → CLI commands and entrypoint
- src/nat/front_ends → Frontend plugins (console, FastAPI, MCP)
- src/nat/runtime & src/nat/builder → Core workflow construction and execution
- src/nat/agent → Agent implementations (react_agent, reasoning_agent, etc.)
- src/nat/llm, src/nat/retriever, src/nat/memory, src/nat/object_store → Pluggable back-ends
- src/nat/observability & src/nat/profiler → Telemetry, tracing, profiling
- src/nat/eval → Evaluation and benchmarking

##### README Insights
- CLI as primary user interface (“nat run”, “nat evaluate”, etc.)
- UI chat interface is an optional FastAPI frontend
- Plugins for console, MCP client/server, and FastAPI show a plugin pattern

##### Main Components and Their Interactions
a. User (CLI)
– Initiates commands via shell (`nat run`, `nat evaluate`, `nat info`, etc.)
b. CLI Layer (`src/nat/cli`)
– Parses commands, flags, config files
– Resolves workflows and delegates to:
- Frontend plugins (console, HTTP, MCP)
- Core runtime engine
c. Frontend Plugins (`src/nat/front_ends`)
– Console Frontend: Handles interactive console I/O
– FastAPI Frontend: Exposes HTTP & WebSocket endpoints + chat UI
– MCP Frontend: Exposes/publishes tools over the Model Context Protocol
d. Core Runtime & Builder
– Builder: Constructs workflows (agents + tools) from YAML/CLI input
– Runtime: Loads components, manages sessions, executes workflows
e. Workflow Engine
– Agents (react_agent, reasoning_agent, etc.) coordinate:
- Tool invocations (code execution, document search, external APIs)
- LLM calls via providers (OpenAI, NVIDIA NIM, LangChain, etc.)
– Retrievers handle embeddings & vector stores (Milvus, NeMo Retriever)
– Memory & Object Store for stateful storage (Redis, S3, MySQL)
f. Observability & Profiling
– Traces, spans, metrics exported to platforms (OpenTelemetry, Phoenix, Weave)
g. Evaluation
– Built-in evaluators for workflows, RAG, SWE benchmarks

##### Architectural Patterns and Principles
- Plugin Pattern: CLI → plugin registry → frontend, LLM, retriever, memory, object store
- Separation of Concerns:
– CLI layer vs. frontend vs. core runtime vs. back-end adapters
- Composition over Inheritance: Workflows composed of small functions/agents/tools
- Reactive/Observer Pattern: Internal event streams for intermediate steps & telemetry

##### Technologies, Frameworks, Libraries
- Python 3.11+
- FastAPI (HTTP/WS frontend)
- OpenTelemetry / Phoenix / Weave / Langfuse
- Vector stores: Milvus, DataFlywheel
- MCP Protocol
- Optional frameworks: LangChain, LlamaIndex, CrewAI, Semantic Kernel, Google ADK

##### Final Checklist
[ ] Clearly labeled components (CLI, Console FE, FastAPI FE, MCP FE, Runtime, Builder, Agents, Tools, Back-ends, Observability, Evaluation)
[ ] Plugin registry as central hub in CLI layer
[ ] Callouts or annotations for major file paths (e.g., src/nat/cli, src/nat/front_ends)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User"]:::user
    Entry["Entry Point"]:::cli
    MainCmd["Main CLI"]:::cli
    Commands["Commands & Type Registry"]:::cli
    Registry["Plugin Registry"]:::cli

    User -->|"runs nat command"| Entry
    Entry -->|"invokes main"| MainCmd
    MainCmd -->|"parses input"| Commands
    Commands -->|"loads plugins"| Registry

    %% Frontend Plugins
    subgraph "Frontend Plugins"
        direction TB
        Console["Console FE"]:::frontend
        FastAPIFE["FastAPI FE"]:::frontend
        MCPFE["MCP FE"]:::frontend
    end

    Registry -->|"dispatch CLI"| Console
    Registry -->|"dispatch CLI"| FastAPIFE
    Registry -->|"dispatch CLI"| MCPFE

    %% Core Runtime & Builder
    subgraph "Core Components"
        direction TB
        Builder["Builder"]:::core
        Runtime["Runtime"]:::core
    end

    Registry -->|"invoke builder"| Builder
    Builder -->|"workflow config"| Runtime

    Console -->|"calls runtime"| Runtime
    FastAPIFE -->|"HTTP/WS request"| Runtime
    MCPFE -->|"MCP protocol"| Runtime

    %% Workflow Engine
    subgraph "Workflow Engine"
        direction TB
        Agents["Agents"]:::engine
    end

    Runtime -->|"execute workflow"| Agents

    %% Adapters & Storage
    subgraph "Adapters & Storage"
        direction TB
        subgraph "LLM Providers"
            LLM["LLM Plugins"]:::backend
        end
        subgraph "Retrievers"
            Retriever["Retrievers"]:::backend
        end
        subgraph "Memory Stores"
            Memory["Memory Store"]:::backend
        end
        subgraph "Object Store"
            ObjStore["Object Store"]:::backend
        end
    end

    Agents -->|"LLM calls"| LLM
    Agents -->|"vector search"| Retriever
    Agents -->|"read/write state"| Memory
    Agents -->|"object ops"| ObjStore

    %% Observability & Profiling
    subgraph "Observability & Profiling"
        direction TB
        Obs["Observability"]:::obs
        Prof["Profiling"]:::obs
    end

    Runtime -->|"emit traces/metrics"| Obs
    Runtime -->|"collect profiles"| Prof

    %% Evaluation
    Eval["Evaluation"]:::eval

    Runtime -->|"benchmark workflows"| Eval
    Runtime -->|"return results"| Commands
    Commands -->|"output to"| User

    %% Click Events
    click Entry "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/cli/entrypoint.py"
    click MainCmd "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/cli/main.py"
    click Commands "https://github.com/nvidia/nemo-agent-toolkit/tree/develop/src/nat/cli/commands/**"
    click Commands "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/cli/type_registry.py"
    click Registry "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/cli/register_workflow.py"
    click Registry "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/cli/type_registry.py"
    click Console "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/console/console_front_end_plugin.py"
    click Console "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/console/register.py"
    click FastAPIFE "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/fastapi/fastapi_front_end_plugin.py"
    click FastAPIFE "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/fastapi/fastapi_front_end_controller.py"
    click MCPFE "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/mcp/mcp_front_end_plugin.py"
    click MCPFE "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/front_ends/mcp/register.py"
    click Builder "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/builder/workflow_builder.py"
    click Builder "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/builder/builder.py"
    click Runtime "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/runtime/runner.py"
    click Runtime "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/runtime/session.py"
    click Agents "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/agent/react_agent/agent.py"
    click Agents "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/agent/reasoning_agent/reasoning_agent.py"
    click LLM "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/llm/openai_llm.py"
    click LLM "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/llm/nim_llm.py"
    click LLM "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/llm/azure_openai_llm.py"
    click LLM "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/llm/register.py"
    click Retriever "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/retriever/milvus/retriever.py"
    click Retriever "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/retriever/nemo_retriever/retriever.py"
    click Memory "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/memory/models.py"
    click Memory "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/memory/interfaces.py"
    click ObjStore "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/object_store/in_memory_object_store.py"
    click ObjStore "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/object_store/register.py"
    click Obs "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/observability/exporter/*.py"
    click Obs "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/observability/processor/*.py"
    click Prof "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/profiler/profile_runner.py"
    click Prof "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/profiler/calc/calc_runner.py"
    click Eval "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/eval/evaluate.py"
    click Eval "https://github.com/nvidia/nemo-agent-toolkit/blob/develop/src/nat/eval/rag_evaluator/evaluate.py"

    %% Styles
    classDef user fill:#a2fca2,stroke:#333,stroke-width:1px
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef cli fill:#90ee90,stroke:#333,stroke-width:1px
    classDef core fill:#ffa500,stroke:#333,stroke-width:1px
    classDef engine fill:#dda0dd,stroke:#333,stroke-width:1px
    classDef backend fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef obs fill:#ff6961,stroke:#333,stroke-width:1px
    classDef eval fill:#ffb6c1,stroke:#333,stroke-width:1px
```
