How nvidia/nemo-agent-toolkit works
Project Type and Purpose
• Type: A modular, framework-agnostic Python toolkit for building, running, profiling, observing, and evaluating “agent” workflows via a CLI and multiple frontend plugins.
• Purpose: Provide a unifying API and CLI (“nat” commands) that orchestrates agents, tools, LLM providers, retrievers, memory stores, observability, and evaluation systems.
File Structure (key directories)
• src/nat/cli → CLI commands and entrypoint
• src/nat/front_ends → Frontend plugins (console, FastAPI, MCP)
• src/nat/runtime & src/nat/builder → Core workflow construction and execution
• src/nat/agent → Agent implementations (react_agent, reasoning_agent, etc.)
• src/nat/llm, src/nat/retriever, src/nat/memory, src/nat/object_store → Pluggable back-ends
• src/nat/observability & src/nat/profiler → Telemetry, tracing, profiling
• src/nat/eval → Evaluation and benchmarking
README Insights
• CLI as primary user interface (“nat run”, “nat evaluate”, etc.)
• UI chat interface is an optional FastAPI frontend
• Plugins for console, MCP client/server, and FastAPI show a plugin pattern
Main Components and Their Interactions
a. User (CLI)
– Initiates commands via shell (nat run, nat evaluate, nat info, etc.)
b. CLI Layer (src/nat/cli)
– Parses commands, flags, config files
– Resolves workflows and delegates to:
• Frontend plugins (console, HTTP, MCP)
• Core runtime engine
c. Frontend Plugins (src/nat/front_ends)
– Console Frontend: Handles interactive console I/O
– FastAPI Frontend: Exposes HTTP & WebSocket endpoints + chat UI
– MCP Frontend: Exposes/publishes tools over the Model Context Protocol
d. Core Runtime & Builder
– Builder: Constructs workflows (agents + tools) from YAML/CLI input
– Runtime: Loads components, manages sessions, executes workflows
e. Workflow Engine
– Agents (react_agent, reasoning_agent, etc.) coordinate:
• Tool invocations (code execution, document search, external APIs)
• LLM calls via providers (OpenAI, NVIDIA NIM, LangChain, etc.)
– Retrievers handle embeddings & vector stores (Milvus, NeMo Retriever)
– Memory & Object Store for stateful storage (Redis, S3, MySQL)
f. Observability & Profiling
– Traces, spans, metrics exported to platforms (OpenTelemetry, Phoenix, Weave)
g. Evaluation
– Built-in evaluators for workflows, RAG, SWE benchmarks
Architectural Patterns and Principles
• Plugin Pattern: CLI → plugin registry → frontend, LLM, retriever, memory, object store
• Separation of Concerns:
– CLI layer vs. frontend vs. core runtime vs. back-end adapters
• Composition over Inheritance: Workflows composed of small functions/agents/tools
• Reactive/Observer Pattern: Internal event streams for intermediate steps & telemetry
Technologies, Frameworks, Libraries
• Python 3.11+
• FastAPI (HTTP/WS frontend)
• OpenTelemetry / Phoenix / Weave / Langfuse
• Vector stores: Milvus, DataFlywheel
• MCP Protocol
• Optional frameworks: LangChain, LlamaIndex, CrewAI, Semantic Kernel, Google ADK
Final Checklist
[ ] Clearly labeled components (CLI, Console FE, FastAPI FE, MCP FE, Runtime, Builder, Agents, Tools, Back-ends, Observability, Evaluation)
[ ] Plugin registry as central hub in CLI layer
[ ] Callouts or annotations for major file paths (e.g., src/nat/cli, src/nat/front_ends)