# nextgennvidia/riva-agi architecture

> Architecture diagram of the GitHub repository nextgennvidia/riva-agi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nextgennvidia/riva-agi
- Repository: https://github.com/nextgennvidia/riva-agi

## Overview

Riva-AGI combines multi-agent task orchestration, a document-backed RAG knowledge service, and a Gemini-powered voice interface. The principal task workflow accepts a request, classifies and routes it through orchestration, invokes a registered agent and its tools, then returns a response. RAG separately ingests and privacy-processes documents for retrieval and answer synthesis; voice provides a browser-facing conversation service. Relationships are limited to those shown in excerpts or the source index; unsampled integrations are omitted where their wiring is uncertain.

Main entry points: [`orchestration/orchestrator/main.py`](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/main.py) (Orchestrator graph), [`voice_speech/web_server.py`](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/web_server.py) (Voice web server), [`voice_speech/engine/gemini/session.py`](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/session.py) (Gemini voice session), [`rag_knowledge/ingestion/pipeline.py`](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/pipeline.py) (Document ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Task orchestration"]
  node_chat_entry["Chat entry<br/>[chat.py]"]
  node_orchestrator["Orchestrator graph<br/>[main.py]"]
  node_intent["Intent routing"]
  node_planner["Task planner<br/>[planner.py]"]
  node_executor["Task executor<br/>[executor.py]"]
  node_reviewer["Response reviewer<br/>[reviewer.py]"]
  node_state["Task state<br/>[state_manager.py]"]
end

subgraph group_agent_tools["Agents and tools"]
  node_registry["Agent registry<br/>[registry.py]"]
  node_agent_pool["Specialist agents<br/>[researcher.py]"]
  node_llm["Gemini agent runtime<br/>[llm.py]"]
  node_tool_registry["Tool registry<br/>[registry.py]"]
  node_builtin_tools["Built-in tools<br/>[web_tools.py]"]
end

subgraph group_knowledge["RAG knowledge"]
  node_ingestion["Document ingestion<br/>[pipeline.py]"]
  node_loaders["Document loaders<br/>[dispatch.py]"]
  node_privacy["Privacy checks<br/>[privacy.py]"]
  node_pdf_loader["PDF reader<br/>[pdf.py]"]
  node_word_reader["Word reader<br/>[word.py]"]
  node_qdrant[("Qdrant store<br/>[qdrant_storage.py]")]
  node_retriever["Knowledge retrieval<br/>[retriever.py]"]
  node_rag_service["RAG answer service<br/>[service.py]"]
  node_rag_gemini["Gemini synthesis<br/>[gemini_client.py]"]
  node_rag_prompts["RAG prompts<br/>[prompts.py]"]
end

subgraph group_voice["Voice conversation"]
  node_voice_server["Voice web server<br/>[web_server.py]"]
  node_voice_session["Gemini voice session<br/>[session.py]"]
  node_voice_stream["Voice streaming<br/>[streaming.py]"]
  node_voice_tools["Voice tools<br/>[tools.py]"]
  node_conversation_state["Conversation state<br/>[state.py]"]
  node_session_manager["Session manager<br/>[session_manager.py]"]
  node_voice_settings["Voice settings<br/>[settings.py]"]
  node_voice_prompts["Voice prompts<br/>[prompts.py]"]
end

node_user(("User"))
node_browser(("Browser user"))

node_user -->|"submits task"| node_chat_entry
node_chat_entry -->|"runs task"| node_orchestrator
node_orchestrator -->|"classifies request"| node_intent
node_orchestrator -->|"plans work"| node_planner
node_orchestrator -->|"dispatches work"| node_executor
node_orchestrator -->|"routes review"| node_reviewer
node_orchestrator -->|"looks up agents"| node_registry
node_orchestrator -->|"updates task state"| node_state
node_agent_pool -->|"registers agents"| node_registry
node_orchestrator -->|"invokes handler"| node_agent_pool
node_agent_pool -->|"calls Gemini"| node_llm
node_llm -->|"resolves tools"| node_tool_registry
node_tool_registry -->|"registers tools"| node_builtin_tools
node_ingestion -->|"loads documents"| node_loaders
node_ingestion -->|"checks privacy"| node_privacy
node_loaders -->|"dispatches PDFs"| node_pdf_loader
node_loaders -->|"dispatches Word files"| node_word_reader
node_ingestion -->|"upserts documents"| node_qdrant
node_rag_service -->|"retrieves matches"| node_retriever
node_retriever -->|"searches store"| node_qdrant
node_rag_service -->|"synthesizes answer"| node_rag_gemini
node_rag_gemini -->|"formats prompt"| node_rag_prompts
node_browser -.->|"opens conversation"| node_voice_server
node_voice_server -->|"manages sessions"| node_session_manager
node_voice_server -->|"uses state"| node_conversation_state
node_voice_server -->|"starts sessions"| node_voice_session
node_voice_server -->|"uses streaming"| node_voice_stream
node_voice_session -->|"loads settings"| node_voice_settings
node_voice_session -->|"loads prompts"| node_voice_prompts
node_voice_session -->|"uses tools"| node_voice_tools
node_voice_tools -->|"uses agent tools"| node_tool_registry

click node_chat_entry "https://github.com/nextgennvidia/riva-agi/blob/main/chat.py"
click node_orchestrator "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/main.py"
click node_intent "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/intent_classifier.py"
click node_planner "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/planner.py"
click node_executor "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/executor.py"
click node_reviewer "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/reviewer.py"
click node_state "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/state_manager.py"
click node_registry "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/registry.py"
click node_agent_pool "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/agents/researcher.py"
click node_llm "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/llm.py"
click node_tool_registry "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/tools/registry.py"
click node_builtin_tools "https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/tools/builtin/web_tools.py"
click node_ingestion "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/pipeline.py"
click node_loaders "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/loaders/dispatch.py"
click node_privacy "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/privacy.py"
click node_pdf_loader "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/loaders/pdf.py"
click node_word_reader "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/readers/word.py"
click node_qdrant "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/storage/qdrant_storage.py"
click node_retriever "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/retrieval/retriever.py"
click node_rag_service "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/service.py"
click node_rag_gemini "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/clients/gemini_client.py"
click node_rag_prompts "https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/prompts.py"
click node_voice_server "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/web_server.py"
click node_voice_session "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/session.py"
click node_voice_stream "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/streaming.py"
click node_voice_tools "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/tools.py"
click node_conversation_state "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/conversation/state.py"
click node_session_manager "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/conversation/session_manager.py"
click node_voice_settings "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/config/settings.py"
click node_voice_prompts "https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/config/prompts.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chat_entry,node_orchestrator,node_intent,node_planner,node_executor,node_reviewer,node_state,node_user,node_browser toneBlue
class node_registry,node_agent_pool,node_llm,node_tool_registry,node_builtin_tools toneAmber
class node_ingestion,node_loaders,node_privacy,node_pdf_loader,node_word_reader,node_qdrant,node_retriever,node_rag_service,node_rag_gemini,node_rag_prompts toneMint
class node_voice_server,node_voice_session,node_voice_stream,node_voice_tools,node_conversation_state,node_session_manager,node_voice_settings,node_voice_prompts toneRose
```

## Components

### Task orchestration

- [Chat entry](https://github.com/nextgennvidia/riva-agi/blob/main/chat.py): `chat.py`
- [Orchestrator graph](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/main.py): `orchestration/orchestrator/main.py`
- [Intent routing](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/intent_classifier.py): `orchestration/orchestrator/intent_classifier.py`
- [Task planner](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/planner.py): `orchestration/orchestrator/planner.py`
- [Task executor](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/executor.py): `orchestration/orchestrator/executor.py`
- [Response reviewer](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/reviewer.py): `orchestration/orchestrator/reviewer.py`
- [Task state](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/state_manager.py): `orchestration/orchestrator/state_manager.py`

### Agents and tools

- [Agent registry](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/registry.py): `orchestration/orchestrator/registry.py`
- [Specialist agents](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/agents/researcher.py): `orchestration/agents/researcher.py`
- [Gemini agent runtime](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/orchestrator/llm.py): `orchestration/orchestrator/llm.py`
- [Tool registry](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/tools/registry.py): `orchestration/tools/registry.py`
- [Built-in tools](https://github.com/nextgennvidia/riva-agi/blob/main/orchestration/tools/builtin/web_tools.py): `orchestration/tools/builtin/web_tools.py`

### RAG knowledge

- [Document ingestion](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/pipeline.py): `rag_knowledge/ingestion/pipeline.py`
- [Document loaders](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/loaders/dispatch.py): `rag_knowledge/ingestion/loaders/dispatch.py`
- [Privacy checks](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/privacy.py): `rag_knowledge/ingestion/privacy.py`
- [PDF reader](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/loaders/pdf.py): `rag_knowledge/ingestion/loaders/pdf.py`
- [Word reader](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/ingestion/readers/word.py): `rag_knowledge/ingestion/readers/word.py`
- [Qdrant store](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/storage/qdrant_storage.py): `rag_knowledge/storage/qdrant_storage.py`
- [Knowledge retrieval](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/retrieval/retriever.py): `rag_knowledge/retrieval/retriever.py`
- [RAG answer service](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/service.py): `rag_knowledge/service.py`
- [Gemini synthesis](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/clients/gemini_client.py): `rag_knowledge/clients/gemini_client.py`
- [RAG prompts](https://github.com/nextgennvidia/riva-agi/blob/main/rag_knowledge/prompts.py): `rag_knowledge/prompts.py`

### Voice conversation

- [Voice web server](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/web_server.py): `voice_speech/web_server.py`
- [Gemini voice session](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/session.py): `voice_speech/engine/gemini/session.py`
- [Voice streaming](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/streaming.py): `voice_speech/engine/gemini/streaming.py`
- [Voice tools](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/gemini/tools.py): `voice_speech/engine/gemini/tools.py`
- [Conversation state](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/conversation/state.py): `voice_speech/engine/conversation/state.py`
- [Session manager](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/conversation/session_manager.py): `voice_speech/engine/conversation/session_manager.py`
- [Voice settings](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/config/settings.py): `voice_speech/engine/config/settings.py`
- [Voice prompts](https://github.com/nextgennvidia/riva-agi/blob/main/voice_speech/engine/config/prompts.py): `voice_speech/engine/config/prompts.py`

### Other components

- **User**
- **Browser user**

## Connections

- User → Chat entry: submits task (evidence: `chat.py`)
- Chat entry → Orchestrator graph: runs task (evidence: `chat.py`)
- Orchestrator graph → Intent routing: classifies request (evidence: `orchestration/orchestrator/main.py`)
- Orchestrator graph → Task planner: plans work (evidence: `orchestration/orchestrator/main.py`)
- Orchestrator graph → Task executor: dispatches work (evidence: `orchestration/orchestrator/main.py`)
- Orchestrator graph → Response reviewer: routes review (evidence: `orchestration/orchestrator/main.py`)
- Orchestrator graph → Agent registry: looks up agents (evidence: `orchestration/orchestrator/main.py`)
- Orchestrator graph → Task state: updates task state (evidence: `orchestration/orchestrator/main.py`)
- Specialist agents → Agent registry: registers agents (evidence: `orchestration/agents/researcher.py`)
- Orchestrator graph → Specialist agents: invokes handler (evidence: `orchestration/orchestrator/main.py`)
- Specialist agents → Gemini agent runtime: calls Gemini (evidence: `orchestration/agents/researcher.py`)
- Gemini agent runtime → Tool registry: resolves tools (evidence: `orchestration/orchestrator/llm.py`)
- Tool registry → Built-in tools: registers tools (evidence: `orchestration/tools/builtin/web_tools.py`)
- Document ingestion → Document loaders: loads documents (evidence: `rag_knowledge/ingestion/pipeline.py`)
- Document ingestion → Privacy checks: checks privacy (evidence: `rag_knowledge/ingestion/pipeline.py`)
- Document loaders → PDF reader: dispatches PDFs (evidence: `rag_knowledge/ingestion/loaders/dispatch.py`)
- Document loaders → Word reader: dispatches Word files (evidence: `rag_knowledge/ingestion/loaders/dispatch.py`)
- Document ingestion → Qdrant store: upserts documents (evidence: `rag_knowledge/ingestion/pipeline.py`)
- RAG answer service → Knowledge retrieval: retrieves matches (evidence: `rag_knowledge/service.py`)
- Knowledge retrieval → Qdrant store: searches store (evidence: `rag_knowledge/retrieval/retriever.py`)
- RAG answer service → Gemini synthesis: synthesizes answer (evidence: `rag_knowledge/service.py`)
- Gemini synthesis → RAG prompts: formats prompt (evidence: `rag_knowledge/clients/gemini_client.py`)
- Browser user → Voice web server: opens conversation
- Voice web server → Session manager: manages sessions (evidence: `voice_speech/web_server.py`)
- Voice web server → Conversation state: uses state (evidence: `voice_speech/web_server.py`)
- Voice web server → Gemini voice session: starts sessions (evidence: `voice_speech/web_server.py`)
- Voice web server → Voice streaming: uses streaming (evidence: `voice_speech/web_server.py`)
- Gemini voice session → Voice settings: loads settings (evidence: `voice_speech/engine/gemini/session.py`)
- Gemini voice session → Voice prompts: loads prompts (evidence: `voice_speech/engine/gemini/session.py`)
- Gemini voice session → Voice tools: uses tools (evidence: `voice_speech/engine/gemini/session.py`)
- Voice tools → Tool registry: uses agent tools (evidence: `voice_speech/engine/gemini/tools.py`)
