# gagancreates/feynmanai architecture

> Architecture diagram of the GitHub repository gagancreates/feynmanai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gagancreates/feynmanai
- Repository: https://github.com/gagancreates/feynmanai

## Overview

FeynmanAI is a voice-first AI tutor that turns student questions into spoken, incremental canvas explanations. The main flow is voice or chat input → agent prompt preparation with canvas context → streamed model actions → live tldraw edits and chat history; voice output speaks the explanation. The graph also shows multilingual speech services, model selection, context selection, and canvas linting. README documents the service-level flow; source excerpts establish key client and prompt calls. Some backend route-to-service wiring and persistence details are not sampled, so those links are omitted where unverified.

Main entry points: [`agent/client/agent/TldrawAgent.ts`](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgent.ts) (Tldraw agent), [`agent/worker/do/AgentService.ts`](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/do/AgentService.ts) (Model streaming), [`agent/client/agent/TldrawAgentApp.ts`](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgentApp.ts) (Agent app), [`agent/worker/routes/tts.ts`](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/routes/tts.ts) (Speech synthesis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["Tutor experience"]
  node_app["Canvas app<br/>[App.tsx]"]
  node_chat["Chat panel<br/>[ChatPanel.tsx]"]
  node_voicein["Voice capture<br/>[useVoiceInput.ts]"]
  node_voiceout["Voice playback<br/>[useVoiceOutput.ts]"]
  node_history["Chat history<br/>[ChatHistory.tsx]"]
end

subgraph group_agent["Agent runtime"]
  node_agentapp["Agent app<br/>[TldrawAgentApp.ts]"]
  node_agentcore["Tldraw agent<br/>[TldrawAgent.ts]"]
  node_persistence["App persistence"]
end

subgraph group_prompt["Prompt and context"]
  node_context["Canvas context"]
  node_promptparts["Prompt parts<br/>[PromptPartUtil.ts]"]
  node_focused["Shape format<br/>[FocusedShape.ts]"]
end

subgraph group_backend["Worker services"]
  node_stt["Speech transcription<br/>[stt.ts]"]
  node_tts["Speech synthesis<br/>[tts.ts]"]
  node_promptbuild["Prompt assembly<br/>[buildMessages.ts]"]
  node_systemprompt["Tutor instructions"]
  node_service["Model streaming<br/>[AgentService.ts]"]
  node_models["Model catalog<br/>[models.ts]"]
  node_worker["HTTP worker<br/>[worker.ts]"]
end

subgraph group_canvas["Canvas actions"]
  node_actionmanager["Action dispatch"]
  node_actionutils["Canvas actions<br/>[AgentActionUtil.ts]"]
  node_tldraw["Live canvas<br/>[App.tsx]"]
  node_lint["Canvas linting"]
end

node_student(("Student"))
node_sarvam["Sarvam AI"]
node_gemini_tts["Gemini TTS"]
node_provider["LLM providers"]

node_student -->|"asks question"| node_chat
node_student -->|"speaks question"| node_voicein
node_app -->|"renders"| node_chat
node_app -->|"uses playback"| node_voiceout
node_voicein -->|"sends audio"| node_stt
node_stt -->|"transcribes with"| node_sarvam
node_chat -.->|"submits prompt"| node_agentapp
node_agentapp -.->|"hosts agent"| node_agentcore
node_agentcore -->|"collects context"| node_context
node_agentcore -->|"transforms parts"| node_promptparts
node_context -->|"stores shapes"| node_focused
node_agentcore -->|"posts prompt"| node_worker
node_worker -->|"routes stream"| node_service
node_service -->|"builds messages"| node_promptbuild
node_service -->|"builds instructions"| node_systemprompt
node_service -->|"selects model"| node_models
node_service -->|"streams completion"| node_provider
node_service -->|"streams actions"| node_agentcore
node_agentcore -->|"dispatches actions"| node_actionmanager
node_actionmanager -->|"applies action"| node_actionutils
node_actionmanager -->|"edits canvas"| node_tldraw
node_agentcore -.->|"checks canvas"| node_lint
node_agentcore -.->|"updates history"| node_history
node_agentapp -.->|"persists state"| node_persistence
node_agentcore -->|"sends response text"| node_tts
node_tts -->|"synthesizes speech"| node_gemini_tts
node_tts -->|"returns audio"| node_voiceout
node_voiceout -->|"speaks response"| node_student

click node_app "https://github.com/gagancreates/feynmanai/blob/main/agent/client/App.tsx"
click node_chat "https://github.com/gagancreates/feynmanai/blob/main/agent/client/components/ChatPanel.tsx"
click node_voicein "https://github.com/gagancreates/feynmanai/blob/main/agent/client/hooks/useVoiceInput.ts"
click node_voiceout "https://github.com/gagancreates/feynmanai/blob/main/agent/client/hooks/useVoiceOutput.ts"
click node_stt "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/routes/stt.ts"
click node_tts "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/routes/tts.ts"
click node_agentapp "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgentApp.ts"
click node_agentcore "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgent.ts"
click node_context "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentContextManager.ts"
click node_promptparts "https://github.com/gagancreates/feynmanai/blob/main/agent/client/parts/PromptPartUtil.ts"
click node_focused "https://github.com/gagancreates/feynmanai/blob/main/agent/shared/format/FocusedShape.ts"
click node_promptbuild "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/prompt/buildMessages.ts"
click node_systemprompt "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/prompt/buildSystemPrompt.ts"
click node_service "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/do/AgentService.ts"
click node_models "https://github.com/gagancreates/feynmanai/blob/main/agent/shared/models.ts"
click node_worker "https://github.com/gagancreates/feynmanai/blob/main/agent/worker/worker.ts"
click node_actionmanager "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentActionManager.ts"
click node_actionutils "https://github.com/gagancreates/feynmanai/blob/main/agent/client/actions/AgentActionUtil.ts"
click node_tldraw "https://github.com/gagancreates/feynmanai/blob/main/agent/client/App.tsx"
click node_lint "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentLintManager.ts"
click node_history "https://github.com/gagancreates/feynmanai/blob/main/agent/client/components/chat-history/ChatHistory.tsx"
click node_persistence "https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentAppPersistenceManager.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_chat,node_voicein,node_voiceout,node_history toneBlue
class node_agentapp,node_agentcore,node_persistence toneAmber
class node_context,node_promptparts,node_focused toneMint
class node_stt,node_tts,node_promptbuild,node_systemprompt,node_service,node_models,node_worker toneRose
class node_actionmanager,node_actionutils,node_tldraw,node_lint,node_student,node_sarvam,node_gemini_tts,node_provider toneIndigo
```

## Components

### Tutor experience

- [Canvas app](https://github.com/gagancreates/feynmanai/blob/main/agent/client/App.tsx): `agent/client/App.tsx`
- [Chat panel](https://github.com/gagancreates/feynmanai/blob/main/agent/client/components/ChatPanel.tsx): `agent/client/components/ChatPanel.tsx`
- [Voice capture](https://github.com/gagancreates/feynmanai/blob/main/agent/client/hooks/useVoiceInput.ts): `agent/client/hooks/useVoiceInput.ts`
- [Voice playback](https://github.com/gagancreates/feynmanai/blob/main/agent/client/hooks/useVoiceOutput.ts): `agent/client/hooks/useVoiceOutput.ts`
- [Chat history](https://github.com/gagancreates/feynmanai/blob/main/agent/client/components/chat-history/ChatHistory.tsx): `agent/client/components/chat-history/ChatHistory.tsx`

### Agent runtime

- [Agent app](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgentApp.ts): `agent/client/agent/TldrawAgentApp.ts`
- [Tldraw agent](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/TldrawAgent.ts): `agent/client/agent/TldrawAgent.ts`
- [App persistence](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentAppPersistenceManager.ts): `agent/client/agent/managers/AgentAppPersistenceManager.ts`

### Prompt and context

- [Canvas context](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentContextManager.ts): `agent/client/agent/managers/AgentContextManager.ts`
- [Prompt parts](https://github.com/gagancreates/feynmanai/blob/main/agent/client/parts/PromptPartUtil.ts): `agent/client/parts/PromptPartUtil.ts`
- [Shape format](https://github.com/gagancreates/feynmanai/blob/main/agent/shared/format/FocusedShape.ts): `agent/shared/format/FocusedShape.ts`

### Worker services

- [Speech transcription](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/routes/stt.ts): `agent/worker/routes/stt.ts`
- [Speech synthesis](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/routes/tts.ts): `agent/worker/routes/tts.ts`
- [Prompt assembly](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/prompt/buildMessages.ts): `agent/worker/prompt/buildMessages.ts`
- [Tutor instructions](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/prompt/buildSystemPrompt.ts): `agent/worker/prompt/buildSystemPrompt.ts`
- [Model streaming](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/do/AgentService.ts): `agent/worker/do/AgentService.ts`
- [Model catalog](https://github.com/gagancreates/feynmanai/blob/main/agent/shared/models.ts): `agent/shared/models.ts`
- [HTTP worker](https://github.com/gagancreates/feynmanai/blob/main/agent/worker/worker.ts): `agent/worker/worker.ts`

### Canvas actions

- [Action dispatch](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentActionManager.ts): `agent/client/agent/managers/AgentActionManager.ts`
- [Canvas actions](https://github.com/gagancreates/feynmanai/blob/main/agent/client/actions/AgentActionUtil.ts): `agent/client/actions/AgentActionUtil.ts`
- [Live canvas](https://github.com/gagancreates/feynmanai/blob/main/agent/client/App.tsx): `agent/client/App.tsx`
- [Canvas linting](https://github.com/gagancreates/feynmanai/blob/main/agent/client/agent/managers/AgentLintManager.ts): `agent/client/agent/managers/AgentLintManager.ts`

### Other components

- **Student**
- **Sarvam AI**
- **Gemini TTS**
- **LLM providers**

## Connections

- Student → Chat panel: asks question
- Student → Voice capture: speaks question
- Canvas app → Chat panel: renders (evidence: `agent/client/App.tsx`)
- Canvas app → Voice playback: uses playback (evidence: `agent/client/App.tsx`)
- Voice capture → Speech transcription: sends audio
- Speech transcription → Sarvam AI: transcribes with
- Chat panel → Agent app: submits prompt
- Agent app → Tldraw agent: hosts agent
- Tldraw agent → Canvas context: collects context (evidence: `agent/client/agent/TldrawAgent.ts`)
- Tldraw agent → Prompt parts: transforms parts (evidence: `agent/client/agent/TldrawAgent.ts`)
- Canvas context → Shape format: stores shapes (evidence: `agent/client/agent/managers/AgentContextManager.ts`)
- Tldraw agent → HTTP worker: posts prompt (evidence: `agent/client/agent/TldrawAgent.ts`)
- HTTP worker → Model streaming: routes stream (evidence: `agent/worker/worker.ts`)
- Model streaming → Prompt assembly: builds messages (evidence: `agent/worker/do/AgentService.ts`)
- Model streaming → Tutor instructions: builds instructions (evidence: `agent/worker/do/AgentService.ts`)
- Model streaming → Model catalog: selects model (evidence: `agent/worker/do/AgentService.ts`)
- Model streaming → LLM providers: streams completion
- Model streaming → Tldraw agent: streams actions
- Tldraw agent → Action dispatch: dispatches actions (evidence: `agent/client/agent/TldrawAgent.ts`)
- Action dispatch → Canvas actions: applies action (evidence: `agent/client/agent/managers/AgentActionManager.ts`)
- Action dispatch → Live canvas: edits canvas (evidence: `agent/client/agent/managers/AgentActionManager.ts`)
- Tldraw agent → Canvas linting: checks canvas (evidence: `agent/client/agent/TldrawAgent.ts`)
- Tldraw agent → Chat history: updates history
- Agent app → App persistence: persists state
- Tldraw agent → Speech synthesis: sends response text
- Speech synthesis → Gemini TTS: synthesizes speech
- Speech synthesis → Voice playback: returns audio
- Voice playback → Student: speaks response
