How gagancreates/feynmanai works
FeynmanAI is a voice-first AI tutor that turns student questions into spoken, incremental canvas explanations. The main flow is voice or chat input → agent prompt preparation with canvas context → streamed model actions → live tldraw edits and chat history; voice output speaks the explanation. The graph also shows multilingual speech services, model selection, context selection, and canvas linting. README documents the service-level flow; source excerpts establish key client and prompt calls. Some backend route-to-service wiring and persistence details are not sampled, so those links are omitted where unverified.
Main entry points: TldrawAgent.ts Tldraw agent · AgentService.ts Model streaming · TldrawAgentApp.ts Agent app · tts.ts Speech synthesis
Components 26
Tutor experience
- Canvas app
agent/client/App.tsx - Chat panel
agent/client/components/ChatPanel.tsx - Voice capture
agent/client/hooks/useVoiceInput.ts - Voice playback
agent/client/hooks/useVoiceOutput.ts - Chat history
agent/client/components/chat-history/ChatHistory.tsx
Agent runtime
- Agent app
agent/client/agent/TldrawAgentApp.ts - Tldraw agent
agent/client/agent/TldrawAgent.ts - App persistence
agent/client/agent/managers/AgentAppPersistenceManager.ts
Prompt and context
- Canvas context
agent/client/agent/managers/AgentContextManager.ts - Prompt parts
agent/client/parts/PromptPartUtil.ts - Shape format
agent/shared/format/FocusedShape.ts
Worker services
- Speech transcription
agent/worker/routes/stt.ts - Speech synthesis
agent/worker/routes/tts.ts - Prompt assembly
agent/worker/prompt/buildMessages.ts - Tutor instructions
agent/worker/prompt/buildSystemPrompt.ts - Model streaming
agent/worker/do/AgentService.ts - Model catalog
agent/shared/models.ts - HTTP worker
agent/worker/worker.ts
Canvas actions
- Action dispatch
agent/client/agent/managers/AgentActionManager.ts - Canvas actions
agent/client/actions/AgentActionUtil.ts - Live canvas
agent/client/App.tsx - Canvas linting
agent/client/agent/managers/AgentLintManager.ts
Other components
- Student
- Sarvam AI
- Gemini TTS
- LLM providers
Connections 28
- Student → Chat panel: asks question
- Student → Voice capture: speaks question
- Canvas app → Chat panel: renders
agent/client/App.tsx - Canvas app → Voice playback: uses playback
agent/client/App.tsx - Voice capture → Speech transcription: sends audio
- Speech transcription → Sarvam AI: transcribes with
- Chat panel → Agent app: submits prompt
- Agent app → Tldraw agent: hosts agent
- Tldraw agent → Canvas context: collects context
agent/client/agent/TldrawAgent.ts - Tldraw agent → Prompt parts: transforms parts
agent/client/agent/TldrawAgent.ts - Canvas context → Shape format: stores shapes
agent/client/agent/managers/AgentContextManager.ts - Tldraw agent → HTTP worker: posts prompt
agent/client/agent/TldrawAgent.ts - HTTP worker → Model streaming: routes stream
agent/worker/worker.ts - Model streaming → Prompt assembly: builds messages
agent/worker/do/AgentService.ts - Model streaming → Tutor instructions: builds instructions
agent/worker/do/AgentService.ts - Model streaming → Model catalog: selects model
agent/worker/do/AgentService.ts - Model streaming → LLM providers: streams completion
- Model streaming → Tldraw agent: streams actions
- Tldraw agent → Action dispatch: dispatches actions
agent/client/agent/TldrawAgent.ts - Action dispatch → Canvas actions: applies action
agent/client/agent/managers/AgentActionManager.ts - Action dispatch → Live canvas: edits canvas
agent/client/agent/managers/AgentActionManager.ts - Tldraw agent → Canvas linting: checks canvas
agent/client/agent/TldrawAgent.ts - Tldraw agent → Chat history: updates history
- Agent app → App persistence: persists state
- Tldraw agent → Speech synthesis: sends response text
- Speech synthesis → Gemini TTS: synthesizes speech
- Speech synthesis → Voice playback: returns audio
- Voice playback → Student: speaks response