How bach-dt/2026_ausculpatch works
The product described by the README is a voice-operated Universal Robot interface. Users submit text, browser-recognized speech, or AusculPatch Bluetooth audio; the backend obtains a command, passes it to the tool-using agent, and returns a response for display and optional speech. The principal path and the documented robot, camera, drawing, and calculation capabilities are represented below. Source excerpts establish the main calls; uninspected runtime internals and hardware behavior are not inferred.
Main entry points: ur_agent.py UR command agent · App.tsx Voice control UI · server.py HTTP routes · drawing_tool.py Image-to-drawing
Components 18
User interface
- Voice control UI
ur-voice-control-final/frontend/src/App.tsx - Browser speech input
ur-voice-control-final/frontend/src/App.tsx
Command backend
- HTTP routes
ur-voice-control-final/backend/server.py - Placeholder chatbot
ur-voice-control-final/backend/chatbot.py
Voice input
- Patch audio classifier
ur-voice-control-final/backend/voice_model/detectV2.py
Agent capabilities
- UR command agent
ur-voice-control-final/backend/ur_agent.py - Robot instructions
ur-voice-control-final/backend/prompts.py - Math tools
ur-voice-control-final/backend/tools/calculation.py
Robot and perception
- Robot control tools
ur-voice-control-final/backend/tools/ur_tools.py - Camera perception
ur-voice-control-final/backend/tools/camera_tool.py - Image-to-drawing
ur-voice-control-final/backend/tools/drawing_tool.py
Other components
- Operator
- Universal Robot arm
- AusculPatch
- OpenAI model
- Google image search
- Camera
- Browser speech output
Connections 21
- Operator → Voice control UI: enters command
ur-voice-control-final/frontend/src/App.tsx - Voice control UI → HTTP routes: posts text
ur-voice-control-final/frontend/src/App.tsx - Voice control UI → Browser speech input: captures speech
ur-voice-control-final/frontend/src/App.tsx - Browser speech input → HTTP routes: posts transcript
ur-voice-control-final/frontend/src/App.tsx - Voice control UI → HTTP routes: requests patch
ur-voice-control-final/frontend/src/App.tsx - AusculPatch → Patch audio classifier: sends audio
README.md - HTTP routes → Patch audio classifier: predicts command
ur-voice-control-final/backend/server.py - Patch audio classifier → AusculPatch: reads audio
ur-voice-control-final/backend/voice_model/detectV2.py - HTTP routes → UR command agent: invokes command
ur-voice-control-final/backend/server.py - UR command agent → Robot instructions: uses instructions
ur-voice-control-final/backend/ur_agent.py - UR command agent → Math tools: registers tools
ur-voice-control-final/backend/ur_agent.py - UR command agent → Robot control tools: registers tools
ur-voice-control-final/backend/ur_agent.py - UR command agent → Camera perception: registers tools
ur-voice-control-final/backend/ur_agent.py - UR command agent → Image-to-drawing: registers tools
ur-voice-control-final/backend/ur_agent.py - UR command agent → OpenAI model: queries model
ur-voice-control-final/backend/ur_agent.py - Robot control tools → Universal Robot arm: controls arm
ur-voice-control-final/backend/tools/ur_tools.py - Image-to-drawing → Universal Robot arm: executes drawing
ur-voice-control-final/backend/tools/drawing_tool.py - Camera perception → Camera: captures frames
ur-voice-control-final/backend/tools/camera_tool.py - Image-to-drawing → Google image search: searches images
ur-voice-control-final/backend/tools/drawing_tool.py - HTTP routes → Voice control UI: returns response
ur-voice-control-final/backend/server.py - Voice control UI → Browser speech output: speaks response
ur-voice-control-final/frontend/src/App.tsx