How devang47/call-assist works
1. What this repo is
- A Node.js/TypeScript backend for an AI phone assistant for dental offices.
- The core product boundary is “incoming phone call handling”: the system receives Twilio audio, converses in real time, and optionally books appointments.
2. Runtime and server entrypoint
- src/index.ts boots the service.
- src/server/app.ts assembles the Fastify application, and src/server/routes/* exposes the public HTTP/WebSocket surface.
- The app is therefore centered around a single backend process that handles both webhook traffic and streaming media.
3. Telephony ingress and call lifecycle
- Twilio is the primary ingress/egress boundary.
- src/server/routes/twilio.ts handles the inbound call webhook and the Media Streams WebSocket used to receive and send audio.
- src/middleware/twilio-signature.ts indicates Twilio request verification is enforced at the edge.
- src/lib/call-debug-log.ts and src/db/queries/call-logs.ts suggest each call is logged for observability and audit.
4. Real-time voice pipeline
- The conversation engine lives in src/voice/audio-proxy.ts, which orchestrates the per-call pipeline.
- Audio flow is: Twilio PCM stream → local VAD → STT → LLM → TTS → Twilio.
- src/pipeline/types.ts defines the shared state contracts between stages, so the pipeline is modular rather than a monolith.
5. Speech boundary detection and local model usage
- src/pipeline/vad.ts uses a local Silero VAD model (models/silero_vad.onnx) to detect when the caller starts/stops speaking.
- This local inference stage is important architecturally because it gates when audio is buffered and sent for transcription.
- The presence of src/types/sherpa-onnx-node.d.ts shows the VAD stack is integrated through native/ONNX bindings.
6. Transcription, language model, and spoken response generation
- src/pipeline/stt.ts handles speech-to-text, backed by Groq Whisper per the README.
- src/pipeline/llm.ts streams model output from Groq and likely handles tool calls for scheduling/practice Q&A.
- src/pipeline/sentence-buffer.ts is a latency/UX boundary: it groups streamed tokens into complete sentences so TTS can start speaking before the full completion is done.
7. Text-to-speech and audio caching
- src/pipeline/tts.ts converts generated text into audio, using Deepgram per the README.
- src/pipeline/tts-cache.ts suggests repeated phrases or common utterances are cached to reduce response latency.
- This stage is where text is converted back into Twilio-compatible audio frames for playback to the caller.
8. Conversation policy and domain prompt
- src/voice/prompts.ts contains the assistant persona and likely the dental-office operating rules.
- This is the main domain boundary for behavior: intake questions, scheduling intent, practice info, escalation, and conversational style.
9. Scheduling subsystem
- src/scheduling/google-calendar.ts encapsulates Google OAuth and Calendar API access.
- src/scheduling/availability.ts computes open slots from free/busy data.
- src/scheduling/booking.ts creates or updates calendar events.
- Architecturally, this is the external “action” subsystem that the LLM can invoke when the caller wants an appointment.
10. Persistence and practice-level data
- The repo uses Drizzle (drizzle.config.ts, drizzle/*) and a schema in src/db/schema.ts.
- Query modules for patients, transcripts, and call-logs show the database stores conversational history and patient/practice-related records.
- src/db/practice-context.ts indicates multi-practice tenancy or practice-specific configuration, which is a key boundary in both scheduling and prompt behavior.
11. Supporting services
- src/services/sms.ts suggests the system can send SMS follow-ups or confirmations outside the live call path.
- src/lib/time.ts is a utility layer for date/time handling, which matters because scheduling is calendar-driven and time zone sensitive.
12. Operational and developer tooling
- scripts/* are mostly benchmarking and diagnostic entrypoints for STT, LLM, TTS, and full end-to-end latency.
- scripts/seed.ts implies the database can be populated with practice/demo data.
- This repo is built around latency measurement as an architectural concern, not just functionality.
13. Key external dependencies and boundaries
- Twilio: telephony and audio streaming.
- Groq: STT and LLM inference.
- Deepgram: TTS.
- Google Calendar: availability lookup and booking.
- Cloudflare Tunnel/ngrok: local development exposure so Twilio can reach the dev server.
- These services define the main failure domains and latency profile of the system.
14. How to draw the architecture
- Put Twilio at the edge feeding a Fastify server.
- Split the server into two paths: webhooks/auth routes and the real-time audio stream handler.
- Show the audio-proxy orchestrating VAD → STT → LLM → sentence buffering → TTS.
- Attach Google Calendar and the database as side subsystems used for scheduling, state, and logs.
- Show optional SMS as a secondary outbound channel.
Main entry points: audio-proxy.ts Audio proxy · llm.ts LLM · twilio.ts Twilio route · app.ts App
Components 24
Telephony edge
Inbound call handling and request verification
- App · Assembles the HTTP/WebSocket server
src/server/app.ts - Twilio route · Handles inbound call webhooks and media stream WebSocket
src/server/routes/twilio.ts - Twilio auth · Verifies Twilio request signatures
src/middleware/twilio-signature.ts
Voice pipeline
Real-time audio conversation stages
- Audio proxy · Coordinates the per-call real-time voice pipeline
src/voice/audio-proxy.ts - Prompts · Defines assistant persona and dental-office behavior
src/voice/prompts.ts - VAD · Detects speaker activity using local Silero ONNX model
src/pipeline/vad.ts - STT · Transcribes caller audio using Groq-backed speech recognition
src/pipeline/stt.ts - LLM · Streams assistant responses and scheduling/tool intent
src/pipeline/llm.ts - Sentence buffer · Groups token streams into speakable sentence chunks
src/pipeline/sentence-buffer.ts - TTS · Synthesizes audio from generated responses
src/pipeline/tts.ts - TTS cache · Caches repeated spoken phrases to reduce latency
src/pipeline/tts-cache.ts - Prefetch · Warms or prepares pipeline resources ahead of use
src/pipeline/prefetch.ts
Scheduling
Calendar lookup and booking actions
- Calendar route · Exposes scheduling-related HTTP surface
src/server/routes/calendar.ts - Calendar API · Google Calendar integration for availability and booking
src/scheduling/google-calendar.ts - Availability · Derives open appointment slots from free/busy data
src/scheduling/availability.ts - Booking · Creates or updates calendar events for appointments
src/scheduling/booking.ts
Data
Persistence and practice context
- Database · Primary persistence for call, transcript, patient, and practice data
src/db/index.ts - Practice context · Loads practice-specific configuration and tenancy context
src/db/practice-context.ts - Call logs · Persists call-level observability and audit records
src/db/queries/call-logs.ts - Transcripts · Stores conversation transcripts and related records
src/db/queries/transcripts.ts - Patients · Persists patient-facing or practice-specific data
src/db/queries/patients.ts - Call debug · Structured call tracing and debugging support
src/lib/call-debug-log.ts
Connections 24
- Boot → App: boots · Initializes the Fastify server
- App → Twilio route: mounts · Registers the Twilio webhook and media stream endpoints
- App → Calendar route: mounts · Registers scheduling routes
- Twilio route → Twilio auth: verifies · Protects inbound Twilio requests at the edge
- Twilio route → Audio proxy: starts stream · Hands live audio off to the conversation engine
- Audio proxy → Prefetch: primes · Prepares pipeline resources for low latency
- Audio proxy → VAD: feeds audio · Sends incoming PCM to speech boundary detection
- VAD → STT: gates · Buffers speech segments for transcription
- STT → LLM: transcripts · Passes recognized caller text into the model
- LLM → Sentence buffer: streams text · Chunks token output into speakable sentences
- Sentence buffer → TTS: queues speech · Converts response chunks into audio generation requests
- TTS → TTS cache: reuses audio · Serves cached phrases when available
- TTS → Twilio route: returns audio · Sends synthesized audio back through the media stream
- LLM → Prompts: guided by · Behavior is shaped by the dental-office prompt policy
- LLM → Calendar API: tool use · Invokes calendar access for scheduling intent
- Calendar API → Availability: reads free/busy · Computes open appointment slots
- Availability → Booking: selects slot · Uses slot computation to create bookings
- Booking → SMS: confirms · May send appointment confirmation or follow-up text
- Audio proxy → Database: persists · Writes conversation and call state
- Audio proxy → Call logs: logs · Emits call diagnostics and audit records
- Audio proxy → Transcripts: stores · Persists live transcript data
- Audio proxy → Patients: looks up · Reads or updates patient-related records
- Audio proxy → Practice context: scopes · Loads practice-specific configuration for the call
- Call debug → Call logs: feeds · Structured debugging output complements call logging