# devang47/call-assist architecture

> Architecture diagram of the GitHub repository devang47/call-assist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devang47/call-assist
- Repository: https://github.com/devang47/call-assist

## Overview

1. **What this repo is**
   - A Node.js/TypeScript backend for an AI phone assistant for dental offices.
   - The core product boundary is “incoming phone call handling”: the system receives Twilio audio, converses in real time, and optionally books appointments.

2. **Runtime and server entrypoint**
   - `src/index.ts` boots the service.
   - `src/server/app.ts` assembles the Fastify application, and `src/server/routes/*` exposes the public HTTP/WebSocket surface.
   - The app is therefore centered around a single backend process that handles both webhook traffic and streaming media.

3. **Telephony ingress and call lifecycle**
   - Twilio is the primary ingress/egress boundary.
   - `src/server/routes/twilio.ts` handles the inbound call webhook and the Media Streams WebSocket used to receive and send audio.
   - `src/middleware/twilio-signature.ts` indicates Twilio request verification is enforced at the edge.
   - `src/lib/call-debug-log.ts` and `src/db/queries/call-logs.ts` suggest each call is logged for observability and audit.

4. **Real-time voice pipeline**
   - The conversation engine lives in `src/voice/audio-proxy.ts`, which orchestrates the per-call pipeline.
   - Audio flow is: Twilio PCM stream → local VAD → STT → LLM → TTS → Twilio.
   - `src/pipeline/types.ts` defines the shared state contracts between stages, so the pipeline is modular rather than a monolith.

5. **Speech boundary detection and local model usage**
   - `src/pipeline/vad.ts` uses a local Silero VAD model (`models/silero_vad.onnx`) to detect when the caller starts/stops speaking.
   - This local inference stage is important architecturally because it gates when audio is buffered and sent for transcription.
   - The presence of `src/types/sherpa-onnx-node.d.ts` shows the VAD stack is integrated through native/ONNX bindings.

6. **Transcription, language model, and spoken response generation**
   - `src/pipeline/stt.ts` handles speech-to-text, backed by Groq Whisper per the README.
   - `src/pipeline/llm.ts` streams model output from Groq and likely handles tool calls for scheduling/practice Q&A.
   - `src/pipeline/sentence-buffer.ts` is a latency/UX boundary: it groups streamed tokens into complete sentences so TTS can start speaking before the full completion is done.

7. **Text-to-speech and audio caching**
   - `src/pipeline/tts.ts` converts generated text into audio, using Deepgram per the README.
   - `src/pipeline/tts-cache.ts` suggests repeated phrases or common utterances are cached to reduce response latency.
   - This stage is where text is converted back into Twilio-compatible audio frames for playback to the caller.

8. **Conversation policy and domain prompt**
   - `src/voice/prompts.ts` contains the assistant persona and likely the dental-office operating rules.
   - This is the main domain boundary for behavior: intake questions, scheduling intent, practice info, escalation, and conversational style.

9. **Scheduling subsystem**
   - `src/scheduling/google-calendar.ts` encapsulates Google OAuth and Calendar API access.
   - `src/scheduling/availability.ts` computes open slots from free/busy data.
   - `src/scheduling/booking.ts` creates or updates calendar events.
   - Architecturally, this is the external “action” subsystem that the LLM can invoke when the caller wants an appointment.

10. **Persistence and practice-level data**
   - The repo uses Drizzle (`drizzle.config.ts`, `drizzle/*`) and a schema in `src/db/schema.ts`.
   - Query modules for `patients`, `transcripts`, and `call-logs` show the database stores conversational history and patient/practice-related records.
   - `src/db/practice-context.ts` indicates multi-practice tenancy or practice-specific configuration, which is a key boundary in both scheduling and prompt behavior.

11. **Supporting services**
   - `src/services/sms.ts` suggests the system can send SMS follow-ups or confirmations outside the live call path.
   - `src/lib/time.ts` is a utility layer for date/time handling, which matters because scheduling is calendar-driven and time zone sensitive.

12. **Operational and developer tooling**
   - `scripts/*` are mostly benchmarking and diagnostic entrypoints for STT, LLM, TTS, and full end-to-end latency.
   - `scripts/seed.ts` implies the database can be populated with practice/demo data.
   - This repo is built around latency measurement as an architectural concern, not just functionality.

13. **Key external dependencies and boundaries**
   - Twilio: telephony and audio streaming.
   - Groq: STT and LLM inference.
   - Deepgram: TTS.
   - Google Calendar: availability lookup and booking.
   - Cloudflare Tunnel/ngrok: local development exposure so Twilio can reach the dev server.
   - These services define the main failure domains and latency profile of the system.

14. **How to draw the architecture**
   - Put Twilio at the edge feeding a Fastify server.
   - Split the server into two paths: webhooks/auth routes and the real-time audio stream handler.
   - Show the audio-proxy orchestrating VAD → STT → LLM → sentence buffering → TTS.
   - Attach Google Calendar and the database as side subsystems used for scheduling, state, and logs.
   - Show optional SMS as a secondary outbound channel.

Main entry points: [`src/voice/audio-proxy.ts`](https://github.com/devang47/call-assist/blob/main/src/voice/audio-proxy.ts) (Audio proxy), [`src/pipeline/llm.ts`](https://github.com/devang47/call-assist/blob/main/src/pipeline/llm.ts) (LLM), [`src/server/routes/twilio.ts`](https://github.com/devang47/call-assist/blob/main/src/server/routes/twilio.ts) (Twilio route), [`src/server/app.ts`](https://github.com/devang47/call-assist/blob/main/src/server/app.ts) (App)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_edge["Telephony edge"]
  node_app["App<br/>fastify setup<br/>[app.ts]"]
  node_twilio_route["Twilio route<br/>webhook/ws<br/>[twilio.ts]"]
  node_sigcheck["Twilio auth<br/>middleware"]
end

subgraph group_voice["Voice pipeline"]
  node_audio_proxy{{"Audio proxy<br/>orchestrator<br/>[audio-proxy.ts]"}}
  node_prompts["Prompts<br/>policy/prompt<br/>[prompts.ts]"]
  node_vad["VAD<br/>speech boundary<br/>[vad.ts]"]
  node_stt["STT<br/>speech-to-text<br/>[stt.ts]"]
  node_llm["LLM<br/>generation<br/>[llm.ts]"]
  node_sentence_buffer["Sentence buffer<br/>streaming buffer<br/>[sentence-buffer.ts]"]
  node_tts["TTS<br/>text-to-speech<br/>[tts.ts]"]
  node_tts_cache["TTS cache<br/>audio cache<br/>[tts-cache.ts]"]
  node_prefetch["Prefetch<br/>latency optimization<br/>[prefetch.ts]"]
end

subgraph group_scheduling["Scheduling"]
  node_calendar_route["Calendar route<br/>http route<br/>[calendar.ts]"]
  node_calendar["Calendar API<br/>external service<br/>[google-calendar.ts]"]
  node_availability["Availability<br/>slot computation<br/>[availability.ts]"]
  node_booking["Booking<br/>calendar write<br/>[booking.ts]"]
end

subgraph group_data["Data"]
  node_db[("Database<br/>drizzle db<br/>[index.ts]")]
  node_practice_context["Practice context<br/>tenant config"]
  node_call_logs["Call logs<br/>query module<br/>[call-logs.ts]"]
  node_transcripts["Transcripts<br/>query module<br/>[transcripts.ts]"]
  node_patients["Patients<br/>query module<br/>[patients.ts]"]
  node_call_debug["Call debug<br/>diagnostic log<br/>[call-debug-log.ts]"]
end

node_entry(("Boot<br/>runtime entry<br/>[index.ts]"))
node_sms["SMS<br/>outbound messaging<br/>[sms.ts]"]

node_entry -->|"boots"| node_app
node_app -->|"mounts"| node_twilio_route
node_app -->|"mounts"| node_calendar_route
node_twilio_route -->|"verifies"| node_sigcheck
node_twilio_route -->|"starts stream"| node_audio_proxy
node_audio_proxy -->|"primes"| node_prefetch
node_audio_proxy -->|"feeds audio"| node_vad
node_vad -->|"gates"| node_stt
node_stt -->|"transcripts"| node_llm
node_llm -->|"streams text"| node_sentence_buffer
node_sentence_buffer -->|"queues speech"| node_tts
node_tts -->|"reuses audio"| node_tts_cache
node_tts -->|"returns audio"| node_twilio_route
node_llm -->|"guided by"| node_prompts
node_llm -->|"tool use"| node_calendar
node_calendar -->|"reads free/busy"| node_availability
node_availability -->|"selects slot"| node_booking
node_booking -->|"confirms"| node_sms
node_audio_proxy -->|"persists"| node_db
node_audio_proxy -->|"logs"| node_call_logs
node_audio_proxy -->|"stores"| node_transcripts
node_audio_proxy -->|"looks up"| node_patients
node_audio_proxy -->|"scopes"| node_practice_context
node_call_debug -->|"feeds"| node_call_logs

click node_entry "https://github.com/devang47/call-assist/blob/main/src/index.ts"
click node_app "https://github.com/devang47/call-assist/blob/main/src/server/app.ts"
click node_twilio_route "https://github.com/devang47/call-assist/blob/main/src/server/routes/twilio.ts"
click node_calendar_route "https://github.com/devang47/call-assist/blob/main/src/server/routes/calendar.ts"
click node_sigcheck "https://github.com/devang47/call-assist/blob/main/src/middleware/twilio-signature.ts"
click node_audio_proxy "https://github.com/devang47/call-assist/blob/main/src/voice/audio-proxy.ts"
click node_prompts "https://github.com/devang47/call-assist/blob/main/src/voice/prompts.ts"
click node_vad "https://github.com/devang47/call-assist/blob/main/src/pipeline/vad.ts"
click node_stt "https://github.com/devang47/call-assist/blob/main/src/pipeline/stt.ts"
click node_llm "https://github.com/devang47/call-assist/blob/main/src/pipeline/llm.ts"
click node_sentence_buffer "https://github.com/devang47/call-assist/blob/main/src/pipeline/sentence-buffer.ts"
click node_tts "https://github.com/devang47/call-assist/blob/main/src/pipeline/tts.ts"
click node_tts_cache "https://github.com/devang47/call-assist/blob/main/src/pipeline/tts-cache.ts"
click node_prefetch "https://github.com/devang47/call-assist/blob/main/src/pipeline/prefetch.ts"
click node_calendar "https://github.com/devang47/call-assist/blob/main/src/scheduling/google-calendar.ts"
click node_availability "https://github.com/devang47/call-assist/blob/main/src/scheduling/availability.ts"
click node_booking "https://github.com/devang47/call-assist/blob/main/src/scheduling/booking.ts"
click node_db "https://github.com/devang47/call-assist/blob/main/src/db/index.ts"
click node_practice_context "https://github.com/devang47/call-assist/blob/main/src/db/practice-context.ts"
click node_call_logs "https://github.com/devang47/call-assist/blob/main/src/db/queries/call-logs.ts"
click node_transcripts "https://github.com/devang47/call-assist/blob/main/src/db/queries/transcripts.ts"
click node_patients "https://github.com/devang47/call-assist/blob/main/src/db/queries/patients.ts"
click node_call_debug "https://github.com/devang47/call-assist/blob/main/src/lib/call-debug-log.ts"
click node_sms "https://github.com/devang47/call-assist/blob/main/src/services/sms.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_twilio_route,node_sigcheck toneBlue
class node_audio_proxy,node_prompts,node_vad,node_stt,node_llm,node_sentence_buffer,node_tts,node_tts_cache,node_prefetch toneAmber
class node_calendar_route,node_calendar,node_availability,node_booking toneMint
class node_db,node_practice_context,node_call_logs,node_transcripts,node_patients,node_call_debug toneRose
class node_entry,node_sms toneNeutral
```

## Components

### Telephony edge

Inbound call handling and request verification

- [App](https://github.com/devang47/call-assist/blob/main/src/server/app.ts): fastify setup · Assembles the HTTP/WebSocket server · `src/server/app.ts`
- [Twilio route](https://github.com/devang47/call-assist/blob/main/src/server/routes/twilio.ts): webhook/ws · Handles inbound call webhooks and media stream WebSocket · `src/server/routes/twilio.ts`
- [Twilio auth](https://github.com/devang47/call-assist/blob/main/src/middleware/twilio-signature.ts): middleware · Verifies Twilio request signatures · `src/middleware/twilio-signature.ts`

### Voice pipeline

Real-time audio conversation stages

- [Audio proxy](https://github.com/devang47/call-assist/blob/main/src/voice/audio-proxy.ts): orchestrator · Coordinates the per-call real-time voice pipeline · `src/voice/audio-proxy.ts`
- [Prompts](https://github.com/devang47/call-assist/blob/main/src/voice/prompts.ts): policy/prompt · Defines assistant persona and dental-office behavior · `src/voice/prompts.ts`
- [VAD](https://github.com/devang47/call-assist/blob/main/src/pipeline/vad.ts): speech boundary · Detects speaker activity using local Silero ONNX model · `src/pipeline/vad.ts`
- [STT](https://github.com/devang47/call-assist/blob/main/src/pipeline/stt.ts): speech-to-text · Transcribes caller audio using Groq-backed speech recognition · `src/pipeline/stt.ts`
- [LLM](https://github.com/devang47/call-assist/blob/main/src/pipeline/llm.ts): generation · Streams assistant responses and scheduling/tool intent · `src/pipeline/llm.ts`
- [Sentence buffer](https://github.com/devang47/call-assist/blob/main/src/pipeline/sentence-buffer.ts): streaming buffer · Groups token streams into speakable sentence chunks · `src/pipeline/sentence-buffer.ts`
- [TTS](https://github.com/devang47/call-assist/blob/main/src/pipeline/tts.ts): text-to-speech · Synthesizes audio from generated responses · `src/pipeline/tts.ts`
- [TTS cache](https://github.com/devang47/call-assist/blob/main/src/pipeline/tts-cache.ts): audio cache · Caches repeated spoken phrases to reduce latency · `src/pipeline/tts-cache.ts`
- [Prefetch](https://github.com/devang47/call-assist/blob/main/src/pipeline/prefetch.ts): latency optimization · Warms or prepares pipeline resources ahead of use · `src/pipeline/prefetch.ts`

### Scheduling

Calendar lookup and booking actions

- [Calendar route](https://github.com/devang47/call-assist/blob/main/src/server/routes/calendar.ts): http route · Exposes scheduling-related HTTP surface · `src/server/routes/calendar.ts`
- [Calendar API](https://github.com/devang47/call-assist/blob/main/src/scheduling/google-calendar.ts): external service · Google Calendar integration for availability and booking · `src/scheduling/google-calendar.ts`
- [Availability](https://github.com/devang47/call-assist/blob/main/src/scheduling/availability.ts): slot computation · Derives open appointment slots from free/busy data · `src/scheduling/availability.ts`
- [Booking](https://github.com/devang47/call-assist/blob/main/src/scheduling/booking.ts): calendar write · Creates or updates calendar events for appointments · `src/scheduling/booking.ts`

### Data

Persistence and practice context

- [Database](https://github.com/devang47/call-assist/blob/main/src/db/index.ts): drizzle db · Primary persistence for call, transcript, patient, and practice data · `src/db/index.ts`
- [Practice context](https://github.com/devang47/call-assist/blob/main/src/db/practice-context.ts): tenant config · Loads practice-specific configuration and tenancy context · `src/db/practice-context.ts`
- [Call logs](https://github.com/devang47/call-assist/blob/main/src/db/queries/call-logs.ts): query module · Persists call-level observability and audit records · `src/db/queries/call-logs.ts`
- [Transcripts](https://github.com/devang47/call-assist/blob/main/src/db/queries/transcripts.ts): query module · Stores conversation transcripts and related records · `src/db/queries/transcripts.ts`
- [Patients](https://github.com/devang47/call-assist/blob/main/src/db/queries/patients.ts): query module · Persists patient-facing or practice-specific data · `src/db/queries/patients.ts`
- [Call debug](https://github.com/devang47/call-assist/blob/main/src/lib/call-debug-log.ts): diagnostic log · Structured call tracing and debugging support · `src/lib/call-debug-log.ts`

### Other components

- [Boot](https://github.com/devang47/call-assist/blob/main/src/index.ts): runtime entry · Starts the backend process · `src/index.ts`
- [SMS](https://github.com/devang47/call-assist/blob/main/src/services/sms.ts): outbound messaging · Sends confirmations or follow-up messages · `src/services/sms.ts`

## Connections

- Boot → App: boots · Initializes the Fastify server
- App → Twilio route: mounts · Registers the Twilio webhook and media stream endpoints
- App → Calendar route: mounts · Registers scheduling routes
- Twilio route → Twilio auth: verifies · Protects inbound Twilio requests at the edge
- Twilio route → Audio proxy: starts stream · Hands live audio off to the conversation engine
- Audio proxy → Prefetch: primes · Prepares pipeline resources for low latency
- Audio proxy → VAD: feeds audio · Sends incoming PCM to speech boundary detection
- VAD → STT: gates · Buffers speech segments for transcription
- STT → LLM: transcripts · Passes recognized caller text into the model
- LLM → Sentence buffer: streams text · Chunks token output into speakable sentences
- Sentence buffer → TTS: queues speech · Converts response chunks into audio generation requests
- TTS → TTS cache: reuses audio · Serves cached phrases when available
- TTS → Twilio route: returns audio · Sends synthesized audio back through the media stream
- LLM → Prompts: guided by · Behavior is shaped by the dental-office prompt policy
- LLM → Calendar API: tool use · Invokes calendar access for scheduling intent
- Calendar API → Availability: reads free/busy · Computes open appointment slots
- Availability → Booking: selects slot · Uses slot computation to create bookings
- Booking → SMS: confirms · May send appointment confirmation or follow-up text
- Audio proxy → Database: persists · Writes conversation and call state
- Audio proxy → Call logs: logs · Emits call diagnostics and audit records
- Audio proxy → Transcripts: stores · Persists live transcript data
- Audio proxy → Patients: looks up · Reads or updates patient-related records
- Audio proxy → Practice context: scopes · Loads practice-specific configuration for the call
- Call debug → Call logs: feeds · Structured debugging output complements call logging