# vishwaksen21/signaction architecture

> Architecture diagram of the GitHub repository vishwaksen21/signaction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vishwaksen21/signaction
- Repository: https://github.com/vishwaksen21/signaction

## Overview

SignAction turns typed text or recorded speech into a gloss and a visual sign-gesture sequence. In the web flow, the Next.js interface submits text or audio through its translation routes to the FastAPI service; text translation also has an offline frontend fallback. The backend’s documented translation endpoints use the core pipeline, whose stages normalize text, map tokens to sign assets (with placeholder fallback), and render a GIF. Offline speech recognition and model setup are separate capabilities. README and sampled files support the core stages; endpoint internals and some frontend wiring were not sampled, so those links are conservative.

Main entry points: [`signaction/pipeline.py`](https://github.com/vishwaksen21/signaction/blob/main/signaction/pipeline.py) (Sign pipeline), [`backend/routes/speech.py`](https://github.com/vishwaksen21/signaction/blob/main/backend/routes/speech.py) (Speech endpoint), [`backend/main.py`](https://github.com/vishwaksen21/signaction/blob/main/backend/main.py) (FastAPI app), [`frontend/app/translate-text/route.ts`](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translate-text/route.ts) (Text route)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web experience"]
  node_webui["Translator UI<br/>[page.tsx]"]
  node_speechui["Speech UI"]
  node_textproxy["Text route<br/>[route.ts]"]
  node_speechproxy["Speech route<br/>[route.ts]"]
  node_offline["Offline translator"]
  node_viewer["Gesture viewer"]
  node_dictionary["Dictionary API<br/>[route.ts]"]
  node_modelsetup["Offline model setup<br/>[page.tsx]"]
end

subgraph group_backend["API service"]
  node_apiapp["FastAPI app<br/>[main.py]"]
  node_textapi["Text endpoint<br/>[translator.py]"]
  node_speechapi["Speech endpoint<br/>[speech.py]"]
  node_dictapi["Dictionary endpoint<br/>[dictionary.py]"]
end

subgraph group_core["Translation core"]
  node_pipeline["Sign pipeline<br/>[pipeline.py]"]
  node_gloss["Gloss preprocessing<br/>[nlp.py]"]
  node_mapping["Token-to-sign mapping<br/>[translate.py]"]
  node_render["Sequence rendering<br/>[render.py]"]
  node_assets["Sign assets"]
end

subgraph group_speech["Speech recognition"]
  node_sttfactory["STT selection<br/>[factory.py]"]
  node_vosk["Vosk backend<br/>[vosk_backend.py]"]
  node_whisper["Whisper backend<br/>[whisper_backend.py]"]
end

node_user(("User"))
node_modelhost["Vosk model host"]

node_user -->|"enters text"| node_webui
node_user -->|"provides audio"| node_speechui
node_webui -->|"submits text"| node_textproxy
node_speechui -->|"submits audio"| node_speechproxy
node_textproxy -->|"forwards request"| node_textapi
node_textproxy -.->|"falls back"| node_offline
node_speechproxy -->|"forwards audio"| node_speechapi
node_apiapp -->|"registers route"| node_textapi
node_apiapp -->|"registers route"| node_speechapi
node_apiapp -->|"registers route"| node_dictapi
node_textapi -->|"runs translation"| node_pipeline
node_speechapi -->|"transcribes audio"| node_sttfactory
node_speechapi -->|"translates transcript"| node_pipeline
node_pipeline -->|"normalizes text"| node_gloss
node_pipeline -->|"maps tokens"| node_mapping
node_pipeline -->|"renders signs"| node_render
node_mapping -->|"reads sign media"| node_assets
node_render -->|"reads media"| node_assets
node_pipeline -->|"returns GIF output"| node_viewer
node_sttfactory -->|"selects backend"| node_vosk
node_sttfactory -->|"selects backend"| node_whisper
node_modelsetup -->|"downloads model"| node_modelhost
node_modelsetup -.->|"configures local model"| node_vosk
node_dictionary -.->|"serves dictionary"| node_dictapi

click node_webui "https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translator/page.tsx"
click node_speechui "https://github.com/vishwaksen21/signaction/blob/main/frontend/components/speech-recorder.tsx"
click node_textproxy "https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translate-text/route.ts"
click node_speechproxy "https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translate-speech/route.ts"
click node_offline "https://github.com/vishwaksen21/signaction/blob/main/frontend/lib/offline-translate.ts"
click node_viewer "https://github.com/vishwaksen21/signaction/blob/main/frontend/components/gesture-sequence-player.tsx"
click node_dictionary "https://github.com/vishwaksen21/signaction/blob/main/frontend/app/api/dictionary/route.ts"
click node_modelsetup "https://github.com/vishwaksen21/signaction/blob/main/frontend/app/offline-setup/page.tsx"
click node_apiapp "https://github.com/vishwaksen21/signaction/blob/main/backend/main.py"
click node_textapi "https://github.com/vishwaksen21/signaction/blob/main/backend/routes/translator.py"
click node_speechapi "https://github.com/vishwaksen21/signaction/blob/main/backend/routes/speech.py"
click node_dictapi "https://github.com/vishwaksen21/signaction/blob/main/backend/routes/dictionary.py"
click node_pipeline "https://github.com/vishwaksen21/signaction/blob/main/signaction/pipeline.py"
click node_gloss "https://github.com/vishwaksen21/signaction/blob/main/signaction/nlp.py"
click node_mapping "https://github.com/vishwaksen21/signaction/blob/main/signaction/translate.py"
click node_render "https://github.com/vishwaksen21/signaction/blob/main/signaction/render.py"
click node_sttfactory "https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/factory.py"
click node_vosk "https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/vosk_backend.py"
click node_whisper "https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/whisper_backend.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webui,node_speechui,node_textproxy,node_speechproxy,node_offline,node_viewer,node_dictionary,node_modelsetup,node_user toneBlue
class node_apiapp,node_textapi,node_speechapi,node_dictapi toneAmber
class node_pipeline,node_gloss,node_mapping,node_render,node_assets toneMint
class node_sttfactory,node_vosk,node_whisper toneRose
class node_modelhost toneIndigo
```

## Components

### Web experience

- [Translator UI](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translator/page.tsx): `frontend/app/translator/page.tsx`
- [Speech UI](https://github.com/vishwaksen21/signaction/blob/main/frontend/components/speech-recorder.tsx): `frontend/components/speech-recorder.tsx`
- [Text route](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translate-text/route.ts): `frontend/app/translate-text/route.ts`
- [Speech route](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/translate-speech/route.ts): `frontend/app/translate-speech/route.ts`
- [Offline translator](https://github.com/vishwaksen21/signaction/blob/main/frontend/lib/offline-translate.ts): `frontend/lib/offline-translate.ts`
- [Gesture viewer](https://github.com/vishwaksen21/signaction/blob/main/frontend/components/gesture-sequence-player.tsx): `frontend/components/gesture-sequence-player.tsx`
- [Dictionary API](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/api/dictionary/route.ts): `frontend/app/api/dictionary/route.ts`
- [Offline model setup](https://github.com/vishwaksen21/signaction/blob/main/frontend/app/offline-setup/page.tsx): `frontend/app/offline-setup/page.tsx`

### API service

- [FastAPI app](https://github.com/vishwaksen21/signaction/blob/main/backend/main.py): `backend/main.py`
- [Text endpoint](https://github.com/vishwaksen21/signaction/blob/main/backend/routes/translator.py): `backend/routes/translator.py`
- [Speech endpoint](https://github.com/vishwaksen21/signaction/blob/main/backend/routes/speech.py): `backend/routes/speech.py`
- [Dictionary endpoint](https://github.com/vishwaksen21/signaction/blob/main/backend/routes/dictionary.py): `backend/routes/dictionary.py`

### Translation core

- [Sign pipeline](https://github.com/vishwaksen21/signaction/blob/main/signaction/pipeline.py): `signaction/pipeline.py`
- [Gloss preprocessing](https://github.com/vishwaksen21/signaction/blob/main/signaction/nlp.py): `signaction/nlp.py`
- [Token-to-sign mapping](https://github.com/vishwaksen21/signaction/blob/main/signaction/translate.py): `signaction/translate.py`
- [Sequence rendering](https://github.com/vishwaksen21/signaction/blob/main/signaction/render.py): `signaction/render.py`
- **Sign assets**

### Speech recognition

- [STT selection](https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/factory.py): `signaction/stt/factory.py`
- [Vosk backend](https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/vosk_backend.py): `signaction/stt/vosk_backend.py`
- [Whisper backend](https://github.com/vishwaksen21/signaction/blob/main/signaction/stt/whisper_backend.py): `signaction/stt/whisper_backend.py`

### Other components

- **User**
- **Vosk model host**

## Connections

- User → Translator UI: enters text
- User → Speech UI: provides audio
- Translator UI → Text route: submits text
- Speech UI → Speech route: submits audio
- Text route → Text endpoint: forwards request
- Text route → Offline translator: falls back
- Speech route → Speech endpoint: forwards audio
- FastAPI app → Text endpoint: registers route
- FastAPI app → Speech endpoint: registers route
- FastAPI app → Dictionary endpoint: registers route
- Text endpoint → Sign pipeline: runs translation
- Speech endpoint → STT selection: transcribes audio
- Speech endpoint → Sign pipeline: translates transcript
- Sign pipeline → Gloss preprocessing: normalizes text
- Sign pipeline → Token-to-sign mapping: maps tokens
- Sign pipeline → Sequence rendering: renders signs
- Token-to-sign mapping → Sign assets: reads sign media
- Sequence rendering → Sign assets: reads media
- Sign pipeline → Gesture viewer: returns GIF output
- STT selection → Vosk backend: selects backend
- STT selection → Whisper backend: selects backend
- Offline model setup → Vosk model host: downloads model
- Offline model setup → Vosk backend: configures local model
- Dictionary API → Dictionary endpoint: serves dictionary