How vishwaksen21/signaction works
SignAction turns typed text or recorded speech into a gloss and a visual sign-gesture sequence. In the web flow, the Next.js interface submits text or audio through its translation routes to the FastAPI service; text translation also has an offline frontend fallback. The backend’s documented translation endpoints use the core pipeline, whose stages normalize text, map tokens to sign assets (with placeholder fallback), and render a GIF. Offline speech recognition and model setup are separate capabilities. README and sampled files support the core stages; endpoint internals and some frontend wiring were not sampled, so those links are conservative.
Main entry points: pipeline.py Sign pipeline · speech.py Speech endpoint · main.py FastAPI app · route.ts Text route
Components 22
Web experience
- Translator UI
frontend/app/translator/page.tsx - Speech UI
frontend/components/speech-recorder.tsx - Text route
frontend/app/translate-text/route.ts - Speech route
frontend/app/translate-speech/route.ts - Offline translator
frontend/lib/offline-translate.ts - Gesture viewer
frontend/components/gesture-sequence-player.tsx - Dictionary API
frontend/app/api/dictionary/route.ts - Offline model setup
frontend/app/offline-setup/page.tsx
API service
- FastAPI app
backend/main.py - Text endpoint
backend/routes/translator.py - Speech endpoint
backend/routes/speech.py - Dictionary endpoint
backend/routes/dictionary.py
Translation core
- Sign pipeline
signaction/pipeline.py - Gloss preprocessing
signaction/nlp.py - Token-to-sign mapping
signaction/translate.py - Sequence rendering
signaction/render.py - Sign assets
Speech recognition
- STT selection
signaction/stt/factory.py - Vosk backend
signaction/stt/vosk_backend.py - Whisper backend
signaction/stt/whisper_backend.py
Other components
- User
- Vosk model host
Connections 24
- User → Translator UI: enters text
- User → Speech UI: provides audio
- Translator UI → Text route: submits text
- Speech UI → Speech route: submits audio
- Text route → Text endpoint: forwards request
- Text route → Offline translator: falls back
- Speech route → Speech endpoint: forwards audio
- FastAPI app → Text endpoint: registers route
- FastAPI app → Speech endpoint: registers route
- FastAPI app → Dictionary endpoint: registers route
- Text endpoint → Sign pipeline: runs translation
- Speech endpoint → STT selection: transcribes audio
- Speech endpoint → Sign pipeline: translates transcript
- Sign pipeline → Gloss preprocessing: normalizes text
- Sign pipeline → Token-to-sign mapping: maps tokens
- Sign pipeline → Sequence rendering: renders signs
- Token-to-sign mapping → Sign assets: reads sign media
- Sequence rendering → Sign assets: reads media
- Sign pipeline → Gesture viewer: returns GIF output
- STT selection → Vosk backend: selects backend
- STT selection → Whisper backend: selects backend
- Offline model setup → Vosk model host: downloads model
- Offline model setup → Vosk backend: configures local model
- Dictionary API → Dictionary endpoint: serves dictionary