How jahnavigajjala-3/signspeak works
SignSpeak is a sign-language video translation product: users provide a recording or camera capture, the web interface presents a translation, and the backend runs a PyTorch inference pipeline. The repository also includes How2Sign dataset preparation and model-training code. The backend and ML edges below are grounded in sampled calls or SOURCE INDEX references. Frontend wiring and several listed ML modules were not excerpted, so uncertain links are omitted or marked unverified.
Main entry points: train.py Training routines · predict.py Prediction and decoding · translation.py Translation endpoint · inference_service.py Inference service
Components 25
Web experience
- Camera workspace
frontend/src/components/CameraWorkspace.tsx - Video upload
frontend/src/components/VideoUploader.tsx - Video playback
frontend/src/components/VideoPlayer.tsx - Mode selector
frontend/src/components/ModeSelector.tsx - Processing state
frontend/src/components/ProcessingState.tsx - Translation result
frontend/src/components/TranslationResult.tsx - Speech player
frontend/src/components/SpeechPlayer.tsx - API client
frontend/src/lib/api.ts
API service
- FastAPI app
backend/app/main.py - Translation endpoint
backend/app/api/translation.py - Health endpoint
backend/app/api/health.py - Response schemas
backend/app/schemas/translation.py
Inference pipeline
- Inference service
backend/app/services/inference_service.py - Prediction and decoding
ml/inference/predict.py
Model and data
- Frame sampling
ml/data/preprocessing/frame_sampler.py - Video frame loader
ml/data/loaders/video_loader.py - How2Sign dataset
ml/data/datasets/how2sign.py - Text vocabulary
ml/data/datasets/vocabulary.py - Batch collation
ml/data/datasets/collate.py - SignSpeak model
ml/models/signspeak.py - Visual encoder
ml/models/visual_encoder/cnn.py - Temporal encoder
ml/models/temporal_encoder/lstm.py
Model training
- Checkpoint handling
ml/training/checkpoint.py - Training routines
ml/training/train.py
Other components
- User
Connections 24
- User → Camera workspace: uses
- API client → Translation endpoint: submits video
- FastAPI app → Translation endpoint: registers
backend/app/main.py - FastAPI app → Health endpoint: registers
backend/app/main.py - Translation endpoint → Inference service: runs inference
backend/app/api/translation.py - Translation endpoint → Response schemas: returns
backend/app/api/translation.py - Health endpoint → Inference service: checks status
backend/app/api/health.py - Health endpoint → Response schemas: uses schema
backend/app/api/health.py - Inference service → Prediction and decoding: predicts video
backend/app/services/inference_service.py - Inference service → Frame sampling: samples indices
backend/app/services/inference_service.py - Prediction and decoding → Frame sampling: samples frames
ml/inference/predict.py - Prediction and decoding → SignSpeak model: runs model
ml/inference/predict.py - Prediction and decoding → Text vocabulary: decodes tokens
ml/inference/predict.py - Prediction and decoding → Checkpoint handling: loads weights
ml/inference/predict.py - Frame sampling → Video frame loader: reads frames
ml/data/preprocessing/frame_sampler.py - How2Sign dataset → Frame sampling: samples video
ml/data/datasets/how2sign.py - Batch collation → Text vocabulary: encodes text
ml/data/datasets/collate.py - SignSpeak model → Visual encoder: extracts features
ml/models/signspeak.py - SignSpeak model → Temporal encoder: encodes sequence
ml/models/signspeak.py - Training routines → How2Sign dataset: loads samples
ml/training/train.py - Training routines → Text vocabulary: builds vocabulary
ml/training/train.py - Training routines → Batch collation: batches data
ml/training/train.py - Training routines → SignSpeak model: trains model
ml/training/train.py - Training routines → Checkpoint handling: saves weights
ml/training/train.py