# jahnavigajjala-3/signspeak architecture

> Architecture diagram of the GitHub repository jahnavigajjala-3/signspeak, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jahnavigajjala-3/signspeak
- Repository: https://github.com/jahnavigajjala-3/signspeak

## Overview

SignSpeak is a sign-language video translation product: users provide a recording or camera capture, the web interface presents a translation, and the backend runs a PyTorch inference pipeline. The repository also includes How2Sign dataset preparation and model-training code. The backend and ML edges below are grounded in sampled calls or SOURCE INDEX references. Frontend wiring and several listed ML modules were not excerpted, so uncertain links are omitted or marked unverified.

Main entry points: [`ml/training/train.py`](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/training/train.py) (Training routines), [`ml/inference/predict.py`](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/inference/predict.py) (Prediction and decoding), [`backend/app/api/translation.py`](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/api/translation.py) (Translation endpoint), [`backend/app/services/inference_service.py`](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/services/inference_service.py) (Inference service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web experience"]
  node_workspace["Camera workspace"]
  node_video_upload["Video upload<br/>[VideoUploader.tsx]"]
  node_video_playback["Video playback<br/>[VideoPlayer.tsx]"]
  node_mode_select["Mode selector<br/>[ModeSelector.tsx]"]
  node_processing_ui["Processing state"]
  node_result_ui["Translation result"]
  node_speech["Speech player<br/>[SpeechPlayer.tsx]"]
  node_api_client["API client<br/>[api.ts]"]
end

subgraph group_api["API service"]
  node_api_app["FastAPI app<br/>[main.py]"]
  node_translation_route["Translation endpoint<br/>[translation.py]"]
  node_health_route["Health endpoint<br/>[health.py]"]
  node_response_schema["Response schemas<br/>[translation.py]"]
end

subgraph group_inference["Inference pipeline"]
  node_inference_service["Inference service"]
  node_predict["Prediction and decoding<br/>[predict.py]"]
end

subgraph group_model["Model and data"]
  node_frame_sampling["Frame sampling<br/>[frame_sampler.py]"]
  node_video_loader["Video frame loader<br/>[video_loader.py]"]
  node_sign_dataset["How2Sign dataset<br/>[how2sign.py]"]
  node_vocabulary["Text vocabulary<br/>[vocabulary.py]"]
  node_collation["Batch collation<br/>[collate.py]"]
  node_sign_model["SignSpeak model<br/>[signspeak.py]"]
  node_visual_encoder["Visual encoder<br/>[cnn.py]"]
  node_temporal_encoder["Temporal encoder<br/>[lstm.py]"]
end

subgraph group_training["Model training"]
  node_checkpoint["Checkpoint handling<br/>[checkpoint.py]"]
  node_training_loop["Training routines<br/>[train.py]"]
end

node_user(("User"))

node_user -.->|"uses"| node_workspace
node_api_client -.->|"submits video"| node_translation_route
node_api_app -->|"registers"| node_translation_route
node_api_app -->|"registers"| node_health_route
node_translation_route -->|"runs inference"| node_inference_service
node_translation_route -->|"returns"| node_response_schema
node_health_route -->|"checks status"| node_inference_service
node_health_route -->|"uses schema"| node_response_schema
node_inference_service -->|"predicts video"| node_predict
node_inference_service -->|"samples indices"| node_frame_sampling
node_predict -->|"samples frames"| node_frame_sampling
node_predict -->|"runs model"| node_sign_model
node_predict -->|"decodes tokens"| node_vocabulary
node_predict -->|"loads weights"| node_checkpoint
node_frame_sampling -->|"reads frames"| node_video_loader
node_sign_dataset -->|"samples video"| node_frame_sampling
node_collation -->|"encodes text"| node_vocabulary
node_sign_model -->|"extracts features"| node_visual_encoder
node_sign_model -->|"encodes sequence"| node_temporal_encoder
node_training_loop -->|"loads samples"| node_sign_dataset
node_training_loop -->|"builds vocabulary"| node_vocabulary
node_training_loop -->|"batches data"| node_collation
node_training_loop -->|"trains model"| node_sign_model
node_training_loop -->|"saves weights"| node_checkpoint

click node_workspace "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/CameraWorkspace.tsx"
click node_video_upload "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/VideoUploader.tsx"
click node_video_playback "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/VideoPlayer.tsx"
click node_mode_select "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/ModeSelector.tsx"
click node_processing_ui "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/ProcessingState.tsx"
click node_result_ui "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/TranslationResult.tsx"
click node_speech "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/SpeechPlayer.tsx"
click node_api_client "https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/lib/api.ts"
click node_api_app "https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/main.py"
click node_translation_route "https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/api/translation.py"
click node_health_route "https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/api/health.py"
click node_response_schema "https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/schemas/translation.py"
click node_inference_service "https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/services/inference_service.py"
click node_predict "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/inference/predict.py"
click node_frame_sampling "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/preprocessing/frame_sampler.py"
click node_video_loader "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/loaders/video_loader.py"
click node_sign_dataset "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/how2sign.py"
click node_vocabulary "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/vocabulary.py"
click node_collation "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/collate.py"
click node_sign_model "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/signspeak.py"
click node_visual_encoder "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/visual_encoder/cnn.py"
click node_temporal_encoder "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/temporal_encoder/lstm.py"
click node_checkpoint "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/training/checkpoint.py"
click node_training_loop "https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/training/train.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workspace,node_video_upload,node_video_playback,node_mode_select,node_processing_ui,node_result_ui,node_speech,node_api_client,node_user toneBlue
class node_api_app,node_translation_route,node_health_route,node_response_schema toneAmber
class node_inference_service,node_predict toneMint
class node_frame_sampling,node_video_loader,node_sign_dataset,node_vocabulary,node_collation,node_sign_model,node_visual_encoder,node_temporal_encoder toneRose
class node_checkpoint,node_training_loop toneIndigo
```

## Components

### Web experience

- [Camera workspace](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/CameraWorkspace.tsx): `frontend/src/components/CameraWorkspace.tsx`
- [Video upload](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/VideoUploader.tsx): `frontend/src/components/VideoUploader.tsx`
- [Video playback](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/VideoPlayer.tsx): `frontend/src/components/VideoPlayer.tsx`
- [Mode selector](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/ModeSelector.tsx): `frontend/src/components/ModeSelector.tsx`
- [Processing state](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/ProcessingState.tsx): `frontend/src/components/ProcessingState.tsx`
- [Translation result](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/TranslationResult.tsx): `frontend/src/components/TranslationResult.tsx`
- [Speech player](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/components/SpeechPlayer.tsx): `frontend/src/components/SpeechPlayer.tsx`
- [API client](https://github.com/jahnavigajjala-3/signspeak/blob/main/frontend/src/lib/api.ts): `frontend/src/lib/api.ts`

### API service

- [FastAPI app](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/main.py): `backend/app/main.py`
- [Translation endpoint](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/api/translation.py): `backend/app/api/translation.py`
- [Health endpoint](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/api/health.py): `backend/app/api/health.py`
- [Response schemas](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/schemas/translation.py): `backend/app/schemas/translation.py`

### Inference pipeline

- [Inference service](https://github.com/jahnavigajjala-3/signspeak/blob/main/backend/app/services/inference_service.py): `backend/app/services/inference_service.py`
- [Prediction and decoding](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/inference/predict.py): `ml/inference/predict.py`

### Model and data

- [Frame sampling](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/preprocessing/frame_sampler.py): `ml/data/preprocessing/frame_sampler.py`
- [Video frame loader](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/loaders/video_loader.py): `ml/data/loaders/video_loader.py`
- [How2Sign dataset](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/how2sign.py): `ml/data/datasets/how2sign.py`
- [Text vocabulary](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/vocabulary.py): `ml/data/datasets/vocabulary.py`
- [Batch collation](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/data/datasets/collate.py): `ml/data/datasets/collate.py`
- [SignSpeak model](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/signspeak.py): `ml/models/signspeak.py`
- [Visual encoder](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/visual_encoder/cnn.py): `ml/models/visual_encoder/cnn.py`
- [Temporal encoder](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/models/temporal_encoder/lstm.py): `ml/models/temporal_encoder/lstm.py`

### Model training

- [Checkpoint handling](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/training/checkpoint.py): `ml/training/checkpoint.py`
- [Training routines](https://github.com/jahnavigajjala-3/signspeak/blob/main/ml/training/train.py): `ml/training/train.py`

### Other components

- **User**

## Connections

- User → Camera workspace: uses
- API client → Translation endpoint: submits video
- FastAPI app → Translation endpoint: registers (evidence: `backend/app/main.py`)
- FastAPI app → Health endpoint: registers (evidence: `backend/app/main.py`)
- Translation endpoint → Inference service: runs inference (evidence: `backend/app/api/translation.py`)
- Translation endpoint → Response schemas: returns (evidence: `backend/app/api/translation.py`)
- Health endpoint → Inference service: checks status (evidence: `backend/app/api/health.py`)
- Health endpoint → Response schemas: uses schema (evidence: `backend/app/api/health.py`)
- Inference service → Prediction and decoding: predicts video (evidence: `backend/app/services/inference_service.py`)
- Inference service → Frame sampling: samples indices (evidence: `backend/app/services/inference_service.py`)
- Prediction and decoding → Frame sampling: samples frames (evidence: `ml/inference/predict.py`)
- Prediction and decoding → SignSpeak model: runs model (evidence: `ml/inference/predict.py`)
- Prediction and decoding → Text vocabulary: decodes tokens (evidence: `ml/inference/predict.py`)
- Prediction and decoding → Checkpoint handling: loads weights (evidence: `ml/inference/predict.py`)
- Frame sampling → Video frame loader: reads frames (evidence: `ml/data/preprocessing/frame_sampler.py`)
- How2Sign dataset → Frame sampling: samples video (evidence: `ml/data/datasets/how2sign.py`)
- Batch collation → Text vocabulary: encodes text (evidence: `ml/data/datasets/collate.py`)
- SignSpeak model → Visual encoder: extracts features (evidence: `ml/models/signspeak.py`)
- SignSpeak model → Temporal encoder: encodes sequence (evidence: `ml/models/signspeak.py`)
- Training routines → How2Sign dataset: loads samples (evidence: `ml/training/train.py`)
- Training routines → Text vocabulary: builds vocabulary (evidence: `ml/training/train.py`)
- Training routines → Batch collation: batches data (evidence: `ml/training/train.py`)
- Training routines → SignSpeak model: trains model (evidence: `ml/training/train.py`)
- Training routines → Checkpoint handling: saves weights (evidence: `ml/training/train.py`)