How neccam/slt works
SignJoey trains and evaluates a joint sign-language recognition and translation model. The user workflow starts with the module CLI, loads sign features, glosses and text, batches examples, then trains or predicts with the model; evaluation scores recognition and translation outputs. The model combines sign embeddings, an encoder, gloss recognition and text decoding, with search for generation. The SOURCE INDEX supports the listed module references; details of unsampled implementation paths and configuration wiring are not inferred.
Main entry points: model.py Joint SLT model · prediction.py Prediction and testing · training.py Training loop · data.py Data loader
Components 21
Data preparation
- Sign datasets
signjoey/dataset.py - Data loader
signjoey/data.py - Gloss and text vocabularies
signjoey/vocabulary.py - Batch preparation
signjoey/batch.py
Joint model
- Model builder
signjoey/builders.py - Joint SLT model
signjoey/model.py - Sign and token embeddings
signjoey/embeddings.py - Sequence encoders
signjoey/encoders.py - Transformer layers
signjoey/transformer_layers.py - Gloss recognition
signjoey/model.py - Text decoders and attention
signjoey/decoders.py - Recognition and translation search
signjoey/search.py
Training and inference
- Module CLI
signjoey/__main__.py - Training loop
signjoey/training.py - Prediction and testing
signjoey/prediction.py - Joint training loss
signjoey/loss.py
Evaluation
- Task metrics
signjoey/metrics.py - BLEU and related scores
signjoey/external_metrics/sacrebleu.py - ROUGE scores
signjoey/external_metrics/mscoco_rouge.py - Phoenix output cleanup
signjoey/phoenix_utils/phoenix_cleanup.py
Other components
- Researcher
Connections 27
- Researcher → Module CLI: runs module
README.md - Module CLI → Training loop: dispatches training
signjoey/__main__.py - Module CLI → Prediction and testing: dispatches prediction
signjoey/__main__.py - Training loop → Data loader: loads data
signjoey/training.py - Prediction and testing → Data loader: loads data
signjoey/prediction.py - Data loader → Sign datasets: constructs datasets
signjoey/data.py - Data loader → Gloss and text vocabularies: builds vocabularies
signjoey/data.py - Training loop → Model builder: builds model
signjoey/training.py - Prediction and testing → Model builder: builds model
signjoey/prediction.py - Training loop → Batch preparation: batches examples
signjoey/training.py - Prediction and testing → Batch preparation: batches examples
signjoey/prediction.py - Model builder → Joint SLT model: constructs
signjoey/model.py - Joint SLT model → Sign and token embeddings: embeds inputs
signjoey/model.py - Joint SLT model → Sequence encoders: encodes sign sequence
signjoey/model.py - Sequence encoders → Transformer layers: uses layers
signjoey/encoders.py - Joint SLT model → Gloss recognition: predicts glosses
signjoey/model.py - Joint SLT model → Text decoders and attention: decodes text
signjoey/model.py - Text decoders and attention → Transformer layers: uses layers
signjoey/decoders.py - Joint SLT model → Recognition and translation search: decodes outputs
signjoey/model.py - Recognition and translation search → Text decoders and attention: invokes decoder
signjoey/search.py - Training loop → Joint SLT model: trains model
signjoey/training.py - Training loop → Joint training loss: computes loss
signjoey/training.py - Prediction and testing → Joint SLT model: runs batches
signjoey/prediction.py - Prediction and testing → Task metrics: scores predictions
signjoey/prediction.py - Task metrics → BLEU and related scores: computes BLEU
signjoey/metrics.py - Task metrics → ROUGE scores: computes ROUGE
signjoey/metrics.py - Prediction and testing → Phoenix output cleanup: cleans outputs
signjoey/prediction.py