How woshisad159/tfnet works
1. Project purpose and top-level flow
- TFNet is a continuous sign language recognition system built in PyTorch.
- The pipeline is: raw video data → offline preprocessing/augmentation → dataset loaders → TFNet model → CTC-style decoding → WER evaluation.
2. Dataset layer
- The repository already includes multiple dataset splits under data/: CE-CSL, CSL-Daily, RWTH, and RWTH-T.
- These directories hold the annotation files used by the training/evaluation pipeline (train/dev/test in CSV, TXT, or corpus CSV formats), while the raw video content is not stored directly in the repo.
3. Offline preprocessing boundary
- CE-CSLDataPreProcess.py converts video data into image/frame data for model consumption.
- videoAugmentation.py provides data expansion/perturbation at the video or frame level.
- This preprocessing step is outside the model runtime; it prepares the training inputs before Train.py or SLR.py run.
4. Configuration and data access
- params/config.ini stores runtime configuration such as dataset paths and experiment settings.
- ReadConfig.py is the configuration access layer, so path changes and dataset-specific options are centralized rather than hardcoded across the training code.
- DataProcessMoudle.py is the main data pipeline component, likely responsible for reading annotations, loading frame sequences, and batching samples.
5. Model composition
- The core network is assembled in Net.py.
- The model is modularized into separate building blocks:
- Transformer.py for attention-based temporal modeling
- BiLSTM.py for recurrent sequence modeling
- SEN.py for channel/spatial enhancement
- Module.py for shared layers or utility blocks
- This separation suggests the architecture is a hybrid temporal recognizer rather than a single monolithic network.
6. Training orchestration
- Train.py appears to contain the training loop, optimization, checkpointing, and validation logic.
- SLR.py is called out in the README as the main entry point, so it likely wraps experiment setup and kicks off training/inference using the components above.
- The training path consumes preprocessed frame sequences and target gloss/text labels from the dataset layer.
7. Inference and decoding
- decode.py handles model output decoding.
- The README lists ctcdecode==0.4, which indicates beam-search decoding over CTC-style sequence outputs.
- This is the boundary where frame-level predictions are converted into symbol sequences for evaluation.
8. Evaluation pipeline
- WER.py computes word error rate for recognition results.
- The evaluation/ and evaluationT/ folders contain dataset-specific evaluation scripts and artifacts for PHOENIX-style corpora.
- These folders include STM/CTM files, merge scripts, and WER scripts, showing that output is converted into corpus evaluation formats before scoring.
9. Dataset-specific evaluation split
- evaluation/ is aligned with the non-T PHOENIX-2014 evaluation flow.
- evaluationT/ is a separate variant for PHOENIX-2014-T, with its own preprocessing and WER scripts.
- This indicates the repo supports at least two closely related benchmark evaluation conventions.
10. External artifacts and runtime dependencies
- The project depends on PyTorch 1.11 and ctcdecode for inference-time decoding.
- The README also points to downloadable pretrained weights, so the repo is designed to run with externally obtained checkpoints rather than training from scratch only.
- The mention of PyCharm is operational rather than architectural; the important runtime boundary is Python/PyTorch.
11. What an architecture diagram should show
- A left-to-right flow from dataset/video inputs into preprocessing, then into the loader/config layer, then into the TFNet model stack, then into CTC decoding, and finally into WER evaluation.
- The model box should be subdivided into the reusable temporal modules (Transformer, BiLSTM, SEN, shared blocks) assembled by Net.py.
- Evaluation should be shown as a separate post-processing path with dataset-specific branches for evaluation/ and evaluationT/.
Main entry points: DataProcessMoudle.py Data loader · Net.py Net · decode.py Decode · Train.py Train loop
Components 21
Inputs
Dataset splits and raw video assets used by the pipeline.
- CE-CSL · Train/dev/test annotations for the CE-CSL corpus.
data/CE-CSL - CSL-Daily · Annotation files and related metadata for CSL-Daily.
data/CSL-Daily - RWTH · Corpus CSV splits for the RWTH benchmark.
data/RWTH - RWTH-T · Corpus CSV splits for the PHOENIX-2014-T benchmark.
data/RWTH-T - Raw video · Source video content used to generate model-ready frame data.
video.rar
Preprocessing
Offline frame extraction and augmentation before training.
- Frame prep · Converts video into image/frame sequences for model consumption.
CE-CSLDataPreProcess.py - Augment · Expands and perturbs video or frame data before training.
videoAugmentation.py
Config & Data
Configuration access and sample loading for training and evaluation.
- Config · Central experiment and dataset path configuration.
params/config.ini - ReadConfig · Loads and exposes configuration values to the rest of the code.
ReadConfig.py - Data loader · Reads annotations, loads frame sequences, and batches samples.
DataProcessMoudle.py - SLR · Main entry script that wires setup and execution.
SLR.py - Train loop · Runs optimization, checkpointing, and validation.
Train.py
TFNet Model
Core hybrid sequence model assembled from reusable modules.
- Net · Builds the TFNet network from modular sequence components.
Net.py - Transformer · Attention-based temporal modeling block.
Transformer.py - BiLSTM · Recurrent sequence modeling block.
BiLSTM.py - SEN · Channel/spatial enhancement module.
SEN.py - Module · Reusable layers and helper components used by the network.
Module.py
Decode & Eval
CTC decoding and corpus-level scoring for benchmark outputs.
- Decode · Converts model outputs into symbol sequences, likely with beam search.
decode.py - WER · Computes word error rate for recognition results.
WER.py - Eval · PHOENIX-2014 evaluation flow for non-T benchmarks.
evaluation/evaluation.py - Eval-T · PHOENIX-2014-T evaluation flow with its own scripts.
evaluationT/evaluationT.py
Connections 23
- Raw video → Frame prep: source · Raw video is converted into frame sequences.
- Raw video → Augment: source · Video-level or frame-level augmentation operates on raw media.
- Frame prep → Data loader: frames · Prepared frame sequences feed the data loader.
- Augment → Data loader: augmented data · Augmented samples are consumed during training.
- Config → ReadConfig: loaded by · Configuration is accessed through a dedicated reader.
- ReadConfig → Data loader: paths/settings · Dataset locations and runtime options guide sample loading.
- CE-CSL → Data loader: annotations · CE-CSL splits provide labels and sample definitions.
- CSL-Daily → Data loader: annotations · CSL-Daily splits provide labels and sample definitions.
- RWTH → Data loader: annotations · RWTH splits provide labels and sample definitions.
- RWTH-T → Data loader: annotations · RWTH-T splits provide labels and sample definitions.
- SLR → Train loop: starts · Main entry point launches training or experiment execution.
- Data loader → Train loop: batches · Training consumes batched frame sequences and targets.
- Train loop → Net: builds · The training loop instantiates the TFNet model.
- Net → Transformer: uses · Temporal attention block inside the model.
- Net → BiLSTM: uses · Recurrent sequence block inside the model.
- Net → SEN: uses · Feature enhancement block inside the model.
- Net → Module: uses · Shared layers and utility blocks supporting the model.
- Train loop → Decode: outputs · Model predictions are decoded during validation or inference.
- Decode → WER: scored by · Decoded sequences are evaluated with WER.
- Decode → Eval: formatted for · Outputs are converted into corpus evaluation artifacts.
- Decode → Eval-T: formatted for · Outputs are converted into PHOENIX-2014-T evaluation artifacts.
- Eval → WER: reports · Corpus evaluation scripts produce WER results.
- Eval-T → WER: reports · T-variant evaluation scripts produce WER results.