# woshisad159/tfnet architecture

> Architecture diagram of the GitHub repository woshisad159/tfnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/woshisad159/tfnet
- Repository: https://github.com/woshisad159/tfnet

## Overview

1. **Project purpose and top-level flow**
   - TFNet is a continuous sign language recognition system built in PyTorch.
   - The pipeline is: raw video data → offline preprocessing/augmentation → dataset loaders → TFNet model → CTC-style decoding → WER evaluation.

2. **Dataset layer**
   - The repository already includes multiple dataset splits under `data/`: `CE-CSL`, `CSL-Daily`, `RWTH`, and `RWTH-T`.
   - These directories hold the annotation files used by the training/evaluation pipeline (`train/dev/test` in CSV, TXT, or corpus CSV formats), while the raw video content is not stored directly in the repo.

3. **Offline preprocessing boundary**
   - `CE-CSLDataPreProcess.py` converts video data into image/frame data for model consumption.
   - `videoAugmentation.py` provides data expansion/perturbation at the video or frame level.
   - This preprocessing step is outside the model runtime; it prepares the training inputs before `Train.py` or `SLR.py` run.

4. **Configuration and data access**
   - `params/config.ini` stores runtime configuration such as dataset paths and experiment settings.
   - `ReadConfig.py` is the configuration access layer, so path changes and dataset-specific options are centralized rather than hardcoded across the training code.
   - `DataProcessMoudle.py` is the main data pipeline component, likely responsible for reading annotations, loading frame sequences, and batching samples.

5. **Model composition**
   - The core network is assembled in `Net.py`.
   - The model is modularized into separate building blocks:
     - `Transformer.py` for attention-based temporal modeling
     - `BiLSTM.py` for recurrent sequence modeling
     - `SEN.py` for channel/spatial enhancement
     - `Module.py` for shared layers or utility blocks
   - This separation suggests the architecture is a hybrid temporal recognizer rather than a single monolithic network.

6. **Training orchestration**
   - `Train.py` appears to contain the training loop, optimization, checkpointing, and validation logic.
   - `SLR.py` is called out in the README as the main entry point, so it likely wraps experiment setup and kicks off training/inference using the components above.
   - The training path consumes preprocessed frame sequences and target gloss/text labels from the dataset layer.

7. **Inference and decoding**
   - `decode.py` handles model output decoding.
   - The README lists `ctcdecode==0.4`, which indicates beam-search decoding over CTC-style sequence outputs.
   - This is the boundary where frame-level predictions are converted into symbol sequences for evaluation.

8. **Evaluation pipeline**
   - `WER.py` computes word error rate for recognition results.
   - The `evaluation/` and `evaluationT/` folders contain dataset-specific evaluation scripts and artifacts for PHOENIX-style corpora.
   - These folders include STM/CTM files, merge scripts, and WER scripts, showing that output is converted into corpus evaluation formats before scoring.

9. **Dataset-specific evaluation split**
   - `evaluation/` is aligned with the non-T PHOENIX-2014 evaluation flow.
   - `evaluationT/` is a separate variant for PHOENIX-2014-T, with its own preprocessing and WER scripts.
   - This indicates the repo supports at least two closely related benchmark evaluation conventions.

10. **External artifacts and runtime dependencies**
   - The project depends on PyTorch 1.11 and `ctcdecode` for inference-time decoding.
   - The README also points to downloadable pretrained weights, so the repo is designed to run with externally obtained checkpoints rather than training from scratch only.
   - The mention of PyCharm is operational rather than architectural; the important runtime boundary is Python/PyTorch.

11. **What an architecture diagram should show**
   - A left-to-right flow from dataset/video inputs into preprocessing, then into the loader/config layer, then into the TFNet model stack, then into CTC decoding, and finally into WER evaluation.
   - The model box should be subdivided into the reusable temporal modules (`Transformer`, `BiLSTM`, `SEN`, shared blocks) assembled by `Net.py`.
   - Evaluation should be shown as a separate post-processing path with dataset-specific branches for `evaluation/` and `evaluationT/`.

Main entry points: [`DataProcessMoudle.py`](https://github.com/woshisad159/tfnet/blob/main/DataProcessMoudle.py) (Data loader), [`Net.py`](https://github.com/woshisad159/tfnet/blob/main/Net.py) (Net), [`decode.py`](https://github.com/woshisad159/tfnet/blob/main/decode.py) (Decode), [`Train.py`](https://github.com/woshisad159/tfnet/blob/main/Train.py) (Train loop)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_inputs["Inputs"]
  node_n_cecsl[("CE-CSL<br/>dataset split")]
  node_n_csldaily[("CSL-Daily<br/>dataset split")]
  node_n_rwth[("RWTH<br/>dataset split")]
  node_n_rwtht[("RWTH-T<br/>dataset split")]
  node_n_video[("Raw video<br/>external media<br/>[video.rar]")]
end

subgraph group_g_prep["Preprocessing"]
  node_n_preprocess["Frame prep<br/>offline preprocessing"]
  node_n_augment["Augment"]
end

subgraph group_g_data["Config & Data"]
  node_n_config["Config<br/>runtime settings<br/>[config.ini]"]
  node_n_readconfig["ReadConfig<br/>config access<br/>[ReadConfig.py]"]
  node_n_dataproc["Data loader<br/>data pipeline"]
  node_n_slr["SLR<br/>entry point<br/>[SLR.py]"]
  node_n_train["Train loop<br/>training orchestration<br/>[Train.py]"]
end

subgraph group_g_model["TFNet Model"]
  node_n_net["Net<br/>model assembly<br/>[Net.py]"]
  node_n_transformer["Transformer<br/>temporal attention<br/>[Transformer.py]"]
  node_n_bilstm["BiLSTM<br/>sequence encoder<br/>[BiLSTM.py]"]
  node_n_sen["SEN<br/>feature enhancement<br/>[SEN.py]"]
  node_n_module["Module<br/>shared blocks<br/>[Module.py]"]
end

subgraph group_g_post["Decode & Eval"]
  node_n_decode["Decode<br/>ctc decoding<br/>[decode.py]"]
  node_n_wer["WER<br/>evaluation metric<br/>[WER.py]"]
  node_n_eval["Eval<br/>[evaluation.py]"]
  node_n_evalt["Eval-T<br/>corpus eval<br/>[evaluationT.py]"]
end

node_n_video -->|"source"| node_n_preprocess
node_n_video -->|"source"| node_n_augment
node_n_preprocess -->|"frames"| node_n_dataproc
node_n_augment -->|"augmented data"| node_n_dataproc
node_n_config -->|"loaded by"| node_n_readconfig
node_n_readconfig -->|"paths/settings"| node_n_dataproc
node_n_cecsl -->|"annotations"| node_n_dataproc
node_n_csldaily -->|"annotations"| node_n_dataproc
node_n_rwth -->|"annotations"| node_n_dataproc
node_n_rwtht -->|"annotations"| node_n_dataproc
node_n_slr -->|"starts"| node_n_train
node_n_dataproc -->|"batches"| node_n_train
node_n_train -->|"builds"| node_n_net
node_n_net -->|"uses"| node_n_transformer
node_n_net -->|"uses"| node_n_bilstm
node_n_net -->|"uses"| node_n_sen
node_n_net -->|"uses"| node_n_module
node_n_train -->|"outputs"| node_n_decode
node_n_decode -->|"scored by"| node_n_wer
node_n_decode -->|"formatted for"| node_n_eval
node_n_decode -->|"formatted for"| node_n_evalt
node_n_eval -->|"reports"| node_n_wer
node_n_evalt -->|"reports"| node_n_wer

click node_n_cecsl "https://github.com/woshisad159/tfnet/tree/main/data/CE-CSL"
click node_n_csldaily "https://github.com/woshisad159/tfnet/tree/main/data/CSL-Daily"
click node_n_rwth "https://github.com/woshisad159/tfnet/tree/main/data/RWTH"
click node_n_rwtht "https://github.com/woshisad159/tfnet/tree/main/data/RWTH-T"
click node_n_video "https://github.com/woshisad159/tfnet/blob/main/video.rar"
click node_n_preprocess "https://github.com/woshisad159/tfnet/blob/main/CE-CSLDataPreProcess.py"
click node_n_augment "https://github.com/woshisad159/tfnet/blob/main/videoAugmentation.py"
click node_n_config "https://github.com/woshisad159/tfnet/blob/main/params/config.ini"
click node_n_readconfig "https://github.com/woshisad159/tfnet/blob/main/ReadConfig.py"
click node_n_dataproc "https://github.com/woshisad159/tfnet/blob/main/DataProcessMoudle.py"
click node_n_slr "https://github.com/woshisad159/tfnet/blob/main/SLR.py"
click node_n_train "https://github.com/woshisad159/tfnet/blob/main/Train.py"
click node_n_net "https://github.com/woshisad159/tfnet/blob/main/Net.py"
click node_n_transformer "https://github.com/woshisad159/tfnet/blob/main/Transformer.py"
click node_n_bilstm "https://github.com/woshisad159/tfnet/blob/main/BiLSTM.py"
click node_n_sen "https://github.com/woshisad159/tfnet/blob/main/SEN.py"
click node_n_module "https://github.com/woshisad159/tfnet/blob/main/Module.py"
click node_n_decode "https://github.com/woshisad159/tfnet/blob/main/decode.py"
click node_n_wer "https://github.com/woshisad159/tfnet/blob/main/WER.py"
click node_n_eval "https://github.com/woshisad159/tfnet/blob/main/evaluation/evaluation.py"
click node_n_evalt "https://github.com/woshisad159/tfnet/blob/main/evaluationT/evaluationT.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_cecsl,node_n_csldaily,node_n_rwth,node_n_rwtht,node_n_video toneBlue
class node_n_preprocess,node_n_augment toneAmber
class node_n_config,node_n_readconfig,node_n_dataproc,node_n_slr,node_n_train toneMint
class node_n_net,node_n_transformer,node_n_bilstm,node_n_sen,node_n_module toneRose
class node_n_decode,node_n_wer,node_n_eval,node_n_evalt toneIndigo
```

## Components

### Inputs

Dataset splits and raw video assets used by the pipeline.

- [CE-CSL](https://github.com/woshisad159/tfnet/tree/main/data/CE-CSL): dataset split · Train/dev/test annotations for the CE-CSL corpus. · `data/CE-CSL`
- [CSL-Daily](https://github.com/woshisad159/tfnet/tree/main/data/CSL-Daily): dataset split · Annotation files and related metadata for CSL-Daily. · `data/CSL-Daily`
- [RWTH](https://github.com/woshisad159/tfnet/tree/main/data/RWTH): dataset split · Corpus CSV splits for the RWTH benchmark. · `data/RWTH`
- [RWTH-T](https://github.com/woshisad159/tfnet/tree/main/data/RWTH-T): dataset split · Corpus CSV splits for the PHOENIX-2014-T benchmark. · `data/RWTH-T`
- [Raw video](https://github.com/woshisad159/tfnet/blob/main/video.rar): external media · Source video content used to generate model-ready frame data. · `video.rar`

### Preprocessing

Offline frame extraction and augmentation before training.

- [Frame prep](https://github.com/woshisad159/tfnet/blob/main/CE-CSLDataPreProcess.py): offline preprocessing · Converts video into image/frame sequences for model consumption. · `CE-CSLDataPreProcess.py`
- [Augment](https://github.com/woshisad159/tfnet/blob/main/videoAugmentation.py): video augmentation · Expands and perturbs video or frame data before training. · `videoAugmentation.py`

### Config & Data

Configuration access and sample loading for training and evaluation.

- [Config](https://github.com/woshisad159/tfnet/blob/main/params/config.ini): runtime settings · Central experiment and dataset path configuration. · `params/config.ini`
- [ReadConfig](https://github.com/woshisad159/tfnet/blob/main/ReadConfig.py): config access · Loads and exposes configuration values to the rest of the code. · `ReadConfig.py`
- [Data loader](https://github.com/woshisad159/tfnet/blob/main/DataProcessMoudle.py): data pipeline · Reads annotations, loads frame sequences, and batches samples. · `DataProcessMoudle.py`
- [SLR](https://github.com/woshisad159/tfnet/blob/main/SLR.py): entry point · Main entry script that wires setup and execution. · `SLR.py`
- [Train loop](https://github.com/woshisad159/tfnet/blob/main/Train.py): training orchestration · Runs optimization, checkpointing, and validation. · `Train.py`

### TFNet Model

Core hybrid sequence model assembled from reusable modules.

- [Net](https://github.com/woshisad159/tfnet/blob/main/Net.py): model assembly · Builds the TFNet network from modular sequence components. · `Net.py`
- [Transformer](https://github.com/woshisad159/tfnet/blob/main/Transformer.py): temporal attention · Attention-based temporal modeling block. · `Transformer.py`
- [BiLSTM](https://github.com/woshisad159/tfnet/blob/main/BiLSTM.py): sequence encoder · Recurrent sequence modeling block. · `BiLSTM.py`
- [SEN](https://github.com/woshisad159/tfnet/blob/main/SEN.py): feature enhancement · Channel/spatial enhancement module. · `SEN.py`
- [Module](https://github.com/woshisad159/tfnet/blob/main/Module.py): shared blocks · Reusable layers and helper components used by the network. · `Module.py`

### Decode & Eval

CTC decoding and corpus-level scoring for benchmark outputs.

- [Decode](https://github.com/woshisad159/tfnet/blob/main/decode.py): ctc decoding · Converts model outputs into symbol sequences, likely with beam search. · `decode.py`
- [WER](https://github.com/woshisad159/tfnet/blob/main/WER.py): evaluation metric · Computes word error rate for recognition results. · `WER.py`
- [Eval](https://github.com/woshisad159/tfnet/blob/main/evaluation/evaluation.py): corpus eval · PHOENIX-2014 evaluation flow for non-T benchmarks. · `evaluation/evaluation.py`
- [Eval-T](https://github.com/woshisad159/tfnet/blob/main/evaluationT/evaluationT.py): corpus eval · PHOENIX-2014-T evaluation flow with its own scripts. · `evaluationT/evaluationT.py`

## Connections

- Raw video → Frame prep: source · Raw video is converted into frame sequences.
- Raw video → Augment: source · Video-level or frame-level augmentation operates on raw media.
- Frame prep → Data loader: frames · Prepared frame sequences feed the data loader.
- Augment → Data loader: augmented data · Augmented samples are consumed during training.
- Config → ReadConfig: loaded by · Configuration is accessed through a dedicated reader.
- ReadConfig → Data loader: paths/settings · Dataset locations and runtime options guide sample loading.
- CE-CSL → Data loader: annotations · CE-CSL splits provide labels and sample definitions.
- CSL-Daily → Data loader: annotations · CSL-Daily splits provide labels and sample definitions.
- RWTH → Data loader: annotations · RWTH splits provide labels and sample definitions.
- RWTH-T → Data loader: annotations · RWTH-T splits provide labels and sample definitions.
- SLR → Train loop: starts · Main entry point launches training or experiment execution.
- Data loader → Train loop: batches · Training consumes batched frame sequences and targets.
- Train loop → Net: builds · The training loop instantiates the TFNet model.
- Net → Transformer: uses · Temporal attention block inside the model.
- Net → BiLSTM: uses · Recurrent sequence block inside the model.
- Net → SEN: uses · Feature enhancement block inside the model.
- Net → Module: uses · Shared layers and utility blocks supporting the model.
- Train loop → Decode: outputs · Model predictions are decoded during validation or inference.
- Decode → WER: scored by · Decoded sequences are evaluated with WER.
- Decode → Eval: formatted for · Outputs are converted into corpus evaluation artifacts.
- Decode → Eval-T: formatted for · Outputs are converted into PHOENIX-2014-T evaluation artifacts.
- Eval → WER: reports · Corpus evaluation scripts produce WER results.
- Eval-T → WER: reports · T-variant evaluation scripts produce WER results.