# ffs333/2nd_place_gislr architecture

> Architecture diagram of the GitHub repository ffs333/2nd_place_gislr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ffs333/2nd_place_gislr
- Repository: https://github.com/ffs333/2nd_place_gislr

## Overview

##### Project Type and Purpose
- This is a research‐oriented machine‐learning solution library and training/inference toolkit for sign language recognition, packaged as a Python project with supporting notebooks.
- Its primary purpose is to preprocess landmark data, train CNN and transformer models on Kaggle competition data, convert models to Keras/TFLite, and build an ensemble for final inference.

##### High-Level Architectural Components
a. Data Layer
- Raw competition data (CSV or video‐extracted landmarks)
b. Core Utility Library (`GISLR_utils/`)
- data.py, mixup_data.py, partial_aug_data.py for preprocessing & augmentations
- losses.py, models.py for model definitions and loss functions
- pipeline.py and core.py orchestrate end-to-end workflows
- keras_models/ subpackage for Keras model definitions and conversion helpers
- transformer_code/ folder containing standalone PyTorch transformer training scripts and utilities
c. Training Pipelines
- CNN pipeline (EfficientNet B0): built on PyTorch, uses torch‐audio and custom augmentations
- Transformer pipeline (BERT/DeBERTa variants): separate script in `transformer_code/train_craft.py`
- Hyperparameter tuning with Optuna, one‐cycle and Ranger schedulers
d. Model Conversion Layer
- Scripts in keras_models/ rewrite PyTorch models in Keras
- ONNX & tf2onnx for intermediate conversions where needed
- Custom TFLite‐friendly `DepthwiseConv2D` implementation for speed
e. Ensemble & Inference
- A tf.Module that aggregates converted models
- Notebook examples (`kaggle_inference_notebook_example.ipynb`, `training_notebook_example.ipynb`)
- Weighted ensemble without softmax

##### Component Relationships and Data Flow
- Data Layer → Core Utils → Preprocessing & Augmentation → Two parallel branches:
- CNN Training (PyTorch) → weights
- Transformer Training (PyTorch) → weights
- Both branches feed into Model Conversion Layer → produce Keras/TFLite artifacts
- Converted models are wrapped into an Ensemble module → final inference

##### Architectural Patterns & Design Principles
- Modular library structure separating data, models, augmentations, and training routines
- Clear separation between training (PyTorch) and production/inference (TensorFlow/TFLite)
- Reuse of augmentation utilities across CNN and transformer pipelines
- Hyperparameter tuning as a decoupled step (Optuna integration)

##### Technologies & Frameworks
- Python 3.x, PyTorch for initial training
- TensorFlow/Keras and TFLite for production models
- ONNX and tf2onnx for intermediate graph conversions
- torchaudio for time/frequency augmentations
- Optuna for hyperparameter search
- Jupyter notebooks for examples and demos

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        Kaggle[(Kaggle Dataset)]:::external
        Img["imgs/"]:::dataPrep
    end
    click Img "https://github.com/ffs333/2nd_place_gislr/tree/main/imgs/"

    %% Core Utility Library
    subgraph "Core Utility Library"
        subgraph "Preprocessing & Augmentations"
            Data["data.py"]:::dataPrep
            Mixup["mixup_data.py"]:::dataPrep
            Partial["partial_aug_data.py"]:::dataPrep
        end
        subgraph "Models & Losses"
            Models["models.py"]:::dataPrep
            Losses["losses.py"]:::dataPrep
        end
        subgraph "Transformer Utilities"
            EffTrans["eff_transormer.py"]:::dataPrep
            Utils["utils.py"]:::dataPrep
        end
        Core["core.py"]:::dataPrep
        Pipeline["pipeline.py"]:::dataPrep
    end
    click Data "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/data.py"
    click Mixup "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/mixup_data.py"
    click Partial "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/partial_aug_data.py"
    click Models "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/models.py"
    click Losses "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/losses.py"
    click EffTrans "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/eff_transormer.py"
    click Utils "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/utils.py"
    click Core "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/core.py"
    click Pipeline "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/pipeline.py"

    %% Training Pipelines
    subgraph "CNN Training Pipeline"
        AudAug["Audio/Image Augmentations"]:::training
        CNN["PyTorch EfficientNet B0 Training"]:::training
        WeightsCNN[(CNN Weights)]:::external
    end

    subgraph "Transformer Training Pipeline"
        TransTrain["train_craft.py"]:::training
        TransUtil["Transformer Utils"]:::training
        WeightsTrans[(Transformer Weights)]:::external
    end
    click TransTrain "https://github.com/ffs333/2nd_place_gislr/blob/main/GISLR_utils/transformer_code/train_craft.py"
    click TransUtil "https://github.com/ffs333/2nd_place_gislr/tree/main/GISLR_utils/transformer_code/utils/"

    %% Conversion & Ensemble Layer
    subgraph "Conversion & Ensemble Layer"
        subgraph "Keras Model Conversion"
            KMSub["keras_models/"]:::conversion
            ONNX["ONNX & tf2onnx"]:::conversion
            TFLiteImp["Custom TFLite DepthwiseConv2D"]:::conversion
        end
        KerasArtifacts[(Keras/TFLite Models)]:::conversion
        Ensemble["Ensemble Module (tf.Module)"]:::conversion
    end
    click KMSub "https://github.com/ffs333/2nd_place_gislr/tree/main/GISLR_utils/keras_models/"

    %% Notebooks
    subgraph "Notebooks"
        TrainNB["training_notebook_example.ipynb"]:::external
        InfNB["kaggle_inference_notebook_example.ipynb"]:::external
    end

    %% Connections
    Kaggle -->|provides raw data| Data
    Img -->|assets for diagrams| Core
    Data -->|raw inputs| Pipeline
    Mixup -->|augment & normalize| Pipeline
    Partial -->|partial augment| Pipeline
    Pipeline -->|"feed CNN pipeline"| AudAug
    AudAug --> CNN
    Pipeline -->|"feed Transformer pipeline"| TransTrain
    TransUtil --> TransTrain
    CNN -->|produce weights| WeightsCNN
    TransTrain -->|produce weights| WeightsTrans
    WeightsCNN --> KMSub
    WeightsTrans --> KMSub
    KMSub --> ONNX
    KMSub --> TFLiteImp
    ONNX --> KerasArtifacts
    TFLiteImp --> KerasArtifacts
    KerasArtifacts --> Ensemble
    Ensemble --> InfNB
    Pipeline --> TrainNB

    %% Styles
    classDef dataPrep fill:#ccffcc,stroke:#006600,color:#000000
    classDef training fill:#cce5ff,stroke:#004085,color:#000000
    classDef conversion fill:#ffe5b4,stroke:#994d00,color:#000000
    classDef external fill:#f2f2f2,stroke:#999999,color:#000000
```
