How ffs333/2nd_place_gislr works
Project Type and Purpose
• This is a research‐oriented machine‐learning solution library and training/inference toolkit for sign language recognition, packaged as a Python project with supporting notebooks.
• Its primary purpose is to preprocess landmark data, train CNN and transformer models on Kaggle competition data, convert models to Keras/TFLite, and build an ensemble for final inference.
High-Level Architectural Components
a. Data Layer
• Raw competition data (CSV or video‐extracted landmarks)
b. Core Utility Library (GISLR_utils/)
• data.py, mixup_data.py, partial_aug_data.py for preprocessing & augmentations
• losses.py, models.py for model definitions and loss functions
• pipeline.py and core.py orchestrate end-to-end workflows
• keras_models/ subpackage for Keras model definitions and conversion helpers
• transformer_code/ folder containing standalone PyTorch transformer training scripts and utilities
c. Training Pipelines
• CNN pipeline (EfficientNet B0): built on PyTorch, uses torch‐audio and custom augmentations
• Transformer pipeline (BERT/DeBERTa variants): separate script in transformer_code/train_craft.py
• Hyperparameter tuning with Optuna, one‐cycle and Ranger schedulers
d. Model Conversion Layer
• Scripts in keras_models/ rewrite PyTorch models in Keras
• ONNX & tf2onnx for intermediate conversions where needed
• Custom TFLite‐friendly DepthwiseConv2D implementation for speed
e. Ensemble & Inference
• A tf.Module that aggregates converted models
• Notebook examples (kaggle_inference_notebook_example.ipynb, training_notebook_example.ipynb)
• Weighted ensemble without softmax
Component Relationships and Data Flow
• Data Layer → Core Utils → Preprocessing & Augmentation → Two parallel branches:
• CNN Training (PyTorch) → weights
• Transformer Training (PyTorch) → weights
• Both branches feed into Model Conversion Layer → produce Keras/TFLite artifacts
• Converted models are wrapped into an Ensemble module → final inference
Architectural Patterns & Design Principles
• Modular library structure separating data, models, augmentations, and training routines
• Clear separation between training (PyTorch) and production/inference (TensorFlow/TFLite)
• Reuse of augmentation utilities across CNN and transformer pipelines
• Hyperparameter tuning as a decoupled step (Optuna integration)
Technologies & Frameworks
• Python 3.x, PyTorch for initial training
• TensorFlow/Keras and TFLite for production models
• ONNX and tf2onnx for intermediate graph conversions
• torchaudio for time/frequency augmentations
• Optuna for hyperparameter search
• Jupyter notebooks for examples and demos