How vamosc/clip4str works
Project type and purpose
• This is a research‐oriented, open-source machine learning framework for scene text recognition (STR).
• It implements the CLIP4STR method: a two-branch model (visual branch + cross-modal branch) built on pre-trained vision-language models (CLIP/OpenCLIP).
• Provides end-to-end pipelines for data preparation, training, evaluation, inference, and dataset conversion.
File structure
Top-level:
• bench.py, train.py, read.py, test.py, tune.py – entry points for benchmarking, training, reading, testing, hyperparameter tuning.
• configs/ – YAML configurations for datasets, models, experiments, tuning.
• scripts/ – shell scripts wrapping common commands (training, testing, reading).
• tools/ – Python scripts to convert and preprocess various STR datasets into LMDB or other formats.
• strhub/ – core Python package containing subpackages:
• clip/ – CLIP tokenizer, model loader, vocabulary files.
• data/ – dataset definitions, augmentation, data module (PyTorch Lightning), utils.
• models/ – model implementations for different STR backbones: abinet, crnn, parseq, trba, vitstr, vl_str, str_adapter. Each has modules, system definitions, forward logic.
• dist_utils.py – distributed training utilities.
• hubconf.py – Torch Hub entry point.
• requirements.txt – Python dependencies.
README for additional insights
• Confirms two-branch architecture: a visual branch (CLIP vision encoder) and a cross-modal branch (text encoder + adapter) that refines predictions.
• Relies on PyTorch Lightning to orchestrate training and validation loops.
• Describes how to prepare datasets, where pre-trained weights reside, and how to launch experiments via scripts.
• Shows typical directory layout (dataset/, code/, output/, pretrained/).
Elements
• Clear component labels: “Dataset Converter”, “DataModule”, “CLIP Vision Encoder”, “Cross-Modal Branch”, “Lightning Trainer”, “Inference Script”
• Data conversion → DataModule
• DataModule → Model (vision encoder & text encoder)
• Model → Trainer & checkpointing
• Trainer → Inference
• Rectangles for code modules (data, model, tools)
• Parallelograms for config files
• Clouds for external dependencies (pre-trained weights, PyTorch Lightning)