# vamosc/clip4str architecture

> Architecture diagram of the GitHub repository vamosc/clip4str, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vamosc/clip4str
- Repository: https://github.com/vamosc/clip4str

## Overview

##### Project type and purpose
- This is a research‐oriented, open-source machine learning framework for scene text recognition (STR).
- It implements the CLIP4STR method: a two-branch model (visual branch + cross-modal branch) built on pre-trained vision-language models (CLIP/OpenCLIP).
- Provides end-to-end pipelines for data preparation, training, evaluation, inference, and dataset conversion.

##### File structure
Top-level:
- bench.py, train.py, read.py, test.py, tune.py – entry points for benchmarking, training, reading, testing, hyperparameter tuning.
- configs/ – YAML configurations for datasets, models, experiments, tuning.
- scripts/ – shell scripts wrapping common commands (training, testing, reading).
- tools/ – Python scripts to convert and preprocess various STR datasets into LMDB or other formats.
- strhub/ – core Python package containing subpackages:
- clip/ – CLIP tokenizer, model loader, vocabulary files.
- data/ – dataset definitions, augmentation, data module (PyTorch Lightning), utils.
- models/ – model implementations for different STR backbones: abinet, crnn, parseq, trba, vitstr, vl_str, str_adapter. Each has modules, system definitions, forward logic.
- dist_utils.py – distributed training utilities.
- hubconf.py – Torch Hub entry point.
- requirements.txt – Python dependencies.

##### README for additional insights
- Confirms two-branch architecture: a visual branch (CLIP vision encoder) and a cross-modal branch (text encoder + adapter) that refines predictions.
- Relies on PyTorch Lightning to orchestrate training and validation loops.
- Describes how to prepare datasets, where pre-trained weights reside, and how to launch experiments via scripts.
- Shows typical directory layout (dataset/, code/, output/, pretrained/).

##### Elements
- Clear component labels: “Dataset Converter”, “DataModule”, “CLIP Vision Encoder”, “Cross-Modal Branch”, “Lightning Trainer”, “Inference Script”
- Data conversion → DataModule
- DataModule → Model (vision encoder & text encoder)
- Model → Trainer & checkpointing
- Trainer → Inference
- Rectangles for code modules (data, model, tools)
- Parallelograms for config files
- Clouds for external dependencies (pre-trained weights, PyTorch Lightning)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources and Conversion
    RawData["Raw Images & Annotations"]:::db
    Converter["Data Converter Tools"]:::script
    LMDB["Standardized Dataset (LMDB)"]:::db

    RawData -->|convert via| Converter
    Converter -->|outputs| LMDB

    %% Configuration Layer
    Config["Configuration Files"]:::config

    %% Data Pipeline
    subgraph "Data Pipeline"
        direction TB
        Utils["Data Utilities"]:::module
        DatasetDef["Dataset Definitions"]:::module
        Augment["Data Augmentation Module"]:::module
        DataModule["PyTorch Lightning DataModule"]:::module
    end

    LMDB -->|read| Utils
    Utils --> DatasetDef
    DatasetDef --> Augment
    Augment --> DataModule
    Config --> DataModule

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        subgraph "CLIP Interface"
            VisionLoader["CLIP Vision Encoder Loader"]:::model
            CLIPCore["CLIP Core Interface"]:::model
            Tokenizer["CLIP Tokenizer"]:::model
            Vocab["CLIP Vocabulary"]:::model
        end
        DistUtils["Distributed Training Utilities"]:::module
        subgraph "Backbones"
            ABINet["ABINet Backbone"]:::model
            CRNN["CRNN Backbone"]:::model
            PARSeq["PARSeq Backbone"]:::model
            TRBA["TRBA Backbone"]:::model
            ViTSTR["ViTSTR Backbone"]:::model
            VLSTR["VL_STR Cross-Modal Branch"]:::model
            STRAdapter["STR Adapter Module"]:::model
        end
        PostProcess["Model Post-Processing"]:::module
    end

    DataModule -->|provides batches| VisionLoader
    DataModule -->|provides batches| CLIPCore
    VisionLoader --> CLIPCore
    CLIPCore --> Tokenizer
    Tokenizer --> Vocab
    CLIPCore -->|features| ABINet & CRNN & PARSeq & TRBA & ViTSTR & VLSTR & STRAdapter
    ABINet & CRNN & PARSeq & TRBA & ViTSTR & VLSTR & STRAdapter --> PostProcess
    DistUtils --> VisionLoader
    DistUtils --> CLIPCore

    Config --> VisionLoader
    Config --> CLIPCore
    Config --> DistUtils
    Config --> ABINet & CRNN & PARSeq & TRBA & ViTSTR & VLSTR & STRAdapter

    %% Training & Orchestration
    subgraph "Training & Orchestration"
        direction TB
        Train["Training Script"]:::script
        Bench["Benchmarking Entry Point"]:::script
        Tune["Hyperparameter Tuning Script"]:::script
        Hubconf["Torch Hub Entry Point"]:::script
        Requirements["Python Dependencies"]:::external
    end

    Train -->|uses| DataModule
    Train -->|uses| VisionLoader
    Train -->|uses| CLIPCore
    Train -->|writes| Checkpoints["Model Checkpoints"]:::db
    Bench -->|uses| DataModule
    Tune -->|reads| Config
    Hubconf -->|exposes| VisionLoader
    Requirements -->|provides| train.py & scripts

    %% Inference & Evaluation
    subgraph "Inference & Evaluation"
        direction TB
        Read["Inference Script"]:::script
        Test["Evaluation/Test Script"]:::script
        Scripts["Shell Orchestration Scripts"]:::script
        ExampleImage["Example Images"]:::db
    end

    Checkpoints --> Read
    Read -->|loads| DataModule
    Read -->|loads| VisionLoader
    Test -->|loads| Checkpoints
    Scripts --> Read & Test & Train

    Config --> Read
    Config --> Test

    ExampleImage --> Read

    %% Styles
    classDef db fill:#d1e8ff,stroke:#0366d6;
    classDef module fill:#e2f0d9,stroke:#22863a;
    classDef model fill:#fde047,stroke:#bf8700;
    classDef config fill:#f0db4f,stroke:#b08900;
    classDef script fill:#f5e1da,stroke:#d73a49;
    classDef external fill:#f0f0f0,stroke:#444;

    %% Click Events
    click Converter "https://github.com/vamosc/clip4str/tree/main/tools/"
    click Utils "https://github.com/vamosc/clip4str/blob/main/strhub/data/utils.py"
    click DatasetDef "https://github.com/vamosc/clip4str/blob/main/strhub/data/dataset.py"
    click Augment "https://github.com/vamosc/clip4str/blob/main/strhub/data/augment.py"
    click DataModule "https://github.com/vamosc/clip4str/blob/main/strhub/data/module.py"
    click Config "https://github.com/vamosc/clip4str/tree/main/configs/"
    click VisionLoader "https://github.com/vamosc/clip4str/blob/main/strhub/clip/model.py"
    click CLIPCore "https://github.com/vamosc/clip4str/blob/main/strhub/clip/clip.py"
    click Tokenizer "https://github.com/vamosc/clip4str/blob/main/strhub/clip/simple_tokenizer.py"
    click Vocab "https://github.com/vamosc/clip4str/blob/main/strhub/clip/bpe_simple_vocab_16e6.txt.gz"
    click DistUtils "https://github.com/vamosc/clip4str/blob/main/strhub/dist_utils.py"
    click PostProcess "https://github.com/vamosc/clip4str/blob/main/strhub/models/post_process.py"
    click ABINet "https://github.com/vamosc/clip4str/tree/main/strhub/models/abinet/"
    click CRNN "https://github.com/vamosc/clip4str/tree/main/strhub/models/crnn/"
    click PARSeq "https://github.com/vamosc/clip4str/tree/main/strhub/models/parseq/"
    click TRBA "https://github.com/vamosc/clip4str/tree/main/strhub/models/trba/"
    click ViTSTR "https://github.com/vamosc/clip4str/tree/main/strhub/models/vitstr/"
    click VLSTR "https://github.com/vamosc/clip4str/tree/main/strhub/models/vl_str/"
    click STRAdapter "https://github.com/vamosc/clip4str/tree/main/strhub/models/str_adapter/"
    click Train "https://github.com/vamosc/clip4str/blob/main/train.py"
    click Bench "https://github.com/vamosc/clip4str/blob/main/bench.py"
    click Tune "https://github.com/vamosc/clip4str/blob/main/tune.py"
    click Read "https://github.com/vamosc/clip4str/blob/main/read.py"
    click Test "https://github.com/vamosc/clip4str/blob/main/test.py"
    click Scripts "https://github.com/vamosc/clip4str/tree/main/scripts/"
    click Hubconf "https://github.com/vamosc/clip4str/blob/main/hubconf.py"
    click Requirements "https://github.com/vamosc/clip4str/blob/main/requirements.txt"
    click ExampleImage "https://github.com/vamosc/clip4str/tree/main/misc/test_image"
```
