# ayushbits/pe-ocr-sanskrit architecture

> Architecture diagram of the GitHub repository ayushbits/pe-ocr-sanskrit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushbits/pe-ocr-sanskrit
- Repository: https://github.com/ayushbits/pe-ocr-sanskrit

## Overview

##### Project Type and Purpose
- This is a machine-learning research pipeline for post-OCR text correction in Sanskrit.
- It bundles raw OCR outputs, ground-truth annotations, transliteration utilities, training/inference scripts for neural correction models, and evaluation tools.

##### File Structure Analysis
- OCR-Images-Annotation/
- Contains raw OCR-extracted text files and their manually post-edited ground-truth for multiple books (BHS, GG, GOS).
- dev-transliterate-scripts/
- Utilities (C++ programs) to transliterate between Devanagari and SLP1.
- train-scripts/
- Model training (train_ByT5.py, train_BART.py, CopyNet folder) and inference (inference-byt5.py, inference-bart.py) code.
- preds/
- Model prediction CSVs on the out-of-domain test set.
- CSV split files (train_devnagari.csv, val_devnagari.csv, test_devnagari.csv, ood-test.csv)
- Evaluation scripts (metrics.py, word_count_cer.py) and dependencies (fastwer).

##### README Highlights
- Describes dataset splits, OOD test set, transliteration utilities, training scripts, and evaluation procedures (CER/WER calculation).

a. Main Components
- Data Sources
– OCR text folder (OCR-Images-Annotation)
– CSV splits for in-domain and OOD test sets
- Preprocessing & Utilities
– Transliteration module (Dev ↔ SLP1)
– Data cleaning/splitting (CSV generation)
- Model Training
– BART pipeline
– ByT5 pipeline
– CopyNet pipeline
- Inference
– Scripts to generate predictions on OOD test set
- Evaluation
– metrics.py / word_count_cer.py using fastwer
- Outputs
– preds/*.csv
b. Relationships & Data Flow
1. Raw OCR files → (Preprocessing) → CSV datasets → (Transliteration as needed)
2. CSV datasets → (Train-scripts) → Trained models
3. Trained models + OOD CSV → (Inference) → Predictions (preds/)
4. Predictions + Ground-truth → (Evaluation scripts) → CER/WER results
c. Architectural Patterns
- Linear data-processing pipeline (ETL → training → inference → evaluation)
- Modular codebase: separate scripts for transliteration, training, inference, and evaluation
d. Key Technologies
- Python (pandas, PyTorch/Transformers)
- C++ transliteration utilities
- fastwer for error-rate metrics
- CSV/text files and folder-based annotation structure

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Sources"
        OCR["Raw OCR Texts\n(OCR-Images-Annotation/)"]:::data
    end

    subgraph "Preprocessing & Utilities"
        PT["Data Cleaning & Splitting"]:::process
        DevTS["dev-transliterate-scripts/"]:::process
        TL1["transliterate-Dev_to_SLP1.cpp"]:::process
        TL2["transliterate-SLP1_to_Dev.cpp"]:::process
    end

    PT --> DS["CSV Splits\ntrain_devnagari.csv\nval_devnagari.csv\ntest_devnagari.csv\nood-test.csv"]:::data
    DevTS --> TL1
    DevTS --> TL2
    OCR --> PT

    subgraph "Model Training"
        BART["BART Pipeline\ntrain_BART.py"]:::model
        ByT5["ByT5 Pipeline\ntrain_ByT5.py"]:::model
        Copy["CopyNet Pipeline\n(train-scripts/CopyNet/)"]:::model
    end

    DS --> BART
    DS --> ByT5
    DS --> Copy

    subgraph "Inference"
        InfB["inference-bart.py"]:::model
        InfT["inference-byt5.py"]:::model
    end

    BART --> InfB
    ByT5 --> InfT
    Copy --> InfB
    Copy --> InfT
    DS --> InfB
    DS --> InfT

    InfB --> preds
    InfT --> preds

    preds["Prediction CSVs\n(preds/)"]:::data

    subgraph "Evaluation"
        M["metrics.py"]:::eval
        W["word_count_cer.py"]:::eval
    end

    preds --> M
    preds --> W

    click OCR "https://github.com/ayushbits/pe-ocr-sanskrit/tree/main/OCR-Images-Annotation/"
    click DevTS "https://github.com/ayushbits/pe-ocr-sanskrit/tree/main/dev-transliterate-scripts/"
    click TL1 "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/dev-transliterate-scripts/transliterate-Dev_to_SLP1.cpp"
    click TL2 "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/dev-transliterate-scripts/transliterate-SLP1_to_Dev.cpp"
    click BART "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/train-scripts/train_BART.py"
    click ByT5 "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/train-scripts/train_ByT5.py"
    click Copy "https://github.com/ayushbits/pe-ocr-sanskrit/tree/main/train-scripts/CopyNet/"
    click InfB "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/train-scripts/inference-bart.py"
    click InfT "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/train-scripts/inference-byt5.py"
    click preds "https://github.com/ayushbits/pe-ocr-sanskrit/tree/main/preds/"
    click M "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/metrics.py"
    click W "https://github.com/ayushbits/pe-ocr-sanskrit/blob/main/word_count_cer.py"

    classDef data fill:#D5E8D4,stroke:#82B366,color:#000
    classDef process fill:#FFF2CC,stroke:#D6B656,color:#000
    classDef model fill:#F8CECC,stroke:#B85450,color:#000
    classDef eval fill:#E1D5E7,stroke:#9673A6,color:#000
```
