How ayushbits/pe-ocr-sanskrit works
Project Type and Purpose
• This is a machine-learning research pipeline for post-OCR text correction in Sanskrit.
• It bundles raw OCR outputs, ground-truth annotations, transliteration utilities, training/inference scripts for neural correction models, and evaluation tools.
File Structure Analysis
• OCR-Images-Annotation/
• Contains raw OCR-extracted text files and their manually post-edited ground-truth for multiple books (BHS, GG, GOS).
• dev-transliterate-scripts/
• Utilities (C++ programs) to transliterate between Devanagari and SLP1.
• train-scripts/
• Model training (train_ByT5.py, train_BART.py, CopyNet folder) and inference (inference-byt5.py, inference-bart.py) code.
• preds/
• Model prediction CSVs on the out-of-domain test set.
• CSV split files (train_devnagari.csv, val_devnagari.csv, test_devnagari.csv, ood-test.csv)
• Evaluation scripts (metrics.py, word_count_cer.py) and dependencies (fastwer).
README Highlights
• Describes dataset splits, OOD test set, transliteration utilities, training scripts, and evaluation procedures (CER/WER calculation).
a. Main Components
• Data Sources
– OCR text folder (OCR-Images-Annotation)
– CSV splits for in-domain and OOD test sets
• Preprocessing & Utilities
– Transliteration module (Dev ↔ SLP1)
– Data cleaning/splitting (CSV generation)
• Model Training
– BART pipeline
– ByT5 pipeline
– CopyNet pipeline
• Inference
– Scripts to generate predictions on OOD test set
• Evaluation
– metrics.py / word_count_cer.py using fastwer
• Outputs
– preds/*.csv
b. Relationships & Data Flow
1. Raw OCR files → (Preprocessing) → CSV datasets → (Transliteration as needed)
2. CSV datasets → (Train-scripts) → Trained models
3. Trained models + OOD CSV → (Inference) → Predictions (preds/)
4. Predictions + Ground-truth → (Evaluation scripts) → CER/WER results
c. Architectural Patterns
• Linear data-processing pipeline (ETL → training → inference → evaluation)
• Modular codebase: separate scripts for transliteration, training, inference, and evaluation
d. Key Technologies
• Python (pandas, PyTorch/Transformers)
• C++ transliteration utilities
• fastwer for error-rate metrics
• CSV/text files and folder-based annotation structure