How sarahgoel/indic-nlp-challenge works
Indic NLP Pipeline processes scanned manuscript PDFs into page images, OCR text, normalized text, translations, and DOCX exports. The main user workflow is exposed by the complete pipeline runner; README also documents standalone OCR export and translation, plus a separate normalization/segmentation pilot. The graph preserves OCR alternatives, dataset construction, and NLP stages where indexed source files establish their presence. Some relationships are documented or visible in excerpts; unsampled implementation details are not inferred.
Main entry points: run_pipeline.py Pipeline Runner · 18_translate_and_export.py Translation DOCX Export · production_runner.py Production OCR Runner · corpus_builder.py Corpus Builder
Components 25
Ingest and OCR
- Pipeline Runner
scripts/run_pipeline.py - Page Images
- Image Preparation
src/indicnlp_pipeline/preprocessing/image_prep.py - Production OCR Runner
src/indicnlp_pipeline/ocr/production_runner.py - Tesseract OCR
src/indicnlp_pipeline/ocr/tesseract_engine.py - Paddle OCR
src/indicnlp_pipeline/ocr/paddle_engine.py - OCR Comparison
src/indicnlp_pipeline/ocr/ocr_comparator.py - OCR Records
Text Processing
- Unicode Normalizer
src/indicnlp_pipeline/normalization/unicode_normalizer.py - Language Identifier
src/indicnlp_pipeline/normalization/language_identifier.py - Sentence Segmenter
src/indicnlp_pipeline/segmentation/sentence_segmenter.py - Unicode Tokenizer
src/indicnlp_pipeline/segmentation/tokenizer.py
Translation and Export
- Translation Runner
src/indicnlp_pipeline/translation/translation_runner.py - IndicTrans Engine
src/indicnlp_pipeline/translation/indictrans_engine.py - Gemini Engine
src/indicnlp_pipeline/translation/gemini_engine.py - Translation Records
- OCR DOCX Export
scripts/17_export_ocr_docx.py - Translation DOCX Export
scripts/18_translate_and_export.py
Dataset Operations
- Corpus Builder
src/indicnlp_pipeline/utils/corpus_builder.py - Canonical Dataset
src/indicnlp_pipeline/utils/canonical_dataset.py - Ground Truth
data/ground_truth/ground_truth.jsonl
Shared Support
- Pipeline Configuration
src/indicnlp_pipeline/utils/config_loader.py - Logging
src/indicnlp_pipeline/utils/logger.py
Other components
- Manuscript User
- PDF Manuscript
Connections 21
- Manuscript User → Pipeline Runner: runs
README.md - PDF Manuscript → Pipeline Runner: supplies PDF
README.md - Pipeline Runner → Page Images: extracts pages
README.md - Pipeline Runner → Tesseract OCR: runs OCR
scripts/run_pipeline.py - Production OCR Runner → Image Preparation: uses preprocessing
src/indicnlp_pipeline/ocr/production_runner.py - Tesseract OCR → OCR Records: returns OCR text
src/indicnlp_pipeline/ocr/tesseract_engine.py - Pipeline Runner → Unicode Normalizer: normalizes text
scripts/run_pipeline.py - Pipeline Runner → IndicTrans Engine: translates text
scripts/run_pipeline.py - Pipeline Runner → Translation DOCX Export: exports result
README.md - Manuscript User → OCR DOCX Export: requests OCR export
README.md - OCR Records → OCR DOCX Export: reads OCR records
scripts/17_export_ocr_docx.py - Manuscript User → Translation DOCX Export: requests translation
README.md - Translation DOCX Export → IndicTrans Engine: uses translation engine
scripts/18_translate_and_export.py - Translation DOCX Export → Translation Records: writes translation
scripts/18_translate_and_export.py - Sentence Segmenter → Unicode Tokenizer: provides segments
src/indicnlp_pipeline/segmentation/tokenizer.py - Production OCR Runner → OCR Comparison
- Paddle OCR → OCR Comparison
- Translation Runner → Gemini Engine
- Corpus Builder → Ground Truth
- Canonical Dataset → Production OCR Runner: references OCR provenance
src/indicnlp_pipeline/utils/canonical_dataset.py - Corpus Builder → Logging: uses logging
src/indicnlp_pipeline/utils/corpus_builder.py