# sarahgoel/indic-nlp-challenge architecture

> Architecture diagram of the GitHub repository sarahgoel/indic-nlp-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarahgoel/indic-nlp-challenge
- Repository: https://github.com/sarahgoel/indic-nlp-challenge

## Overview

Indic NLP Pipeline processes scanned manuscript PDFs into page images, OCR text, normalized text, translations, and DOCX exports. The main user workflow is exposed by the complete pipeline runner; README also documents standalone OCR export and translation, plus a separate normalization/segmentation pilot. The graph preserves OCR alternatives, dataset construction, and NLP stages where indexed source files establish their presence. Some relationships are documented or visible in excerpts; unsampled implementation details are not inferred.

Main entry points: [`scripts/run_pipeline.py`](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/run_pipeline.py) (Pipeline Runner), [`scripts/18_translate_and_export.py`](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/18_translate_and_export.py) (Translation DOCX Export), [`src/indicnlp_pipeline/ocr/production_runner.py`](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/production_runner.py) (Production OCR Runner), [`src/indicnlp_pipeline/utils/corpus_builder.py`](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/corpus_builder.py) (Corpus Builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest_ocr["Ingest and OCR"]
  node_pipeline_runner["Pipeline Runner<br/>[run_pipeline.py]"]
  node_page_images["Page Images"]
  node_image_preparation["Image Preparation<br/>[image_prep.py]"]
  node_ocr_runner["Production OCR Runner"]
  node_tesseract["Tesseract OCR"]
  node_paddle["Paddle OCR<br/>[paddle_engine.py]"]
  node_ocr_comparison["OCR Comparison<br/>[ocr_comparator.py]"]
  node_ocr_records["OCR Records"]
end

subgraph group_text_processing["Text Processing"]
  node_normalizer["Unicode Normalizer"]
  node_language_id["Language Identifier"]
  node_segmenter["Sentence Segmenter"]
  node_tokenizer["Unicode Tokenizer<br/>[tokenizer.py]"]
end

subgraph group_translation_export["Translation and Export"]
  node_translation_runner["Translation Runner"]
  node_indictrans["IndicTrans Engine"]
  node_gemini["Gemini Engine<br/>[gemini_engine.py]"]
  node_translated_records["Translation Records"]
  node_ocr_docx["OCR DOCX Export"]
  node_translation_docx["Translation DOCX Export"]
end

subgraph group_dataset_ops["Dataset Operations"]
  node_corpus_builder["Corpus Builder<br/>[corpus_builder.py]"]
  node_canonical_dataset["Canonical Dataset"]
  node_ground_truth["Ground Truth<br/>[ground_truth.jsonl]"]
end

subgraph group_support["Shared Support"]
  node_config["Pipeline Configuration<br/>[config_loader.py]"]
  node_logger["Logging<br/>[logger.py]"]
end

node_user(("Manuscript User"))
node_pdf_input["PDF Manuscript"]

node_user -->|"runs"| node_pipeline_runner
node_pdf_input -->|"supplies PDF"| node_pipeline_runner
node_pipeline_runner -->|"extracts pages"| node_page_images
node_pipeline_runner -->|"runs OCR"| node_tesseract
node_ocr_runner -->|"uses preprocessing"| node_image_preparation
node_tesseract -->|"returns OCR text"| node_ocr_records
node_pipeline_runner -->|"normalizes text"| node_normalizer
node_pipeline_runner -->|"translates text"| node_indictrans
node_pipeline_runner -->|"exports result"| node_translation_docx
node_user -->|"requests OCR export"| node_ocr_docx
node_ocr_records -->|"reads OCR records"| node_ocr_docx
node_user -->|"requests translation"| node_translation_docx
node_translation_docx -->|"uses translation engine"| node_indictrans
node_translation_docx -->|"writes translation"| node_translated_records
node_segmenter -->|"provides segments"| node_tokenizer
node_ocr_runner -.-> node_ocr_comparison
node_paddle -.-> node_ocr_comparison
node_translation_runner -.-> node_gemini
node_corpus_builder -.-> node_ground_truth
node_canonical_dataset -->|"references OCR provenance"| node_ocr_runner
node_corpus_builder -->|"uses logging"| node_logger

click node_pipeline_runner "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/run_pipeline.py"
click node_image_preparation "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/preprocessing/image_prep.py"
click node_ocr_runner "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/production_runner.py"
click node_tesseract "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/tesseract_engine.py"
click node_paddle "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/paddle_engine.py"
click node_ocr_comparison "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/ocr_comparator.py"
click node_normalizer "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/normalization/unicode_normalizer.py"
click node_language_id "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/normalization/language_identifier.py"
click node_segmenter "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/segmentation/sentence_segmenter.py"
click node_tokenizer "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/segmentation/tokenizer.py"
click node_translation_runner "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/translation_runner.py"
click node_indictrans "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/indictrans_engine.py"
click node_gemini "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/gemini_engine.py"
click node_ocr_docx "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/17_export_ocr_docx.py"
click node_translation_docx "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/18_translate_and_export.py"
click node_corpus_builder "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/corpus_builder.py"
click node_canonical_dataset "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/canonical_dataset.py"
click node_ground_truth "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/data/ground_truth/ground_truth.jsonl"
click node_config "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/config_loader.py"
click node_logger "https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_runner,node_page_images,node_image_preparation,node_ocr_runner,node_tesseract,node_paddle,node_ocr_comparison,node_ocr_records,node_user toneBlue
class node_normalizer,node_language_id,node_segmenter,node_tokenizer toneAmber
class node_translation_runner,node_indictrans,node_gemini,node_translated_records,node_ocr_docx,node_translation_docx toneMint
class node_corpus_builder,node_canonical_dataset,node_ground_truth toneRose
class node_config,node_logger,node_pdf_input toneIndigo
```

## Components

### Ingest and OCR

- [Pipeline Runner](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/run_pipeline.py): `scripts/run_pipeline.py`
- **Page Images**
- [Image Preparation](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/preprocessing/image_prep.py): `src/indicnlp_pipeline/preprocessing/image_prep.py`
- [Production OCR Runner](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/production_runner.py): `src/indicnlp_pipeline/ocr/production_runner.py`
- [Tesseract OCR](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/tesseract_engine.py): `src/indicnlp_pipeline/ocr/tesseract_engine.py`
- [Paddle OCR](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/paddle_engine.py): `src/indicnlp_pipeline/ocr/paddle_engine.py`
- [OCR Comparison](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/ocr/ocr_comparator.py): `src/indicnlp_pipeline/ocr/ocr_comparator.py`
- **OCR Records**

### Text Processing

- [Unicode Normalizer](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/normalization/unicode_normalizer.py): `src/indicnlp_pipeline/normalization/unicode_normalizer.py`
- [Language Identifier](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/normalization/language_identifier.py): `src/indicnlp_pipeline/normalization/language_identifier.py`
- [Sentence Segmenter](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/segmentation/sentence_segmenter.py): `src/indicnlp_pipeline/segmentation/sentence_segmenter.py`
- [Unicode Tokenizer](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/segmentation/tokenizer.py): `src/indicnlp_pipeline/segmentation/tokenizer.py`

### Translation and Export

- [Translation Runner](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/translation_runner.py): `src/indicnlp_pipeline/translation/translation_runner.py`
- [IndicTrans Engine](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/indictrans_engine.py): `src/indicnlp_pipeline/translation/indictrans_engine.py`
- [Gemini Engine](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/translation/gemini_engine.py): `src/indicnlp_pipeline/translation/gemini_engine.py`
- **Translation Records**
- [OCR DOCX Export](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/17_export_ocr_docx.py): `scripts/17_export_ocr_docx.py`
- [Translation DOCX Export](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/scripts/18_translate_and_export.py): `scripts/18_translate_and_export.py`

### Dataset Operations

- [Corpus Builder](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/corpus_builder.py): `src/indicnlp_pipeline/utils/corpus_builder.py`
- [Canonical Dataset](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/canonical_dataset.py): `src/indicnlp_pipeline/utils/canonical_dataset.py`
- [Ground Truth](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/data/ground_truth/ground_truth.jsonl): `data/ground_truth/ground_truth.jsonl`

### Shared Support

- [Pipeline Configuration](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/config_loader.py): `src/indicnlp_pipeline/utils/config_loader.py`
- [Logging](https://github.com/sarahgoel/indic-nlp-challenge/blob/main/src/indicnlp_pipeline/utils/logger.py): `src/indicnlp_pipeline/utils/logger.py`

### Other components

- **Manuscript User**
- **PDF Manuscript**

## Connections

- Manuscript User → Pipeline Runner: runs (evidence: `README.md`)
- PDF Manuscript → Pipeline Runner: supplies PDF (evidence: `README.md`)
- Pipeline Runner → Page Images: extracts pages (evidence: `README.md`)
- Pipeline Runner → Tesseract OCR: runs OCR (evidence: `scripts/run_pipeline.py`)
- Production OCR Runner → Image Preparation: uses preprocessing (evidence: `src/indicnlp_pipeline/ocr/production_runner.py`)
- Tesseract OCR → OCR Records: returns OCR text (evidence: `src/indicnlp_pipeline/ocr/tesseract_engine.py`)
- Pipeline Runner → Unicode Normalizer: normalizes text (evidence: `scripts/run_pipeline.py`)
- Pipeline Runner → IndicTrans Engine: translates text (evidence: `scripts/run_pipeline.py`)
- Pipeline Runner → Translation DOCX Export: exports result (evidence: `README.md`)
- Manuscript User → OCR DOCX Export: requests OCR export (evidence: `README.md`)
- OCR Records → OCR DOCX Export: reads OCR records (evidence: `scripts/17_export_ocr_docx.py`)
- Manuscript User → Translation DOCX Export: requests translation (evidence: `README.md`)
- Translation DOCX Export → IndicTrans Engine: uses translation engine (evidence: `scripts/18_translate_and_export.py`)
- Translation DOCX Export → Translation Records: writes translation (evidence: `scripts/18_translate_and_export.py`)
- Sentence Segmenter → Unicode Tokenizer: provides segments (evidence: `src/indicnlp_pipeline/segmentation/tokenizer.py`)
- Production OCR Runner → OCR Comparison
- Paddle OCR → OCR Comparison
- Translation Runner → Gemini Engine
- Corpus Builder → Ground Truth
- Canonical Dataset → Production OCR Runner: references OCR provenance (evidence: `src/indicnlp_pipeline/utils/canonical_dataset.py`)
- Corpus Builder → Logging: uses logging (evidence: `src/indicnlp_pipeline/utils/corpus_builder.py`)