# faizancyber007/document-parser architecture

> Architecture diagram of the GitHub repository faizancyber007/document-parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/faizancyber007/document-parser
- Repository: https://github.com/faizancyber007/document-parser

## Overview

#### Purpose and Entry Points
`pak_legal_parser` converts Pakistani bilingual legal PDFs (Urdu Nastaliq/English) into provenance-rich structured JSON: physical blocks, OCR text, legal sections, metadata, corrections, and citation links. The primary single-document entry point is `main.py`; batch execution is `run_batch.py`. Inspection/reprocessing CLIs are `run_triage.py`, `run_segmentation.py`, and `run_self_correction.py`.

An optional HTTP façade accepts asynchronous parse jobs and reports/pushes results: `api/app.py`, `api/jobs.py`, and `api/webhooks.py`.

#### Pipeline Orchestration and Data Contract
`main.py` coordinates phase outputs through shared Pydantic schemas, chiefly `schemas/data_models.py` and `schemas/profile_models.py`. Configuration and secrets are centralized in `config.py` and `config.yaml`.

The core durability boundary is between immutable-ish physical extraction (`raw_blocks`) and derived semantic results (`sections`, metadata, citation graph). Semantic failures should not discard OCR/layout output. Blocks carry IDs, page coordinates, script, OCR source/confidence, errors, and optional inline Markdown.

#### Phase 0–1.5: Profile, Layout, Segment
Triage classifies document type, page script mix, scan status, and quality before expensive work: `modules/triage.py`. This selects a taxonomy/profile rather than assuming every input is a judgment; label sets live in `schemas/taxonomies.py`.

Layout extraction is split by page type in `modules/layout_engine.py`: native/digital PDFs use PyMuPDF text, tables, coordinates, and font-span formatting; scanned pages are rasterized and use lazily loaded DocLayout-YOLO. This is the primary licensing-sensitive boundary because both dependencies are AGPL/commercial.

`modules/segmentation.py` assigns structural roles—titles, lists, tables, headers/footers, footnotes—and establishes reading order and OCR routing intent. It prevents repeating boilerplate and footnotes from contaminating legal semantic tagging.

#### Phase 2: OCR Routing and Provider Resilience
`modules/ocr_router.py` executes the routing plan. Native digital text can remain untouched; non-native blocks first use local Tesseract, then escalate Urdu, tables, and low-confidence results to vision providers.

Provider abstraction, ordered fallbacks, and optional local GPU vision are in `utils/api_clients.py`. External services include Gemini, OpenRouter, Groq, and Hugging Face inference; local engines include Tesseract and EasyOCR (`utils/easyocr_client.py`). Image crop/preprocessing policy resides at `utils/image_utils.py`.

Cloud reliability is a cross-cutting boundary: `utils/rate_limiter.py` enforces per-minute limits and `utils/usage_tracker.py` persists usage and proactively skips daily-exhausted providers.

#### Phase 3–3.7: Legal Enrichment
`modules/semantic_merger.py` sends eligible blocks through the text-provider fallback chain, classifies them into document-specific legal sections, merges same-page geometry, renders Markdown, and creates retrieval chunks where needed. It references source block IDs rather than rewriting raw OCR.

Independent enrichments operate over blocks: `modules/metadata_extractor.py` produces confidence-scored case details; `modules/self_correction.py` runs deterministic defect detectors, permits LLM repair only when the original detector clears, and otherwise emits human-review work; `modules/citation_graph.py` uses regex/deterministic matching for case law, statutes, and footnote-reference edges.

Chart and formula paths are architecturally available for specialized blocks in `modules/chart_extractor.py` and `modules/formula_extractor.py`.

#### Batch and API Execution Models
`modules/batch_orchestrator.py` supports bounded-concurrent batches, content-hash caching, resumable checkpoints, per-document error isolation, and usage reporting. Persistence shapes are defined in `schemas/batch_models.py`.

The API intentionally uses process-local execution: `api/jobs.py` maintains an in-memory job store and one background thread per job; no Redis, broker, or distributed worker is present. `api/webhooks.py` provides best-effort HMAC-SHA256 signed callbacks, while polling remains the durable result access method within the running process.

#### Runtime and Artifacts
The target runtime is Python 3.10+ on a CPU-only laptop, with optional CUDA/local-GPU vision. Required local infrastructure includes the Tesseract binary and language packs. Runtime outputs default under `output/`; real judgment inputs and benchmark artifacts are in `Judgements/`, `LlamaParse_Files/`, and `reference_baselines/`.

Main entry points: [`main.py`](https://github.com/faizancyber007/document-parser/blob/main/main.py) (Single-document parser), [`api/jobs.py`](https://github.com/faizancyber007/document-parser/blob/main/api/jobs.py) (In-memory jobs), [`utils/api_clients.py`](https://github.com/faizancyber007/document-parser/blob/main/utils/api_clients.py) (Provider fallbacks), [`modules/batch_orchestrator.py`](https://github.com/faizancyber007/document-parser/blob/main/modules/batch_orchestrator.py) (Batch orchestrator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_execution["Execution"]
  node_inputs["Legal PDF inputs<br/>document corpus"]
  node_config["Configuration<br/>runtime config<br/>[config.py]"]
  node_main{{"Single-document parser<br/>pipeline orchestrator<br/>[main.py]"}}
  node_models["Shared data models<br/>Pydantic schemas<br/>[data_models.py]"]
  node_batch{{"Batch orchestrator<br/>resumable batch runner"}}
  node_batch_models["Batch models<br/>batch schemas<br/>[batch_models.py]"]
  node_output[("Output artifacts<br/>JSON artifacts")]
end

subgraph group_physical["Physical extraction"]
  node_triage["Document triage<br/>profile classifier<br/>[triage.py]"]
  node_layout["Layout extraction<br/>PDF and raster extractor<br/>[layout_engine.py]"]
  node_raw_blocks[("Raw blocks<br/>durable physical record")]
  node_segmentation["Structural segmentation<br/>reading-order planner<br/>[segmentation.py]"]
  node_ocr["OCR router<br/>OCR escalation<br/>[ocr_router.py]"]
  node_providers["Provider fallbacks<br/>vision provider client<br/>[api_clients.py]"]
  node_rate_limits["Rate limits<br/>cloud safeguard<br/>[rate_limiter.py]"]
end

subgraph group_semantic["Legal enrichment"]
  node_semantic_merger["Semantic merger<br/>section classifier<br/>[semantic_merger.py]"]
  node_metadata["Metadata extraction<br/>case detail extractor"]
  node_correction["Self-correction<br/>review-aware repair<br/>[self_correction.py]"]
  node_citations["Citation graph<br/>legal linker<br/>[citation_graph.py]"]
  node_semantic_results[("Derived legal results<br/>semantic JSON")]
end

subgraph group_api["HTTP façade"]
  node_api_app{{"HTTP API<br/>async parse façade<br/>[app.py]"}}
  node_jobs["In-memory jobs<br/>local job executor<br/>[jobs.py]"]
  node_webhooks["Signed webhooks<br/>best-effort callback<br/>[webhooks.py]"]
end

node_inputs -->|"parse"| node_main
node_config -.->|"configures"| node_main
node_models -.->|"contracts"| node_main
node_main -->|"phase 0"| node_triage
node_triage -->|"page strategy"| node_layout
node_layout -->|"physical extraction"| node_raw_blocks
node_raw_blocks -->|"blocks"| node_segmentation
node_segmentation -->|"routing plan"| node_ocr
node_ocr -->|"escalates"| node_providers
node_rate_limits -.->|"throttles"| node_providers
node_providers -->|"OCR text and confidence"| node_raw_blocks
node_raw_blocks -->|"source block IDs"| node_semantic_merger
node_raw_blocks -->|"block evidence"| node_metadata
node_raw_blocks -->|"detect defects"| node_correction
node_raw_blocks -->|"deterministic matching"| node_citations
node_semantic_merger -->|"sections and chunks"| node_semantic_results
node_metadata -->|"case details"| node_semantic_results
node_correction -->|"repairs and review work"| node_semantic_results
node_citations -->|"citation links"| node_semantic_results
node_main -->|"writes results"| node_output
node_semantic_results -->|"serialize"| node_output
node_batch -->|"per document"| node_main
node_batch_models -.->|"persistence shapes"| node_batch
node_api_app -->|"submits async parse"| node_jobs
node_jobs -->|"background thread"| node_main
node_jobs -.->|"completion callback"| node_webhooks

click node_config "https://github.com/faizancyber007/document-parser/blob/main/config.py"
click node_main "https://github.com/faizancyber007/document-parser/blob/main/main.py"
click node_models "https://github.com/faizancyber007/document-parser/blob/main/schemas/data_models.py"
click node_triage "https://github.com/faizancyber007/document-parser/blob/main/modules/triage.py"
click node_layout "https://github.com/faizancyber007/document-parser/blob/main/modules/layout_engine.py"
click node_segmentation "https://github.com/faizancyber007/document-parser/blob/main/modules/segmentation.py"
click node_ocr "https://github.com/faizancyber007/document-parser/blob/main/modules/ocr_router.py"
click node_providers "https://github.com/faizancyber007/document-parser/blob/main/utils/api_clients.py"
click node_rate_limits "https://github.com/faizancyber007/document-parser/blob/main/utils/rate_limiter.py"
click node_semantic_merger "https://github.com/faizancyber007/document-parser/blob/main/modules/semantic_merger.py"
click node_metadata "https://github.com/faizancyber007/document-parser/blob/main/modules/metadata_extractor.py"
click node_correction "https://github.com/faizancyber007/document-parser/blob/main/modules/self_correction.py"
click node_citations "https://github.com/faizancyber007/document-parser/blob/main/modules/citation_graph.py"
click node_batch "https://github.com/faizancyber007/document-parser/blob/main/modules/batch_orchestrator.py"
click node_batch_models "https://github.com/faizancyber007/document-parser/blob/main/schemas/batch_models.py"
click node_api_app "https://github.com/faizancyber007/document-parser/blob/main/api/app.py"
click node_jobs "https://github.com/faizancyber007/document-parser/blob/main/api/jobs.py"
click node_webhooks "https://github.com/faizancyber007/document-parser/blob/main/api/webhooks.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inputs,node_config,node_main,node_models,node_batch,node_batch_models,node_output toneBlue
class node_triage,node_layout,node_raw_blocks,node_segmentation,node_ocr,node_providers,node_rate_limits toneAmber
class node_semantic_merger,node_metadata,node_correction,node_citations,node_semantic_results toneMint
class node_api_app,node_jobs,node_webhooks toneRose
```

## Components

### Execution

- **Legal PDF inputs**: document corpus
- [Configuration](https://github.com/faizancyber007/document-parser/blob/main/config.py): runtime config · Centralizes settings and secrets. · `config.py`
- [Single-document parser](https://github.com/faizancyber007/document-parser/blob/main/main.py): pipeline orchestrator · Coordinates phase outputs without losing physical extraction. · `main.py`
- [Shared data models](https://github.com/faizancyber007/document-parser/blob/main/schemas/data_models.py): Pydantic schemas · Defines blocks, profiles, and derived result contracts. · `schemas/data_models.py`
- [Batch orchestrator](https://github.com/faizancyber007/document-parser/blob/main/modules/batch_orchestrator.py): resumable batch runner · Provides bounded concurrency, caching, checkpoints, and error isolation. · `modules/batch_orchestrator.py`
- [Batch models](https://github.com/faizancyber007/document-parser/blob/main/schemas/batch_models.py): batch schemas · `schemas/batch_models.py`
- **Output artifacts**: JSON artifacts · Default location for parse results and checkpoints.

### Physical extraction

- [Document triage](https://github.com/faizancyber007/document-parser/blob/main/modules/triage.py): profile classifier · Classifies type, scripts, scan state, and quality. · `modules/triage.py`
- [Layout extraction](https://github.com/faizancyber007/document-parser/blob/main/modules/layout_engine.py): PDF and raster extractor · Uses native PDF extraction or scanned-page layout detection. · `modules/layout_engine.py`
- **Raw blocks**: durable physical record · Immutable-ish OCR, coordinates, script, confidence, and errors.
- [Structural segmentation](https://github.com/faizancyber007/document-parser/blob/main/modules/segmentation.py): reading-order planner · Assigns roles and OCR routing intent. · `modules/segmentation.py`
- [OCR router](https://github.com/faizancyber007/document-parser/blob/main/modules/ocr_router.py): OCR escalation · Keeps native text or escalates difficult blocks. · `modules/ocr_router.py`
- [Provider fallbacks](https://github.com/faizancyber007/document-parser/blob/main/utils/api_clients.py): vision provider client · Routes across local and cloud OCR/vision providers. · `utils/api_clients.py`
- [Rate limits](https://github.com/faizancyber007/document-parser/blob/main/utils/rate_limiter.py): cloud safeguard · Enforces provider per-minute limits. · `utils/rate_limiter.py`

### Legal enrichment

- [Semantic merger](https://github.com/faizancyber007/document-parser/blob/main/modules/semantic_merger.py): section classifier · Builds legal sections, Markdown, and retrieval chunks from source block IDs. · `modules/semantic_merger.py`
- [Metadata extraction](https://github.com/faizancyber007/document-parser/blob/main/modules/metadata_extractor.py): case detail extractor · Produces confidence-scored case metadata. · `modules/metadata_extractor.py`
- [Self-correction](https://github.com/faizancyber007/document-parser/blob/main/modules/self_correction.py): review-aware repair · Uses deterministic detectors and emits human-review work when needed. · `modules/self_correction.py`
- [Citation graph](https://github.com/faizancyber007/document-parser/blob/main/modules/citation_graph.py): legal linker · Finds case law, statute, and footnote-reference links. · `modules/citation_graph.py`
- **Derived legal results**: semantic JSON · Sections, metadata, corrections, and citation links remain separate from raw blocks.

### HTTP façade

- [HTTP API](https://github.com/faizancyber007/document-parser/blob/main/api/app.py): async parse façade · `api/app.py`
- [In-memory jobs](https://github.com/faizancyber007/document-parser/blob/main/api/jobs.py): local job executor · Uses one background thread per job; no broker or distributed workers. · `api/jobs.py`
- [Signed webhooks](https://github.com/faizancyber007/document-parser/blob/main/api/webhooks.py): best-effort callback · Pushes HMAC-SHA256 signed completion callbacks. · `api/webhooks.py`

## Connections

- Legal PDF inputs → Single-document parser: parse
- Configuration → Single-document parser: configures
- Shared data models → Single-document parser: contracts
- Single-document parser → Document triage: phase 0
- Document triage → Layout extraction: page strategy
- Layout extraction → Raw blocks: physical extraction
- Raw blocks → Structural segmentation: blocks
- Structural segmentation → OCR router: routing plan
- OCR router → Provider fallbacks: escalates
- Rate limits → Provider fallbacks: throttles
- Provider fallbacks → Raw blocks: OCR text and confidence
- Raw blocks → Semantic merger: source block IDs
- Raw blocks → Metadata extraction: block evidence
- Raw blocks → Self-correction: detect defects
- Raw blocks → Citation graph: deterministic matching
- Semantic merger → Derived legal results: sections and chunks
- Metadata extraction → Derived legal results: case details
- Self-correction → Derived legal results: repairs and review work
- Citation graph → Derived legal results: citation links
- Single-document parser → Output artifacts: writes results
- Derived legal results → Output artifacts: serialize
- Batch orchestrator → Single-document parser: per document
- Batch models → Batch orchestrator: persistence shapes
- HTTP API → In-memory jobs: submits async parse
- In-memory jobs → Single-document parser: background thread
- In-memory jobs → Signed webhooks: completion callback