faizancyber007/document-parser

Purpose and Entry Points

pak_legal_parser converts Pakistani bilingual legal PDFs (Urdu Nastaliq/English) into provenance-rich structured JSON: physical blocks, OCR text, legal sections, metadata, corrections, and citation links. The primary single-document entry point is main.py; batch execution is run_batch.py. Inspection/reprocessing CLIs are run_triage.py, run_segmentation.py, and run_self_correction.py.

An optional HTTP façade accepts asynchronous parse jobs and reports/pushes results: api/app.py, api/jobs.py, and api/webhooks.py.

Pipeline Orchestration and Data Contract

main.py coordinates phase outputs through shared Pydantic schemas, chiefly schemas/data_models.py and schemas/profile_models.py. Configuration and secrets are centralized in config.py and config.yaml.

The core durability boundary is between immutable-ish physical extraction (raw_blocks) and derived semantic results (sections, metadata, citation graph). Semantic failures should not discard OCR/layout output. Blocks carry IDs, page coordinates, script, OCR source/confidence, errors, and optional inline Markdown.

Phase 0–1.5: Profile, Layout, Segment

Triage classifies document type, page script mix, scan status, and quality before expensive work: modules/triage.py. This selects a taxonomy/profile rather than assuming every input is a judgment; label sets live in schemas/taxonomies.py.

Layout extraction is split by page type in modules/layout_engine.py: native/digital PDFs use PyMuPDF text, tables, coordinates, and font-span formatting; scanned pages are rasterized and use lazily loaded DocLayout-YOLO. This is the primary licensing-sensitive boundary because both dependencies are AGPL/commercial.

modules/segmentation.py assigns structural roles—titles, lists, tables, headers/footers, footnotes—and establishes reading order and OCR routing intent. It prevents repeating boilerplate and footnotes from contaminating legal semantic tagging.

Phase 2: OCR Routing and Provider Resilience

modules/ocr_router.py executes the routing plan. Native digital text can remain untouched; non-native blocks first use local Tesseract, then escalate Urdu, tables, and low-confidence results to vision providers.

Provider abstraction, ordered fallbacks, and optional local GPU vision are in utils/api_clients.py. External services include Gemini, OpenRouter, Groq, and Hugging Face inference; local engines include Tesseract and EasyOCR (utils/easyocr_client.py). Image crop/preprocessing policy resides at utils/image_utils.py.

Cloud reliability is a cross-cutting boundary: utils/rate_limiter.py enforces per-minute limits and utils/usage_tracker.py persists usage and proactively skips daily-exhausted providers.

Phase 3–3.7: Legal Enrichment

modules/semantic_merger.py sends eligible blocks through the text-provider fallback chain, classifies them into document-specific legal sections, merges same-page geometry, renders Markdown, and creates retrieval chunks where needed. It references source block IDs rather than rewriting raw OCR.

Independent enrichments operate over blocks: modules/metadata_extractor.py produces confidence-scored case details; modules/self_correction.py runs deterministic defect detectors, permits LLM repair only when the original detector clears, and otherwise emits human-review work; modules/citation_graph.py uses regex/deterministic matching for case law, statutes, and footnote-reference edges.

Chart and formula paths are architecturally available for specialized blocks in modules/chart_extractor.py and modules/formula_extractor.py.

Batch and API Execution Models

modules/batch_orchestrator.py supports bounded-concurrent batches, content-hash caching, resumable checkpoints, per-document error isolation, and usage reporting. Persistence shapes are defined in schemas/batch_models.py.

The API intentionally uses process-local execution: api/jobs.py maintains an in-memory job store and one background thread per job; no Redis, broker, or distributed worker is present. api/webhooks.py provides best-effort HMAC-SHA256 signed callbacks, while polling remains the durable result access method within the running process.

Runtime and Artifacts

The target runtime is Python 3.10+ on a CPU-only laptop, with optional CUDA/local-GPU vision. Required local infrastructure includes the Tesseract binary and language packs. Runtime outputs default under output/; real judgment inputs and benchmark artifacts are in Judgements/, LlamaParse_Files/, and reference_baselines/.