How darthlp/caosdataextraction works
1. This repo is a batch document-processing pipeline for Dutch CAO PDFs. The core architecture is a five-stage flow: web scraping → PDF text extraction → LLM extraction → schema-based analysis → Excel generation.
2. The main runtime entry points are run_pipeline.py and the individual stage modules in pipelines/. The README notes run_pipeline.py currently references a missing pipelines.p5_run, so the practical execution model is running stages separately.
3. Stage 1 (p1_webscraping.py) is the ingestion boundary. It uses Selenium/Chrome to crawl the Dutch government CAO site, collect PDF links, merge related PDFs per CAO page, and write them into CAO-numbered input folders. It also consults Excel-based frequency/skip metadata to avoid duplicate work.
4. Stage 2 (p2_extract.py) turns PDFs into machine-readable text. It is a hybrid extractor that combines PyPDF2, pdfplumber, and Tesseract OCR, with logic to detect when OCR is needed. This stage also normalizes Unicode glyph patterns and emits parsed outputs under outputs/parsed_pdfs/.
5. Stage 3 (p3_llmExtraction.py) sends parsed content to Google Gemini for raw extraction. It preserves context in markdown form, supports multiple API keys for parallelism, and has retry logic tuned for API failures, quotas, and large-output truncation. Successful and failed cases are tracked per CAO, with failed CAOs cached so future runs can skip them.
6. Stage 4 (p4_analysis.py) is the main structured-extraction layer. It takes LLM output and maps it into Pydantic schemas split into salary and non-salary domains. This stage is the most architecture-heavy: it has adaptive retries, schema simplification when outputs are too large, and special handling for truncated files via compact and super-compact schema variants.
7. The schema layer in schema/ is central to the design. Salary and non-salary models define the contract between Gemini output and downstream files, while excel_output_schema.py defines the final tabular shape. The presence of multiple salary schema variants indicates an intentional size/complexity fallback path.
8. Stage 5 (p5_excel_creation.py) is the delivery boundary. It merges salary and non-salary analysis results, enriches them with CAO metadata, and writes formatted Excel files. This stage also handles Excel-specific limits like cell size and date normalization.
9. Data is organized by CAO number, not just by filename. That is an important architectural boundary because the same PDF name can exist in multiple CAO folders. Inputs, parsed PDFs, LLM outputs, validation results, and logs are all partitioned to support idempotent batch processing.
10. monitoring/ is a cross-cutting subsystem for observability. It records performance, token usage, costs, truncation events, and failed CAO numbers. The log folders encode retry state and are used as operational checkpoints between runs.
11. scripts_pipeline_helper/ contains stage-specific support code that is effectively part of the pipeline’s control plane. It includes shared retry classification, backoff logic, merge/split handling, and output tracking. These helpers are what make the p3/p4 retry behavior deterministic and resumable.
12. scripts/ and tests/ form the offline quality and maintenance layer. Scripts cover extraction analysis, validation against source markdown, duplicate cleanup, Unicode repair, and PDF screening. Tests focus mainly on p4 analysis behavior, including retry logic and integration, which reflects where the most complex branching exists.
13. External dependencies materially shape the architecture: Selenium/Chrome for scraping, OCR tooling for scan-heavy PDFs, and Google Gemini for LLM-based extraction. The pipeline is built for long-running, quota-constrained batch processing, so parallel execution, locking, and resume logic are core design features rather than extras.
Main entry points: p4_analysis.py Schema analysis · p3_llmExtraction.py LLM extraction · p1_webscraping.py Web scraping · p2_extract.py PDF extract
Components 20
Pipeline stages
Core batch flow from scraping to Excel delivery.
- Run pipeline · Top-level orchestrator; README notes one stage reference is missing, so stages are also run individually.
run_pipeline.py - Web scraping · Crawls Dutch government CAO pages, collects PDF links, merges related PDFs, and writes CAO-scoped input folders.
pipelines/p1_webscraping.py - PDF extract · Hybrid text extraction with PyPDF2, pdfplumber, and OCR; normalizes glyphs and writes parsed PDFs.
pipelines/p2_extract.py - LLM extraction · Sends parsed content to Gemini, manages parallel keys, retries, truncation handling, and failed-CAO caching.
pipelines/p3_llmExtraction.py - Schema analysis · Maps LLM output into structured salary and non-salary schemas with adaptive retries and size fallbacks.
pipelines/p4_analysis.py - Excel creation · Merges analysis results, enriches CAO metadata, and writes formatted Excel outputs with cell/date handling.
pipelines/p5_excel_creation.py
Schema layer
Pydantic contracts and fallbacks used by analysis and output generation.
- Salary schema · Primary salary extraction contract used by Gemini output parsing and downstream merging.
schema/salary_schema.py - Compact salary · Reduced salary schema for oversized or truncated extractions.
schema/salary_schema_compact.py - Super compact · Minimal salary schema variant for extreme size constraints.
schema/salary_schema_super_compact.py - Split salary · Salary schema partitioning used to manage prompt and output size.
schema/salary_schema_split.py - Salary prompt split · Prompt-side splitting logic aligned with salary schema decomposition.
schema/salary_prompt_split.py - Non-salary schema · Structured contract for non-salary extraction fields.
schema/non_salary_schema.py - Excel schema · Final Excel row/column shape used by the delivery stage.
schema/excel_output_schema.py
Control plane
Shared retry, logging, resume, and monitoring support.
- Retry logic · Shared retry and backoff behavior for extraction stages.
scripts_pipeline_helper/p3/retry_logic.py - Error classify · Classifies retryable failures and output issues for p3/p4.
scripts_pipeline_helper/p3_p4/retry_error_classification.py - P4 retry · Stage-4 specific retry orchestration and simplification flow.
scripts_pipeline_helper/p4/retry_logic.py - Monitoring · Tracks performance, token usage, costs, truncation events, and failed CAOs.
monitoring/monitoring_3_1.py - Resume utils · Batch resume and checkpoint helpers for long-running, quota-constrained runs.
utils/quota_resume.py
Maintenance
Offline analysis, validation, cleanup, and tests.
- Validation · Validates extraction against source markdown and prompts.
scripts/validation/validate_extraction.py - P4 tests · Covers schema analysis behavior, including retry and integration paths.
tests/test_p4_analysis_integration.py
Connections 19
- Run pipeline → Web scraping: orchestrates · Pipeline entry can invoke the scraping stage.
- Web scraping → PDF extract: feeds PDFs · Scraped PDFs are handed to text extraction.
- PDF extract → LLM extraction: parsed text · Machine-readable text is sent to Gemini extraction.
- LLM extraction → Schema analysis: LLM output · Raw Gemini output is converted into structured analysis.
- Schema analysis → Excel creation: structured results · Analysis outputs are merged into final Excel delivery.
- Web scraping → Resume utils: checkpointing · Scraping work is organized for resumable batch execution.
- PDF extract → Monitoring: logs · Extraction emits operational metrics and performance records.
- LLM extraction → Retry logic: retries · LLM extraction uses shared retry and backoff behavior.
- LLM extraction → Error classify: classifies failures · Failure categories drive retry, skip, and truncation handling.
- Schema analysis → P4 retry: adaptive retries · Schema analysis uses dedicated retry and simplification logic.
- Schema analysis → Salary schema: primary salary · Main analysis path targets the salary schema.
- Schema analysis → Non-salary schema: non-salary · Parallel analysis path targets non-salary extraction.
- Schema analysis → Compact salary: fallback · Oversized outputs can fall back to a compact salary schema.
- Schema analysis → Super compact: fallback · Extreme truncation can fall back to a super-compact salary schema.
- Schema analysis → Split salary: split flow · Schema splitting supports large salary extractions.
- Split salary → Salary prompt split: prompting · Prompt partitioning mirrors schema partitioning.
- Excel creation → Excel schema: formats rows · Delivery stage targets the Excel output contract.
- Validation → Schema analysis: validates · Offline validation checks structured analysis quality.
- P4 tests → Schema analysis: covers · Tests focus on p4 analysis and retry behavior.