# darthlp/caosdataextraction architecture

> Architecture diagram of the GitHub repository darthlp/caosdataextraction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darthlp/caosdataextraction
- Repository: https://github.com/darthlp/caosdataextraction

## Overview

1. This repo is a batch document-processing pipeline for Dutch CAO PDFs. The core architecture is a five-stage flow: web scraping → PDF text extraction → LLM extraction → schema-based analysis → Excel generation.

2. The main runtime entry points are `run_pipeline.py` and the individual stage modules in `pipelines/`. The README notes `run_pipeline.py` currently references a missing `pipelines.p5_run`, so the practical execution model is running stages separately.

3. Stage 1 (`p1_webscraping.py`) is the ingestion boundary. It uses Selenium/Chrome to crawl the Dutch government CAO site, collect PDF links, merge related PDFs per CAO page, and write them into CAO-numbered input folders. It also consults Excel-based frequency/skip metadata to avoid duplicate work.

4. Stage 2 (`p2_extract.py`) turns PDFs into machine-readable text. It is a hybrid extractor that combines PyPDF2, pdfplumber, and Tesseract OCR, with logic to detect when OCR is needed. This stage also normalizes Unicode glyph patterns and emits parsed outputs under `outputs/parsed_pdfs/`.

5. Stage 3 (`p3_llmExtraction.py`) sends parsed content to Google Gemini for raw extraction. It preserves context in markdown form, supports multiple API keys for parallelism, and has retry logic tuned for API failures, quotas, and large-output truncation. Successful and failed cases are tracked per CAO, with failed CAOs cached so future runs can skip them.

6. Stage 4 (`p4_analysis.py`) is the main structured-extraction layer. It takes LLM output and maps it into Pydantic schemas split into salary and non-salary domains. This stage is the most architecture-heavy: it has adaptive retries, schema simplification when outputs are too large, and special handling for truncated files via compact and super-compact schema variants.

7. The schema layer in `schema/` is central to the design. Salary and non-salary models define the contract between Gemini output and downstream files, while `excel_output_schema.py` defines the final tabular shape. The presence of multiple salary schema variants indicates an intentional size/complexity fallback path.

8. Stage 5 (`p5_excel_creation.py`) is the delivery boundary. It merges salary and non-salary analysis results, enriches them with CAO metadata, and writes formatted Excel files. This stage also handles Excel-specific limits like cell size and date normalization.

9. Data is organized by CAO number, not just by filename. That is an important architectural boundary because the same PDF name can exist in multiple CAO folders. Inputs, parsed PDFs, LLM outputs, validation results, and logs are all partitioned to support idempotent batch processing.

10. `monitoring/` is a cross-cutting subsystem for observability. It records performance, token usage, costs, truncation events, and failed CAO numbers. The log folders encode retry state and are used as operational checkpoints between runs.

11. `scripts_pipeline_helper/` contains stage-specific support code that is effectively part of the pipeline’s control plane. It includes shared retry classification, backoff logic, merge/split handling, and output tracking. These helpers are what make the p3/p4 retry behavior deterministic and resumable.

12. `scripts/` and `tests/` form the offline quality and maintenance layer. Scripts cover extraction analysis, validation against source markdown, duplicate cleanup, Unicode repair, and PDF screening. Tests focus mainly on p4 analysis behavior, including retry logic and integration, which reflects where the most complex branching exists.

13. External dependencies materially shape the architecture: Selenium/Chrome for scraping, OCR tooling for scan-heavy PDFs, and Google Gemini for LLM-based extraction. The pipeline is built for long-running, quota-constrained batch processing, so parallel execution, locking, and resume logic are core design features rather than extras.

Main entry points: [`pipelines/p4_analysis.py`](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p4_analysis.py) (Schema analysis), [`pipelines/p3_llmExtraction.py`](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p3_llmExtraction.py) (LLM extraction), [`pipelines/p1_webscraping.py`](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p1_webscraping.py) (Web scraping), [`pipelines/p2_extract.py`](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p2_extract.py) (PDF extract)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Pipeline stages"]
  node_n1["Run pipeline<br/>entrypoint<br/>[run_pipeline.py]"]
  node_n2["Web scraping<br/>stage p1<br/>[p1_webscraping.py]"]
  node_n3["PDF extract<br/>stage p2<br/>[p2_extract.py]"]
  node_n4["LLM extraction<br/>stage p3"]
  node_n5["Schema analysis<br/>stage p4<br/>[p4_analysis.py]"]
  node_n6["Excel creation<br/>stage p5"]
end

subgraph group_g2["Schema layer"]
  node_n7["Salary schema<br/>pydantic model<br/>[salary_schema.py]"]
  node_n8["Compact salary<br/>fallback schema"]
  node_n9["Super compact<br/>fallback schema"]
  node_n10["Split salary<br/>schema split"]
  node_n11["Salary prompt split<br/>prompt partition"]
  node_n12["Non-salary schema<br/>pydantic model"]
  node_n13["Excel schema<br/>tabular schema"]
end

subgraph group_g3["Control plane"]
  node_n14{{"Retry logic<br/>helpers<br/>[retry_logic.py]"}}
  node_n15{{"Error classify<br/>helpers"}}
  node_n16{{"P4 retry<br/>helpers<br/>[retry_logic.py]"}}
  node_n17(("Monitoring<br/>observability<br/>[monitoring_3_1.py]"))
  node_n18{{"Resume utils<br/>state mgmt<br/>[quota_resume.py]"}}
end

subgraph group_g4["Maintenance"]
  node_n19["Validation<br/>offline QA"]
  node_n20["P4 tests<br/>test suite"]
end

node_n1 -->|"orchestrates"| node_n2
node_n2 -->|"feeds PDFs"| node_n3
node_n3 -->|"parsed text"| node_n4
node_n4 -->|"LLM output"| node_n5
node_n5 -->|"structured results"| node_n6
node_n2 -.->|"checkpointing"| node_n18
node_n3 -.->|"logs"| node_n17
node_n4 -.->|"retries"| node_n14
node_n4 -.->|"classifies failures"| node_n15
node_n5 -.->|"adaptive retries"| node_n16
node_n5 -->|"primary salary"| node_n7
node_n5 -->|"non-salary"| node_n12
node_n5 -.->|"fallback"| node_n8
node_n5 -.->|"fallback"| node_n9
node_n5 -.->|"split flow"| node_n10
node_n10 -.->|"prompting"| node_n11
node_n6 -->|"formats rows"| node_n13
node_n19 -.->|"validates"| node_n5
node_n20 -.->|"covers"| node_n5

click node_n1 "https://github.com/darthlp/caosdataextraction/blob/main/run_pipeline.py"
click node_n2 "https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p1_webscraping.py"
click node_n3 "https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p2_extract.py"
click node_n4 "https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p3_llmExtraction.py"
click node_n5 "https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p4_analysis.py"
click node_n6 "https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p5_excel_creation.py"
click node_n7 "https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema.py"
click node_n8 "https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_compact.py"
click node_n9 "https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_super_compact.py"
click node_n10 "https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_split.py"
click node_n11 "https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_prompt_split.py"
click node_n12 "https://github.com/darthlp/caosdataextraction/blob/main/schema/non_salary_schema.py"
click node_n13 "https://github.com/darthlp/caosdataextraction/blob/main/schema/excel_output_schema.py"
click node_n14 "https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p3/retry_logic.py"
click node_n15 "https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p3_p4/retry_error_classification.py"
click node_n16 "https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p4/retry_logic.py"
click node_n17 "https://github.com/darthlp/caosdataextraction/blob/main/monitoring/monitoring_3_1.py"
click node_n18 "https://github.com/darthlp/caosdataextraction/blob/main/utils/quota_resume.py"
click node_n19 "https://github.com/darthlp/caosdataextraction/blob/main/scripts/validation/validate_extraction.py"
click node_n20 "https://github.com/darthlp/caosdataextraction/blob/main/tests/test_p4_analysis_integration.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3,node_n4,node_n5,node_n6 toneBlue
class node_n7,node_n8,node_n9,node_n10,node_n11,node_n12,node_n13 toneAmber
class node_n14,node_n15,node_n16,node_n17,node_n18 toneMint
class node_n19,node_n20 toneRose
```

## Components

### Pipeline stages

Core batch flow from scraping to Excel delivery.

- [Run pipeline](https://github.com/darthlp/caosdataextraction/blob/main/run_pipeline.py): entrypoint · Top-level orchestrator; README notes one stage reference is missing, so stages are also run individually. · `run_pipeline.py`
- [Web scraping](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p1_webscraping.py): stage p1 · Crawls Dutch government CAO pages, collects PDF links, merges related PDFs, and writes CAO-scoped input folders. · `pipelines/p1_webscraping.py`
- [PDF extract](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p2_extract.py): stage p2 · Hybrid text extraction with PyPDF2, pdfplumber, and OCR; normalizes glyphs and writes parsed PDFs. · `pipelines/p2_extract.py`
- [LLM extraction](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p3_llmExtraction.py): stage p3 · Sends parsed content to Gemini, manages parallel keys, retries, truncation handling, and failed-CAO caching. · `pipelines/p3_llmExtraction.py`
- [Schema analysis](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p4_analysis.py): stage p4 · Maps LLM output into structured salary and non-salary schemas with adaptive retries and size fallbacks. · `pipelines/p4_analysis.py`
- [Excel creation](https://github.com/darthlp/caosdataextraction/blob/main/pipelines/p5_excel_creation.py): stage p5 · Merges analysis results, enriches CAO metadata, and writes formatted Excel outputs with cell/date handling. · `pipelines/p5_excel_creation.py`

### Schema layer

Pydantic contracts and fallbacks used by analysis and output generation.

- [Salary schema](https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema.py): pydantic model · Primary salary extraction contract used by Gemini output parsing and downstream merging. · `schema/salary_schema.py`
- [Compact salary](https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_compact.py): fallback schema · Reduced salary schema for oversized or truncated extractions. · `schema/salary_schema_compact.py`
- [Super compact](https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_super_compact.py): fallback schema · Minimal salary schema variant for extreme size constraints. · `schema/salary_schema_super_compact.py`
- [Split salary](https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_schema_split.py): schema split · Salary schema partitioning used to manage prompt and output size. · `schema/salary_schema_split.py`
- [Salary prompt split](https://github.com/darthlp/caosdataextraction/blob/main/schema/salary_prompt_split.py): prompt partition · Prompt-side splitting logic aligned with salary schema decomposition. · `schema/salary_prompt_split.py`
- [Non-salary schema](https://github.com/darthlp/caosdataextraction/blob/main/schema/non_salary_schema.py): pydantic model · Structured contract for non-salary extraction fields. · `schema/non_salary_schema.py`
- [Excel schema](https://github.com/darthlp/caosdataextraction/blob/main/schema/excel_output_schema.py): tabular schema · Final Excel row/column shape used by the delivery stage. · `schema/excel_output_schema.py`

### Control plane

Shared retry, logging, resume, and monitoring support.

- [Retry logic](https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p3/retry_logic.py): helpers · Shared retry and backoff behavior for extraction stages. · `scripts_pipeline_helper/p3/retry_logic.py`
- [Error classify](https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p3_p4/retry_error_classification.py): helpers · Classifies retryable failures and output issues for p3/p4. · `scripts_pipeline_helper/p3_p4/retry_error_classification.py`
- [P4 retry](https://github.com/darthlp/caosdataextraction/blob/main/scripts_pipeline_helper/p4/retry_logic.py): helpers · Stage-4 specific retry orchestration and simplification flow. · `scripts_pipeline_helper/p4/retry_logic.py`
- [Monitoring](https://github.com/darthlp/caosdataextraction/blob/main/monitoring/monitoring_3_1.py): observability · Tracks performance, token usage, costs, truncation events, and failed CAOs. · `monitoring/monitoring_3_1.py`
- [Resume utils](https://github.com/darthlp/caosdataextraction/blob/main/utils/quota_resume.py): state mgmt · Batch resume and checkpoint helpers for long-running, quota-constrained runs. · `utils/quota_resume.py`

### Maintenance

Offline analysis, validation, cleanup, and tests.

- [Validation](https://github.com/darthlp/caosdataextraction/blob/main/scripts/validation/validate_extraction.py): offline QA · Validates extraction against source markdown and prompts. · `scripts/validation/validate_extraction.py`
- [P4 tests](https://github.com/darthlp/caosdataextraction/blob/main/tests/test_p4_analysis_integration.py): test suite · Covers schema analysis behavior, including retry and integration paths. · `tests/test_p4_analysis_integration.py`

## Connections

- Run pipeline → Web scraping: orchestrates · Pipeline entry can invoke the scraping stage.
- Web scraping → PDF extract: feeds PDFs · Scraped PDFs are handed to text extraction.
- PDF extract → LLM extraction: parsed text · Machine-readable text is sent to Gemini extraction.
- LLM extraction → Schema analysis: LLM output · Raw Gemini output is converted into structured analysis.
- Schema analysis → Excel creation: structured results · Analysis outputs are merged into final Excel delivery.
- Web scraping → Resume utils: checkpointing · Scraping work is organized for resumable batch execution.
- PDF extract → Monitoring: logs · Extraction emits operational metrics and performance records.
- LLM extraction → Retry logic: retries · LLM extraction uses shared retry and backoff behavior.
- LLM extraction → Error classify: classifies failures · Failure categories drive retry, skip, and truncation handling.
- Schema analysis → P4 retry: adaptive retries · Schema analysis uses dedicated retry and simplification logic.
- Schema analysis → Salary schema: primary salary · Main analysis path targets the salary schema.
- Schema analysis → Non-salary schema: non-salary · Parallel analysis path targets non-salary extraction.
- Schema analysis → Compact salary: fallback · Oversized outputs can fall back to a compact salary schema.
- Schema analysis → Super compact: fallback · Extreme truncation can fall back to a super-compact salary schema.
- Schema analysis → Split salary: split flow · Schema splitting supports large salary extractions.
- Split salary → Salary prompt split: prompting · Prompt partitioning mirrors schema partitioning.
- Excel creation → Excel schema: formats rows · Delivery stage targets the Excel output contract.
- Validation → Schema analysis: validates · Offline validation checks structured analysis quality.
- P4 tests → Schema analysis: covers · Tests focus on p4 analysis and retry behavior.
