# safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents architecture

> Architecture diagram of the GitHub repository safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents
- Repository: https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents

## Overview

LEAP evaluates technical R&D documents by parsing them into section graphs, enriching section content, judging sections against retrieved standards, and aggregating section scores into an audit-readiness report. The README identifies graph construction, multi-agent evaluation, and iterative refinement as product capabilities; the sampled orchestrator excerpt confirms parallel section dispatch and aggregation. A separate dataset pipeline prepares graph-structured data and annotations. Some connections are documented only at the README or module-reference level; unsampled implementation details and the refinement loop are therefore not expanded into unverified runtime edges.

Main entry points: [`src/orchestration/pipeline.py`](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/orchestration/pipeline.py) (Pipeline orchestrator), [`src/dataset/pipeline.py`](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/pipeline.py) (Dataset pipeline), [`src/infrastructure/preprocessing/manager.py`](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/manager.py) (Node preprocessing), [`src/infrastructure/llm_judge.py`](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/llm_judge.py) (LLM section judge)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation workflow"]
  node_orchestrator["Pipeline orchestrator<br/>[pipeline.py]"]
  node_section_graph["Section graph<br/>[models.py]"]
  node_separation["Section separation<br/>[separation.py]"]
  node_evaluation_service["Section evaluation service<br/>[evaluation.py]"]
  node_aggregator["Report aggregation<br/>[aggregation.py]"]
  node_report["Evaluation report<br/>[models.py]"]
end

subgraph group_preprocessing["Document processing"]
  node_parser["Document graph parser<br/>[graph_parser.py]"]
  node_preprocessor["Node preprocessing<br/>[manager.py]"]
  node_handlers["Content handlers<br/>[handlers.py]"]
  node_ocr["OCR engine<br/>[ocr_engine.py]"]
  node_vlm["Vision model provider<br/>[vlm_provider.py]"]
end

subgraph group_judging["Section judging"]
  node_judge["LLM section judge<br/>[llm_judge.py]"]
  node_agent["Reasoning agent client<br/>[agent_client.py]"]
  node_llm_provider["LLM provider<br/>[llm_config.py]"]
  node_retrieval["Reference retrieval<br/>[retrieval.py]"]
  node_metrics["Audit metrics registry<br/>[metrics_library.py]"]
end

subgraph group_dataset["Dataset preparation"]
  node_dataset_pipeline["Dataset pipeline<br/>[pipeline.py]"]
  node_ingestion["Dataset ingestion<br/>[ingestion.py]"]
  node_graph_builder["Dataset graph builder<br/>[graph_builder.py]"]
  node_annotation["Sample annotation<br/>[annotation.py]"]
  node_augmentation["Dataset augmentation<br/>[augmentation.py]"]
  node_dataset_storage[("Dataset storage<br/>[storage.py]")]
end

node_operator(("Document operator"))
node_document["R&amp;D document"]

node_operator -->|"runs"| node_orchestrator
node_document -->|"submits"| node_orchestrator
node_orchestrator -->|"parses document"| node_parser
node_parser -->|"builds sections"| node_section_graph
node_orchestrator -->|"enriches nodes"| node_preprocessor
node_preprocessor -->|"normalizes content"| node_handlers
node_preprocessor -->|"extracts text"| node_ocr
node_preprocessor -->|"interprets visuals"| node_vlm
node_orchestrator -->|"separates sections"| node_separation
node_orchestrator -->|"dispatches sections"| node_evaluation_service
node_evaluation_service -->|"evaluates node"| node_judge
node_judge -->|"requests reasoning"| node_agent
node_agent -->|"sends model requests"| node_llm_provider
node_judge -->|"retrieves standards"| node_retrieval
node_judge -->|"scores criteria"| node_metrics
node_orchestrator -->|"aggregates results"| node_aggregator
node_aggregator -->|"produces report"| node_report
node_dataset_pipeline -->|"loads documents"| node_ingestion
node_dataset_pipeline -->|"builds graph"| node_graph_builder
node_dataset_pipeline -->|"annotates samples"| node_annotation
node_dataset_pipeline -->|"augments data"| node_augmentation
node_dataset_pipeline -->|"stores outputs"| node_dataset_storage

click node_orchestrator "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/orchestration/pipeline.py"
click node_parser "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/graph_parser.py"
click node_preprocessor "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/manager.py"
click node_handlers "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/handlers.py"
click node_ocr "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/ocr_engine.py"
click node_vlm "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/vlm_provider.py"
click node_section_graph "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/domain/models.py"
click node_separation "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/separation.py"
click node_evaluation_service "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/evaluation.py"
click node_judge "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/llm_judge.py"
click node_agent "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/agent_client.py"
click node_llm_provider "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/llm_config.py"
click node_retrieval "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/retrieval.py"
click node_metrics "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/metrics_library.py"
click node_aggregator "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/aggregation.py"
click node_report "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/domain/models.py"
click node_dataset_pipeline "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/pipeline.py"
click node_ingestion "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/ingestion.py"
click node_graph_builder "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/graph_builder.py"
click node_annotation "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/annotation.py"
click node_augmentation "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/augmentation.py"
click node_dataset_storage "https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_section_graph,node_separation,node_evaluation_service,node_aggregator,node_report toneBlue
class node_parser,node_preprocessor,node_handlers,node_ocr,node_vlm toneAmber
class node_judge,node_agent,node_llm_provider,node_retrieval,node_metrics toneMint
class node_dataset_pipeline,node_ingestion,node_graph_builder,node_annotation,node_augmentation,node_dataset_storage toneRose
class node_operator,node_document toneIndigo
```

## Components

### Evaluation workflow

- [Pipeline orchestrator](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/orchestration/pipeline.py): `src/orchestration/pipeline.py`
- [Section graph](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/domain/models.py): `src/domain/models.py`
- [Section separation](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/separation.py): `src/application/separation.py`
- [Section evaluation service](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/evaluation.py): `src/application/evaluation.py`
- [Report aggregation](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/application/aggregation.py): `src/application/aggregation.py`
- [Evaluation report](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/domain/models.py): `src/domain/models.py`

### Document processing

- [Document graph parser](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/graph_parser.py): `src/infrastructure/graph_parser.py`
- [Node preprocessing](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/manager.py): `src/infrastructure/preprocessing/manager.py`
- [Content handlers](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/handlers.py): `src/infrastructure/preprocessing/handlers.py`
- [OCR engine](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/preprocessing/ocr_engine.py): `src/infrastructure/preprocessing/ocr_engine.py`
- [Vision model provider](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/vlm_provider.py): `src/infrastructure/vlm_provider.py`

### Section judging

- [LLM section judge](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/llm_judge.py): `src/infrastructure/llm_judge.py`
- [Reasoning agent client](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/agent_client.py): `src/infrastructure/agent_client.py`
- [LLM provider](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/llm_config.py): `src/infrastructure/llm_config.py`
- [Reference retrieval](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/retrieval.py): `src/infrastructure/retrieval.py`
- [Audit metrics registry](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/infrastructure/metrics_library.py): `src/infrastructure/metrics_library.py`

### Dataset preparation

- [Dataset pipeline](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/pipeline.py): `src/dataset/pipeline.py`
- [Dataset ingestion](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/ingestion.py): `src/dataset/ingestion.py`
- [Dataset graph builder](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/graph_builder.py): `src/dataset/graph_builder.py`
- [Sample annotation](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/annotation.py): `src/dataset/annotation.py`
- [Dataset augmentation](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/augmentation.py): `src/dataset/augmentation.py`
- [Dataset storage](https://github.com/safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents/blob/main/src/dataset/storage.py): `src/dataset/storage.py`

### Other components

- **Document operator**
- **R&D document**

## Connections

- Document operator → Pipeline orchestrator: runs (evidence: `README.md`)
- R&D document → Pipeline orchestrator: submits (evidence: `README.md`)
- Pipeline orchestrator → Document graph parser: parses document (evidence: `README.md`)
- Document graph parser → Section graph: builds sections (evidence: `src/infrastructure/graph_parser.py`)
- Pipeline orchestrator → Node preprocessing: enriches nodes (evidence: `src/orchestration/pipeline.py`)
- Node preprocessing → Content handlers: normalizes content (evidence: `src/infrastructure/preprocessing/manager.py`)
- Node preprocessing → OCR engine: extracts text (evidence: `src/infrastructure/preprocessing/manager.py`)
- Node preprocessing → Vision model provider: interprets visuals (evidence: `src/infrastructure/preprocessing/manager.py`)
- Pipeline orchestrator → Section separation: separates sections (evidence: `README.md`)
- Pipeline orchestrator → Section evaluation service: dispatches sections (evidence: `src/orchestration/pipeline.py`)
- Section evaluation service → LLM section judge: evaluates node (evidence: `README.md`)
- LLM section judge → Reasoning agent client: requests reasoning (evidence: `src/infrastructure/llm_judge.py`)
- Reasoning agent client → LLM provider: sends model requests (evidence: `src/infrastructure/agent_client.py`)
- LLM section judge → Reference retrieval: retrieves standards (evidence: `README.md`)
- LLM section judge → Audit metrics registry: scores criteria (evidence: `src/infrastructure/llm_judge.py`)
- Pipeline orchestrator → Report aggregation: aggregates results (evidence: `src/orchestration/pipeline.py`)
- Report aggregation → Evaluation report: produces report (evidence: `src/application/aggregation.py`)
- Dataset pipeline → Dataset ingestion: loads documents (evidence: `src/dataset/pipeline.py`)
- Dataset pipeline → Dataset graph builder: builds graph (evidence: `src/dataset/pipeline.py`)
- Dataset pipeline → Sample annotation: annotates samples (evidence: `src/dataset/pipeline.py`)
- Dataset pipeline → Dataset augmentation: augments data (evidence: `src/dataset/pipeline.py`)
- Dataset pipeline → Dataset storage: stores outputs (evidence: `src/dataset/pipeline.py`)
