How safaehaj/leap-automated-evaluation-pipeline-ai-generated-documents works
LEAP evaluates technical R&D documents by parsing them into section graphs, enriching section content, judging sections against retrieved standards, and aggregating section scores into an audit-readiness report. The README identifies graph construction, multi-agent evaluation, and iterative refinement as product capabilities; the sampled orchestrator excerpt confirms parallel section dispatch and aggregation. A separate dataset pipeline prepares graph-structured data and annotations. Some connections are documented only at the README or module-reference level; unsampled implementation details and the refinement loop are therefore not expanded into unverified runtime edges.
Main entry points: orchestration/pipeline.py Pipeline orchestrator · dataset/pipeline.py Dataset pipeline · manager.py Node preprocessing · llm_judge.py LLM section judge
Components 24
Evaluation workflow
- Pipeline orchestrator
src/orchestration/pipeline.py - Section graph
src/domain/models.py - Section separation
src/application/separation.py - Section evaluation service
src/application/evaluation.py - Report aggregation
src/application/aggregation.py - Evaluation report
src/domain/models.py
Document processing
- Document graph parser
src/infrastructure/graph_parser.py - Node preprocessing
src/infrastructure/preprocessing/manager.py - Content handlers
src/infrastructure/preprocessing/handlers.py - OCR engine
src/infrastructure/preprocessing/ocr_engine.py - Vision model provider
src/infrastructure/vlm_provider.py
Section judging
- LLM section judge
src/infrastructure/llm_judge.py - Reasoning agent client
src/infrastructure/agent_client.py - LLM provider
src/infrastructure/llm_config.py - Reference retrieval
src/infrastructure/retrieval.py - Audit metrics registry
src/infrastructure/metrics_library.py
Dataset preparation
- Dataset pipeline
src/dataset/pipeline.py - Dataset ingestion
src/dataset/ingestion.py - Dataset graph builder
src/dataset/graph_builder.py - Sample annotation
src/dataset/annotation.py - Dataset augmentation
src/dataset/augmentation.py - Dataset storage
src/dataset/storage.py
Other components
- Document operator
- R&D document
Connections 22
- Document operator → Pipeline orchestrator: runs
README.md - R&D document → Pipeline orchestrator: submits
README.md - Pipeline orchestrator → Document graph parser: parses document
README.md - Document graph parser → Section graph: builds sections
src/infrastructure/graph_parser.py - Pipeline orchestrator → Node preprocessing: enriches nodes
src/orchestration/pipeline.py - Node preprocessing → Content handlers: normalizes content
src/infrastructure/preprocessing/manager.py - Node preprocessing → OCR engine: extracts text
src/infrastructure/preprocessing/manager.py - Node preprocessing → Vision model provider: interprets visuals
src/infrastructure/preprocessing/manager.py - Pipeline orchestrator → Section separation: separates sections
README.md - Pipeline orchestrator → Section evaluation service: dispatches sections
src/orchestration/pipeline.py - Section evaluation service → LLM section judge: evaluates node
README.md - LLM section judge → Reasoning agent client: requests reasoning
src/infrastructure/llm_judge.py - Reasoning agent client → LLM provider: sends model requests
src/infrastructure/agent_client.py - LLM section judge → Reference retrieval: retrieves standards
README.md - LLM section judge → Audit metrics registry: scores criteria
src/infrastructure/llm_judge.py - Pipeline orchestrator → Report aggregation: aggregates results
src/orchestration/pipeline.py - Report aggregation → Evaluation report: produces report
src/application/aggregation.py - Dataset pipeline → Dataset ingestion: loads documents
src/dataset/pipeline.py - Dataset pipeline → Dataset graph builder: builds graph
src/dataset/pipeline.py - Dataset pipeline → Sample annotation: annotates samples
src/dataset/pipeline.py - Dataset pipeline → Dataset augmentation: augments data
src/dataset/pipeline.py - Dataset pipeline → Dataset storage: stores outputs
src/dataset/pipeline.py