# ksypan/canonical-pipeline architecture

> Architecture diagram of the GitHub repository ksypan/canonical-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ksypan/canonical-pipeline
- Repository: https://github.com/ksypan/canonical-pipeline

## Overview

This is a batch medical-record extraction pipeline: documents are deidentified, parsed, and sent to Yandex only for fields the parser cannot confirm; validated values are merged, clinically reconciled, finalized, checked, and saved. The service-evidence branch preserves accepted quotations and provenance separately from the official 50-field JSON. The README documents the overall stage order; sampled source confirms the batch orchestrator’s calls. Some internal implementations were not excerpted, so their detailed interactions are not expanded.

Main entry points: [`batch_runner.py`](https://github.com/ksypan/canonical-pipeline/blob/main/batch_runner.py) (Batch orchestrator), [`llm_client.py`](https://github.com/ksypan/canonical-pipeline/blob/main/llm_client.py) (Yandex client), [`evidence_layer.py`](https://github.com/ksypan/canonical-pipeline/blob/main/evidence_layer.py) (Evidence and consistency), [`contract_validation.py`](https://github.com/ksypan/canonical-pipeline/blob/main/contract_validation.py) (Response validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and extraction"]
  node_batch["Batch orchestrator<br/>[batch_runner.py]"]
  node_deidentify["Deidentification<br/>[deidentify_impl.py]"]
  node_parser["Clinical parser<br/>[parser_impl.py]"]
end

subgraph group_llm["LLM and evidence"]
  node_prompt["Request builder"]
  node_privacy["Privacy guard<br/>[privacy_guard.py]"]
  node_client["Yandex client<br/>[llm_client.py]"]
end

subgraph group_clinical["Clinical processing"]
  node_evidence["Evidence and consistency<br/>[evidence_layer.py]"]
  node_clinical_rules["Clinical rules<br/>[clinical_rules.py]"]
  node_merge["Value merge<br/>[merge.py]"]
  node_finalization["Defaults and dependencies<br/>[finalization.py]"]
end

subgraph group_contracts["Contracts and validation"]
  node_contract["Field contract"]
  node_contract_validation["Response validation"]
  node_errors["Pipeline errors<br/>[pipeline_errors.py]"]
  node_runtime_contract["Runtime contract"]
  node_final_validator["Final validation<br/>[final_validator.py]"]
end

subgraph group_outputs["Results and service"]
  node_json_io["JSON I/O<br/>[json_io.py]"]
  node_template["50-field template"]
  node_result["Official result JSON"]
  node_service_result["Service evidence result"]
end

node_user(("Document operator"))
node_yandex{{"Yandex AI Studio"}}

node_user -->|"submits documents"| node_batch
node_batch -->|"deidentifies text"| node_deidentify
node_batch -->|"parses text"| node_parser
node_batch -->|"builds request"| node_prompt
node_batch -->|"checks request"| node_privacy
node_batch -->|"calls when needed"| node_client
node_client -->|"sends request"| node_yandex
node_yandex -->|"returns response"| node_client
node_batch -->|"validates response"| node_contract_validation
node_contract_validation -->|"uses field schema"| node_contract
node_batch -->|"validates evidence"| node_evidence
node_evidence -->|"uses clinical mappings"| node_clinical_rules
node_batch -->|"merges values"| node_merge
node_batch -->|"finalizes values"| node_finalization
node_batch -->|"validates final JSON"| node_final_validator
node_final_validator -->|"checks dependencies"| node_runtime_contract
node_runtime_contract -->|"loads field keys"| node_template
node_batch -->|"writes result"| node_json_io
node_json_io -->|"saves atomically"| node_result
node_batch -->|"returns evidence metadata"| node_service_result
node_batch -->|"handles pipeline errors"| node_errors

click node_batch "https://github.com/ksypan/canonical-pipeline/blob/main/batch_runner.py"
click node_deidentify "https://github.com/ksypan/canonical-pipeline/blob/main/deidentify_impl.py"
click node_parser "https://github.com/ksypan/canonical-pipeline/blob/main/parser_impl.py"
click node_prompt "https://github.com/ksypan/canonical-pipeline/blob/main/prompt_builder_impl.py"
click node_privacy "https://github.com/ksypan/canonical-pipeline/blob/main/privacy_guard.py"
click node_client "https://github.com/ksypan/canonical-pipeline/blob/main/llm_client.py"
click node_contract "https://github.com/ksypan/canonical-pipeline/blob/main/field_contract_impl.py"
click node_contract_validation "https://github.com/ksypan/canonical-pipeline/blob/main/contract_validation.py"
click node_errors "https://github.com/ksypan/canonical-pipeline/blob/main/pipeline_errors.py"
click node_evidence "https://github.com/ksypan/canonical-pipeline/blob/main/evidence_layer.py"
click node_clinical_rules "https://github.com/ksypan/canonical-pipeline/blob/main/clinical_rules.py"
click node_merge "https://github.com/ksypan/canonical-pipeline/blob/main/merge.py"
click node_finalization "https://github.com/ksypan/canonical-pipeline/blob/main/finalization.py"
click node_runtime_contract "https://github.com/ksypan/canonical-pipeline/blob/main/runtime_contract.py"
click node_final_validator "https://github.com/ksypan/canonical-pipeline/blob/main/final_validator.py"
click node_json_io "https://github.com/ksypan/canonical-pipeline/blob/main/json_io.py"
click node_template "https://github.com/ksypan/canonical-pipeline/blob/main/participant-output-template-v2.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_batch,node_deidentify,node_parser toneBlue
class node_prompt,node_privacy,node_client toneAmber
class node_evidence,node_clinical_rules,node_merge,node_finalization toneMint
class node_contract,node_contract_validation,node_errors,node_runtime_contract,node_final_validator toneRose
class node_json_io,node_template,node_result,node_service_result,node_user,node_yandex toneIndigo
```

## Components

### Input and extraction

- [Batch orchestrator](https://github.com/ksypan/canonical-pipeline/blob/main/batch_runner.py): `batch_runner.py`
- [Deidentification](https://github.com/ksypan/canonical-pipeline/blob/main/deidentify_impl.py): `deidentify_impl.py`
- [Clinical parser](https://github.com/ksypan/canonical-pipeline/blob/main/parser_impl.py): `parser_impl.py`

### LLM and evidence

- [Request builder](https://github.com/ksypan/canonical-pipeline/blob/main/prompt_builder_impl.py): `prompt_builder_impl.py`
- [Privacy guard](https://github.com/ksypan/canonical-pipeline/blob/main/privacy_guard.py): `privacy_guard.py`
- [Yandex client](https://github.com/ksypan/canonical-pipeline/blob/main/llm_client.py): `llm_client.py`

### Clinical processing

- [Evidence and consistency](https://github.com/ksypan/canonical-pipeline/blob/main/evidence_layer.py): `evidence_layer.py`
- [Clinical rules](https://github.com/ksypan/canonical-pipeline/blob/main/clinical_rules.py): `clinical_rules.py`
- [Value merge](https://github.com/ksypan/canonical-pipeline/blob/main/merge.py): `merge.py`
- [Defaults and dependencies](https://github.com/ksypan/canonical-pipeline/blob/main/finalization.py): `finalization.py`

### Contracts and validation

- [Field contract](https://github.com/ksypan/canonical-pipeline/blob/main/field_contract_impl.py): `field_contract_impl.py`
- [Response validation](https://github.com/ksypan/canonical-pipeline/blob/main/contract_validation.py): `contract_validation.py`
- [Pipeline errors](https://github.com/ksypan/canonical-pipeline/blob/main/pipeline_errors.py): `pipeline_errors.py`
- [Runtime contract](https://github.com/ksypan/canonical-pipeline/blob/main/runtime_contract.py): `runtime_contract.py`
- [Final validation](https://github.com/ksypan/canonical-pipeline/blob/main/final_validator.py): `final_validator.py`

### Results and service

- [JSON I/O](https://github.com/ksypan/canonical-pipeline/blob/main/json_io.py): `json_io.py`
- [50-field template](https://github.com/ksypan/canonical-pipeline/blob/main/participant-output-template-v2.json): `participant-output-template-v2.json`
- **Official result JSON**
- **Service evidence result**

### Other components

- **Document operator**
- **Yandex AI Studio**

## Connections

- Document operator → Batch orchestrator: submits documents (evidence: `README.md`)
- Batch orchestrator → Deidentification: deidentifies text (evidence: `batch_runner.py`)
- Batch orchestrator → Clinical parser: parses text (evidence: `batch_runner.py`)
- Batch orchestrator → Request builder: builds request (evidence: `batch_runner.py`)
- Batch orchestrator → Privacy guard: checks request (evidence: `batch_runner.py`)
- Batch orchestrator → Yandex client: calls when needed (evidence: `batch_runner.py`)
- Yandex client → Yandex AI Studio: sends request (evidence: `README.md`)
- Yandex AI Studio → Yandex client: returns response (evidence: `README.md`)
- Batch orchestrator → Response validation: validates response (evidence: `batch_runner.py`)
- Response validation → Field contract: uses field schema (evidence: `contract_validation.py`)
- Batch orchestrator → Evidence and consistency: validates evidence (evidence: `batch_runner.py`)
- Evidence and consistency → Clinical rules: uses clinical mappings (evidence: `evidence_layer.py`)
- Batch orchestrator → Value merge: merges values (evidence: `batch_runner.py`)
- Batch orchestrator → Defaults and dependencies: finalizes values (evidence: `batch_runner.py`)
- Batch orchestrator → Final validation: validates final JSON (evidence: `batch_runner.py`)
- Final validation → Runtime contract: checks dependencies (evidence: `final_validator.py`)
- Runtime contract → 50-field template: loads field keys (evidence: `runtime_contract.py`)
- Batch orchestrator → JSON I/O: writes result (evidence: `README.md`)
- JSON I/O → Official result JSON: saves atomically (evidence: `README.md`)
- Batch orchestrator → Service evidence result: returns evidence metadata (evidence: `batch_runner.py`)
- Batch orchestrator → Pipeline errors: handles pipeline errors (evidence: `batch_runner.py`)
