How ksypan/canonical-pipeline works
This is a batch medical-record extraction pipeline: documents are deidentified, parsed, and sent to Yandex only for fields the parser cannot confirm; validated values are merged, clinically reconciled, finalized, checked, and saved. The service-evidence branch preserves accepted quotations and provenance separately from the official 50-field JSON. The README documents the overall stage order; sampled source confirms the batch orchestrator’s calls. Some internal implementations were not excerpted, so their detailed interactions are not expanded.
Main entry points: batch_runner.py Batch orchestrator · llm_client.py Yandex client · evidence_layer.py Evidence and consistency · contract_validation.py Response validation
Components 21
Input and extraction
- Batch orchestrator
batch_runner.py - Deidentification
deidentify_impl.py - Clinical parser
parser_impl.py
LLM and evidence
- Request builder
prompt_builder_impl.py - Privacy guard
privacy_guard.py - Yandex client
llm_client.py
Clinical processing
- Evidence and consistency
evidence_layer.py - Clinical rules
clinical_rules.py - Value merge
merge.py - Defaults and dependencies
finalization.py
Contracts and validation
- Field contract
field_contract_impl.py - Response validation
contract_validation.py - Pipeline errors
pipeline_errors.py - Runtime contract
runtime_contract.py - Final validation
final_validator.py
Results and service
- JSON I/O
json_io.py - 50-field template
participant-output-template-v2.json - Official result JSON
- Service evidence result
Other components
- Document operator
- Yandex AI Studio
Connections 21
- Document operator → Batch orchestrator: submits documents
README.md - Batch orchestrator → Deidentification: deidentifies text
batch_runner.py - Batch orchestrator → Clinical parser: parses text
batch_runner.py - Batch orchestrator → Request builder: builds request
batch_runner.py - Batch orchestrator → Privacy guard: checks request
batch_runner.py - Batch orchestrator → Yandex client: calls when needed
batch_runner.py - Yandex client → Yandex AI Studio: sends request
README.md - Yandex AI Studio → Yandex client: returns response
README.md - Batch orchestrator → Response validation: validates response
batch_runner.py - Response validation → Field contract: uses field schema
contract_validation.py - Batch orchestrator → Evidence and consistency: validates evidence
batch_runner.py - Evidence and consistency → Clinical rules: uses clinical mappings
evidence_layer.py - Batch orchestrator → Value merge: merges values
batch_runner.py - Batch orchestrator → Defaults and dependencies: finalizes values
batch_runner.py - Batch orchestrator → Final validation: validates final JSON
batch_runner.py - Final validation → Runtime contract: checks dependencies
final_validator.py - Runtime contract → 50-field template: loads field keys
runtime_contract.py - Batch orchestrator → JSON I/O: writes result
README.md - JSON I/O → Official result JSON: saves atomically
README.md - Batch orchestrator → Service evidence result: returns evidence metadata
batch_runner.py - Batch orchestrator → Pipeline errors: handles pipeline errors
batch_runner.py