# praneeth132006/entitymatch-pipeline architecture

> Architecture diagram of the GitHub repository praneeth132006/entitymatch-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/praneeth132006/entitymatch-pipeline
- Repository: https://github.com/praneeth132006/entitymatch-pipeline

## Overview

EntityMatch is an offline business-entity resolution pipeline: it retrieves likely cross-source record pairs, trains and applies a CatBoost classifier, validates the resulting files, and packages a reproducible submission. The main workflow is driven by CLI commands in `src/pipeline.py`; the native C++ retriever performs blocking and feature construction. The README also documents methodology generation and reproduction from a ZIP. Source excerpts are partial, so the graph omits internal data handoffs and calls that cannot be established from the available evidence.

Main entry points: [`src/pipeline.py`](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py) (CLI pipeline), [`src/retrieve.cpp`](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/retrieve.cpp) (Candidate retrieval), [`output`](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output) (Candidate pairs), [`src/build_document.py`](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/build_document.py) (Methodology generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Pipeline orchestration"]
  node_pipeline["CLI pipeline<br/>[pipeline.py]"]
  node_validation["Output validation<br/>[pipeline.py]"]
  node_packaging["Submission packaging<br/>[pipeline.py]"]
  node_reproduction["Pipeline reproduction<br/>[pipeline.py]"]
end

subgraph group_matching["Entity matching"]
  node_retriever["Candidate retrieval<br/>[retrieve.cpp]"]
  node_training["Model training<br/>[pipeline.py]"]
  node_model["Trained model"]
  node_prediction["Candidate scoring<br/>[pipeline.py]"]
end

subgraph group_artifacts["Results and packaging"]
  node_candidate_output["Candidate pairs"]
  node_match_output["Matching results"]
  node_validation_report["Validation report"]
  node_submission["Submission ZIP"]
end

subgraph group_methodology["Methodology documents"]
  node_documentation["Methodology generation<br/>[build_document.py]"]
  node_methodology_output["Methodology files"]
end

node_operator(("Operator"))
node_dataset["Input datasets"]

node_operator -->|"runs commands"| node_pipeline
node_pipeline -->|"invokes retrieval"| node_retriever
node_dataset -->|"scanned by"| node_retriever
node_retriever -->|"provides training pairs"| node_training
node_pipeline -->|"runs training"| node_training
node_training -->|"saves model"| node_model
node_pipeline -->|"runs prediction"| node_prediction
node_model -->|"scores with"| node_prediction
node_dataset -->|"provides test data"| node_prediction
node_prediction -->|"writes candidates"| node_candidate_output
node_prediction -->|"writes matches"| node_match_output
node_pipeline -->|"runs validation"| node_validation
node_dataset -->|"checks target IDs"| node_validation
node_candidate_output -->|"checked by"| node_validation
node_match_output -->|"checked by"| node_validation
node_validation -->|"writes report"| node_validation_report
node_pipeline -->|"runs packaging"| node_packaging
node_packaging -->|"creates ZIP"| node_submission
node_pipeline -->|"dispatches generation"| node_documentation
node_documentation -->|"writes documents"| node_methodology_output
node_pipeline -->|"runs reproduction"| node_reproduction
node_reproduction -->|"reruns prediction"| node_prediction
node_reproduction -->|"reruns validation"| node_validation

click node_pipeline "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"
click node_retriever "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/retrieve.cpp"
click node_training "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"
click node_prediction "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"
click node_candidate_output "https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output"
click node_match_output "https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output"
click node_validation "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"
click node_validation_report "https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output"
click node_packaging "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"
click node_submission "https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output"
click node_documentation "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/build_document.py"
click node_methodology_output "https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output"
click node_reproduction "https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_validation,node_packaging,node_reproduction toneBlue
class node_retriever,node_training,node_model,node_prediction toneAmber
class node_candidate_output,node_match_output,node_validation_report,node_submission toneMint
class node_documentation,node_methodology_output toneRose
class node_operator,node_dataset toneIndigo
```

## Components

### Pipeline orchestration

- [CLI pipeline](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [Output validation](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [Submission packaging](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [Pipeline reproduction](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`

### Entity matching

- [Candidate retrieval](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/retrieve.cpp): `src/retrieve.cpp`
- [Model training](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- **Trained model**
- [Candidate scoring](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`

### Results and packaging

- [Candidate pairs](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output): `output`
- [Matching results](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output): `output`
- [Validation report](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output): `output`
- [Submission ZIP](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output): `output`

### Methodology documents

- [Methodology generation](https://github.com/praneeth132006/entitymatch-pipeline/blob/main/src/build_document.py): `src/build_document.py`
- [Methodology files](https://github.com/praneeth132006/entitymatch-pipeline/tree/main/output): `output`

### Other components

- **Operator**
- **Input datasets**

## Connections

- Operator → CLI pipeline: runs commands
- CLI pipeline → Candidate retrieval: invokes retrieval
- Input datasets → Candidate retrieval: scanned by
- Candidate retrieval → Model training: provides training pairs
- CLI pipeline → Model training: runs training
- Model training → Trained model: saves model
- CLI pipeline → Candidate scoring: runs prediction
- Trained model → Candidate scoring: scores with
- Input datasets → Candidate scoring: provides test data
- Candidate scoring → Candidate pairs: writes candidates
- Candidate scoring → Matching results: writes matches
- CLI pipeline → Output validation: runs validation
- Input datasets → Output validation: checks target IDs
- Candidate pairs → Output validation: checked by
- Matching results → Output validation: checked by
- Output validation → Validation report: writes report
- CLI pipeline → Submission packaging: runs packaging
- Submission packaging → Submission ZIP: creates ZIP
- CLI pipeline → Methodology generation: dispatches generation
- Methodology generation → Methodology files: writes documents
- CLI pipeline → Pipeline reproduction: runs reproduction
- Pipeline reproduction → Candidate scoring: reruns prediction
- Pipeline reproduction → Output validation: reruns validation