# chirag-2106/amazon-er-challenge architecture

> Architecture diagram of the GitHub repository chirag-2106/amazon-er-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chirag-2106/amazon-er-challenge
- Repository: https://github.com/chirag-2106/amazon-er-challenge

## Overview

This repository contains scripts for entity resolution across three business-record sources. The principal inference workflow cleans source records, blocks likely S2/S3 candidates for each S1 record, scores candidate pairs with a LightGBM model, and writes candidate and match TSVs. Separate scripts build training features, train or evaluate models, and provide alternative matching, post-processing, locality, and validation workflows. The README is empty; several scripts are represented by their filenames only, so uncertain internal wiring is omitted.

Main entry points: [`run_pipeline_smart_09.py`](https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_smart_09.py) (Smart LightGBM pipeline), [`run_pipeline_lgb.py`](https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_lgb.py) (LightGBM pipeline), [`src/baseline.py`](https://github.com/chirag-2106/amazon-er-challenge/blob/main/src/baseline.py) (Baseline matcher), [`generate_submission.py`](https://github.com/chirag-2106/amazon-er-challenge/blob/main/generate_submission.py) (Rule-based matcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_matching["Matching workflows"]
  node_smart["Smart LightGBM pipeline"]
  node_lgbpipe["LightGBM pipeline"]
  node_baseline["Baseline matcher<br/>[baseline.py]"]
  node_generate["Rule-based matcher"]
  node_candidatefile["Candidate pairs"]
  node_matches["Match results"]
  node_final["Balanced submission"]
end

subgraph group_model["Model lifecycle"]
  node_features["Training features"]
  node_train["Train and evaluate<br/>[2_train_eval.py]"]
  node_modelartifact["LightGBM model"]
  node_predict["Candidate scorer<br/>[3_predict.py]"]
  node_trainpredict["Train and predict"]
  node_ranker["Ranker training"]
  node_retrain["Precision retraining"]
end

subgraph group_quality["Refinement and validation"]
  node_postprocess["Post-process filter"]
  node_locality["Locality filter"]
  node_validator["Submission validator"]
  node_validation["Local evaluation<br/>[local_val.py]"]
  node_diagnostics["Matching diagnostics"]
end

node_operator(("Challenge operator"))
node_testdata["Test datasets"]

node_operator -->|"runs"| node_smart
node_testdata -->|"loads"| node_smart
node_smart -->|"writes candidates"| node_candidatefile
node_smart -->|"writes matches"| node_matches
node_operator -.->|"runs"| node_lgbpipe
node_testdata -.->|"loads"| node_lgbpipe
node_lgbpipe -.->|"writes candidates"| node_candidatefile
node_lgbpipe -.->|"writes matches"| node_matches
node_operator -.->|"runs"| node_baseline
node_testdata -.->|"loads"| node_baseline
node_baseline -.->|"writes candidates"| node_candidatefile
node_baseline -.->|"writes matches"| node_matches
node_operator -.->|"runs"| node_generate
node_testdata -.->|"loads"| node_generate
node_generate -.->|"writes candidates"| node_candidatefile
node_generate -.->|"writes matches"| node_matches
node_operator -->|"runs"| node_features
node_features -->|"features for"| node_train
node_train -->|"trains model"| node_modelartifact
node_candidatefile -->|"loads candidates"| node_predict
node_modelartifact -->|"loads model"| node_predict
node_predict -->|"loads records"| node_testdata
node_predict -->|"writes matches"| node_matches
node_operator -.->|"runs"| node_trainpredict
node_operator -.->|"runs"| node_ranker
node_operator -.->|"runs"| node_retrain
node_operator -.->|"runs"| node_final
node_testdata -.->|"loads"| node_final
node_final -.->|"writes candidates"| node_candidatefile
node_final -.->|"writes matches"| node_matches
node_operator -.->|"runs"| node_postprocess
node_operator -.->|"runs"| node_locality
node_operator -->|"runs"| node_validator
node_matches -->|"validates"| node_validator
node_candidatefile -->|"validates"| node_validator
node_testdata -->|"checks IDs"| node_validator
node_operator -.->|"runs"| node_validation
node_operator -.->|"runs"| node_diagnostics

click node_smart "https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_smart_09.py"
click node_lgbpipe "https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_lgb.py"
click node_baseline "https://github.com/chirag-2106/amazon-er-challenge/blob/main/src/baseline.py"
click node_generate "https://github.com/chirag-2106/amazon-er-challenge/blob/main/generate_submission.py"
click node_features "https://github.com/chirag-2106/amazon-er-challenge/blob/main/1_build_features.py"
click node_train "https://github.com/chirag-2106/amazon-er-challenge/blob/main/2_train_eval.py"
click node_predict "https://github.com/chirag-2106/amazon-er-challenge/blob/main/3_predict.py"
click node_trainpredict "https://github.com/chirag-2106/amazon-er-challenge/blob/main/train_and_predict.py"
click node_ranker "https://github.com/chirag-2106/amazon-er-challenge/blob/main/train_lightgbm_ranker.py"
click node_retrain "https://github.com/chirag-2106/amazon-er-challenge/blob/main/retrain_precision_lgb.py"
click node_final "https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_final_balanced_submission.py"
click node_postprocess "https://github.com/chirag-2106/amazon-er-challenge/blob/main/post_process_filter.py"
click node_locality "https://github.com/chirag-2106/amazon-er-challenge/blob/main/filter_by_locality.py"
click node_validator "https://github.com/chirag-2106/amazon-er-challenge/blob/main/utils/validate_submission.py"
click node_validation "https://github.com/chirag-2106/amazon-er-challenge/blob/main/local_val.py"
click node_diagnostics "https://github.com/chirag-2106/amazon-er-challenge/blob/main/diagnose_blocking_recall.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_smart,node_lgbpipe,node_baseline,node_generate,node_candidatefile,node_matches,node_final toneBlue
class node_features,node_train,node_modelartifact,node_predict,node_trainpredict,node_ranker,node_retrain toneAmber
class node_postprocess,node_locality,node_validator,node_validation,node_diagnostics toneMint
class node_operator,node_testdata toneIndigo
```

## Components

### Matching workflows

- [Smart LightGBM pipeline](https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_smart_09.py): `run_pipeline_smart_09.py`
- [LightGBM pipeline](https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_pipeline_lgb.py): `run_pipeline_lgb.py`
- [Baseline matcher](https://github.com/chirag-2106/amazon-er-challenge/blob/main/src/baseline.py): `src/baseline.py`
- [Rule-based matcher](https://github.com/chirag-2106/amazon-er-challenge/blob/main/generate_submission.py): `generate_submission.py`
- **Candidate pairs**
- **Match results**
- [Balanced submission](https://github.com/chirag-2106/amazon-er-challenge/blob/main/run_final_balanced_submission.py): `run_final_balanced_submission.py`

### Model lifecycle

- [Training features](https://github.com/chirag-2106/amazon-er-challenge/blob/main/1_build_features.py): `1_build_features.py`
- [Train and evaluate](https://github.com/chirag-2106/amazon-er-challenge/blob/main/2_train_eval.py): `2_train_eval.py`
- **LightGBM model**
- [Candidate scorer](https://github.com/chirag-2106/amazon-er-challenge/blob/main/3_predict.py): `3_predict.py`
- [Train and predict](https://github.com/chirag-2106/amazon-er-challenge/blob/main/train_and_predict.py): `train_and_predict.py`
- [Ranker training](https://github.com/chirag-2106/amazon-er-challenge/blob/main/train_lightgbm_ranker.py): `train_lightgbm_ranker.py`
- [Precision retraining](https://github.com/chirag-2106/amazon-er-challenge/blob/main/retrain_precision_lgb.py): `retrain_precision_lgb.py`

### Refinement and validation

- [Post-process filter](https://github.com/chirag-2106/amazon-er-challenge/blob/main/post_process_filter.py): `post_process_filter.py`
- [Locality filter](https://github.com/chirag-2106/amazon-er-challenge/blob/main/filter_by_locality.py): `filter_by_locality.py`
- [Submission validator](https://github.com/chirag-2106/amazon-er-challenge/blob/main/utils/validate_submission.py): `utils/validate_submission.py`
- [Local evaluation](https://github.com/chirag-2106/amazon-er-challenge/blob/main/local_val.py): `local_val.py`
- [Matching diagnostics](https://github.com/chirag-2106/amazon-er-challenge/blob/main/diagnose_blocking_recall.py): `diagnose_blocking_recall.py`

### Other components

- **Challenge operator**
- **Test datasets**

## Connections

- Challenge operator → Smart LightGBM pipeline: runs
- Test datasets → Smart LightGBM pipeline: loads
- Smart LightGBM pipeline → Candidate pairs: writes candidates
- Smart LightGBM pipeline → Match results: writes matches
- Challenge operator → LightGBM pipeline: runs
- Test datasets → LightGBM pipeline: loads
- LightGBM pipeline → Candidate pairs: writes candidates
- LightGBM pipeline → Match results: writes matches
- Challenge operator → Baseline matcher: runs
- Test datasets → Baseline matcher: loads
- Baseline matcher → Candidate pairs: writes candidates
- Baseline matcher → Match results: writes matches
- Challenge operator → Rule-based matcher: runs
- Test datasets → Rule-based matcher: loads
- Rule-based matcher → Candidate pairs: writes candidates
- Rule-based matcher → Match results: writes matches
- Challenge operator → Training features: runs
- Training features → Train and evaluate: features for
- Train and evaluate → LightGBM model: trains model
- Candidate pairs → Candidate scorer: loads candidates
- LightGBM model → Candidate scorer: loads model
- Candidate scorer → Test datasets: loads records
- Candidate scorer → Match results: writes matches
- Challenge operator → Train and predict: runs
- Challenge operator → Ranker training: runs
- Challenge operator → Precision retraining: runs
- Challenge operator → Balanced submission: runs
- Test datasets → Balanced submission: loads
- Balanced submission → Candidate pairs: writes candidates
- Balanced submission → Match results: writes matches
- Challenge operator → Post-process filter: runs
- Challenge operator → Locality filter: runs
- Challenge operator → Submission validator: runs
- Match results → Submission validator: validates
- Candidate pairs → Submission validator: validates
- Test datasets → Submission validator: checks IDs
- Challenge operator → Local evaluation: runs
- Challenge operator → Matching diagnostics: runs