How chirag-2106/amazon-er-challenge works
This repository contains scripts for entity resolution across three business-record sources. The principal inference workflow cleans source records, blocks likely S2/S3 candidates for each S1 record, scores candidate pairs with a LightGBM model, and writes candidate and match TSVs. Separate scripts build training features, train or evaluate models, and provide alternative matching, post-processing, locality, and validation workflows. The README is empty; several scripts are represented by their filenames only, so uncertain internal wiring is omitted.
Main entry points: run_pipeline_smart_09.py Smart LightGBM pipeline · run_pipeline_lgb.py LightGBM pipeline · baseline.py Baseline matcher · generate_submission.py Rule-based matcher
Components 21
Matching workflows
- Smart LightGBM pipeline
run_pipeline_smart_09.py - LightGBM pipeline
run_pipeline_lgb.py - Baseline matcher
src/baseline.py - Rule-based matcher
generate_submission.py - Candidate pairs
- Match results
- Balanced submission
run_final_balanced_submission.py
Model lifecycle
- Training features
1_build_features.py - Train and evaluate
2_train_eval.py - LightGBM model
- Candidate scorer
3_predict.py - Train and predict
train_and_predict.py - Ranker training
train_lightgbm_ranker.py - Precision retraining
retrain_precision_lgb.py
Refinement and validation
- Post-process filter
post_process_filter.py - Locality filter
filter_by_locality.py - Submission validator
utils/validate_submission.py - Local evaluation
local_val.py - Matching diagnostics
diagnose_blocking_recall.py
Other components
- Challenge operator
- Test datasets
Connections 38
- Challenge operator → Smart LightGBM pipeline: runs
- Test datasets → Smart LightGBM pipeline: loads
- Smart LightGBM pipeline → Candidate pairs: writes candidates
- Smart LightGBM pipeline → Match results: writes matches
- Challenge operator → LightGBM pipeline: runs
- Test datasets → LightGBM pipeline: loads
- LightGBM pipeline → Candidate pairs: writes candidates
- LightGBM pipeline → Match results: writes matches
- Challenge operator → Baseline matcher: runs
- Test datasets → Baseline matcher: loads
- Baseline matcher → Candidate pairs: writes candidates
- Baseline matcher → Match results: writes matches
- Challenge operator → Rule-based matcher: runs
- Test datasets → Rule-based matcher: loads
- Rule-based matcher → Candidate pairs: writes candidates
- Rule-based matcher → Match results: writes matches
- Challenge operator → Training features: runs
- Training features → Train and evaluate: features for
- Train and evaluate → LightGBM model: trains model
- Candidate pairs → Candidate scorer: loads candidates
- LightGBM model → Candidate scorer: loads model
- Candidate scorer → Test datasets: loads records
- Candidate scorer → Match results: writes matches
- Challenge operator → Train and predict: runs
- Challenge operator → Ranker training: runs
- Challenge operator → Precision retraining: runs
- Challenge operator → Balanced submission: runs
- Test datasets → Balanced submission: loads
- Balanced submission → Candidate pairs: writes candidates
- Balanced submission → Match results: writes matches
- Challenge operator → Post-process filter: runs
- Challenge operator → Locality filter: runs
- Challenge operator → Submission validator: runs
- Match results → Submission validator: validates
- Candidate pairs → Submission validator: validates
- Test datasets → Submission validator: checks IDs
- Challenge operator → Local evaluation: runs
- Challenge operator → Matching diagnostics: runs