How premliorate/amazon-ml-challenge works
This repository is an Amazon entity-resolution ML task: it generates candidate businesses for each source-1 record, scores candidate pairs, and writes match and candidate TSVs. The README identifies several runnable pipeline variants and mini/train/inference modes; excerpts show the ultimate pipeline’s blocking, pair expansion, feature generation, prediction, and output behavior. Training scripts, threshold analysis, smart inference, and submission validation are distinct supporting capabilities. Several internals and wiring paths are unsampled, so the graph omits uncertain links rather than inferring calls.
Main entry points: pipeline_ultimate.py Ultimate pipeline · validate_submission.py Submission validator · dataset Task datasets · pipeline_power.py Power pipeline
Components 18
Pipelines
- Ultimate pipeline
pipeline_ultimate.py - Power pipeline
pipeline_power.py - Fast pipeline
pipeline_fast.py - Full pipeline
pipeline.py
Matching stages
- Candidate blocking
blocking_v2.py - Pair features
pipeline_ultimate.py - Match outputs
pipeline_ultimate.py
Model development
- Initial trainer
train_model.py - V2 trainer
train_model_v2.py - V3 trainer
train_model_v3.py - Threshold analysis
learn_thresholds.py - Smart inference
inference_smart.py - Ground-truth analysis
analyze_gt.py - Data exploration
eda.py - Mini dataset creation
create_mini_dataset.py
Submission quality
- Submission validator
student_resource/utils/validate_submission.py - Task datasets
student_resource/dataset
Other components
- Task operator
Connections 19
- Task operator → Ultimate pipeline: runs mode
README.md - Task operator → Power pipeline: runs pipeline
README.md - Task operator → Fast pipeline: runs pipeline
README.md - Task operator → Full pipeline: runs pipeline
README.md - Ultimate pipeline → Candidate blocking: produces candidates
- Ultimate pipeline → Pair features: generates features
pipeline_ultimate.py - Ultimate pipeline → Match outputs: writes TSVs
pipeline_ultimate.py - Task operator → Initial trainer: runs training
- Task operator → V2 trainer: runs training
- Task operator → V3 trainer: runs training
- Task operator → Threshold analysis: runs analysis
- Task operator → Smart inference: runs inference
inference_smart.py - Task operator → Ground-truth analysis: runs analysis
- Task operator → Data exploration: runs analysis
- Task operator → Mini dataset creation: creates mini data
- Task operator → Submission validator: validates submission
student_resource/utils/validate_submission.py - Submission validator → Match outputs: checks result format
- Ultimate pipeline → Task datasets: reads task data
- Submission validator → Task datasets: checks entity IDs
student_resource/utils/validate_submission.py