How kumarharshzzz04/amazon-ml works
This repository contains a business-entity-resolution challenge pipeline: prepare source records, generate blocking keys and candidate pairs, engineer pairwise features, train a matcher, and produce submission results. The sampled driver documents an inference stage, but its referenced inference module is absent from the tree, so the graph does not invent its implementation or connections. The submission validator is shown as a separate challenge utility. README content and several stage implementations were not available in the excerpts.
Main entry points: run_pipeline.py Pipeline driver · train.py Matcher training · build_features.py Feature builder · keys.py Blocking keys
Components 23
Pipeline orchestration
- Pipeline driver
code/business_entity_resolution/run_pipeline.py - Record preparation
code/business_entity_resolution/src/prep.py
Candidate generation
- Text normalization
code/business_entity_resolution/src/normalize.py - Blocking keys
code/business_entity_resolution/src/keys.py - Candidate ranking
code/business_entity_resolution/src/candidates.py - Recall evaluation
code/business_entity_resolution/src/eval_recall.py
Feature matching
- Feature text blobs
code/business_entity_resolution/src/blobs.py - Feature builder
code/business_entity_resolution/src/build_features.py - Fast pair features
code/business_entity_resolution/src/feats_fast.py - Feature definitions
code/business_entity_resolution/src/features.py - Matcher training
code/business_entity_resolution/src/train.py
Challenge submission
- Submission validator
student_resource/student_resource/utils/validate_submission.py
Other components
- Challenge records
- Training labels
- Prepared records
- Encoded keys
- Candidate pairs
- Blob records
- Pair feature matrix
- Trained matcher
- Submission files
- Challenge participant
- Challenge scorer
Connections 34
- Challenge records → Record preparation: reads records
- Pipeline driver → Record preparation: dispatches stage
- Record preparation → Prepared records: writes prepared data
- Pipeline driver → Blocking keys: dispatches stage
- Blocking keys → Prepared records: reads records
- Blocking keys → Text normalization: normalizes text
- Blocking keys → Encoded keys: writes keys
- Pipeline driver → Candidate ranking: dispatches stage
- Candidate ranking → Encoded keys: reads keys
- Candidate ranking → Candidate pairs: writes ranked pairs
- Pipeline driver → Recall evaluation: dispatches stage
- Recall evaluation → Prepared records: reads records
- Recall evaluation → Candidate pairs: checks candidates
- Recall evaluation → Training labels: compares labels
- Pipeline driver → Feature text blobs: dispatches stage
- Feature text blobs → Prepared records: reads records
- Feature text blobs → Encoded keys: reads keys
- Feature text blobs → Blob records: writes text blobs
- Pipeline driver → Feature builder: dispatches stage
- Feature builder → Candidate pairs: reads candidates
- Feature builder → Blob records: reads blob records
- Feature builder → Fast pair features: computes chunks
- Fast pair features → Text normalization: tokenizes text
- Feature builder → Pair feature matrix: writes features
- Pipeline driver → Matcher training: dispatches stage
- Matcher training → Pair feature matrix: reads features
- Matcher training → Candidate pairs: reads pair indices
- Matcher training → Training labels: builds labels
- Matcher training → Prepared records: reads record IDs
- Matcher training → Trained matcher: writes model
- Challenge participant → Submission files: provides files
- Submission validator → Submission files: checks format
- Submission validator → Challenge records: checks source IDs
- Submission files → Challenge scorer: submitted results