# kumarharshzzz04/amazon-ml architecture

> Architecture diagram of the GitHub repository kumarharshzzz04/amazon-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kumarharshzzz04/amazon-ml
- Repository: https://github.com/kumarharshzzz04/amazon-ml

## Overview

This repository contains a business-entity-resolution challenge pipeline: prepare source records, generate blocking keys and candidate pairs, engineer pairwise features, train a matcher, and produce submission results. The sampled driver documents an inference stage, but its referenced inference module is absent from the tree, so the graph does not invent its implementation or connections. The submission validator is shown as a separate challenge utility. README content and several stage implementations were not available in the excerpts.

Main entry points: [`code/business_entity_resolution/run_pipeline.py`](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/run_pipeline.py) (Pipeline driver), [`code/business_entity_resolution/src/train.py`](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/train.py) (Matcher training), [`code/business_entity_resolution/src/build_features.py`](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/build_features.py) (Feature builder), [`code/business_entity_resolution/src/keys.py`](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/keys.py) (Blocking keys)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_pipeline["Pipeline driver<br/>[run_pipeline.py]"]
  node_prep["Record preparation<br/>[prep.py]"]
end

subgraph group_candidate["Candidate generation"]
  node_normalize["Text normalization<br/>[normalize.py]"]
  node_keys["Blocking keys<br/>[keys.py]"]
  node_candidates["Candidate ranking<br/>[candidates.py]"]
  node_recall["Recall evaluation<br/>[eval_recall.py]"]
end

subgraph group_matching["Feature matching"]
  node_blobs["Feature text blobs<br/>[blobs.py]"]
  node_feature_builder["Feature builder<br/>[build_features.py]"]
  node_fast_features["Fast pair features<br/>[feats_fast.py]"]
  node_feature_defs["Feature definitions<br/>[features.py]"]
  node_trainer["Matcher training<br/>[train.py]"]
end

subgraph group_submission["Challenge submission"]
  node_validator["Submission validator"]
end

node_challenge_data["Challenge records"]
node_ground_truth["Training labels"]
node_cache["Prepared records"]
node_key_data["Encoded keys"]
node_candidate_data["Candidate pairs"]
node_blob_data["Blob records"]
node_feature_data["Pair feature matrix"]
node_model["Trained matcher"]
node_submission_files["Submission files"]
node_submitter(("Challenge participant"))
node_scorer(("Challenge scorer"))

node_challenge_data -->|"reads records"| node_prep
node_pipeline -->|"dispatches stage"| node_prep
node_prep -->|"writes prepared data"| node_cache
node_pipeline -->|"dispatches stage"| node_keys
node_keys -->|"reads records"| node_cache
node_keys -->|"normalizes text"| node_normalize
node_keys -->|"writes keys"| node_key_data
node_pipeline -->|"dispatches stage"| node_candidates
node_candidates -->|"reads keys"| node_key_data
node_candidates -->|"writes ranked pairs"| node_candidate_data
node_pipeline -->|"dispatches stage"| node_recall
node_recall -->|"reads records"| node_cache
node_recall -->|"checks candidates"| node_candidate_data
node_recall -->|"compares labels"| node_ground_truth
node_pipeline -->|"dispatches stage"| node_blobs
node_blobs -->|"reads records"| node_cache
node_blobs -->|"reads keys"| node_key_data
node_blobs -->|"writes text blobs"| node_blob_data
node_pipeline -->|"dispatches stage"| node_feature_builder
node_feature_builder -->|"reads candidates"| node_candidate_data
node_feature_builder -->|"reads blob records"| node_blob_data
node_feature_builder -->|"computes chunks"| node_fast_features
node_fast_features -->|"tokenizes text"| node_normalize
node_feature_builder -->|"writes features"| node_feature_data
node_pipeline -->|"dispatches stage"| node_trainer
node_trainer -->|"reads features"| node_feature_data
node_trainer -->|"reads pair indices"| node_candidate_data
node_trainer -->|"builds labels"| node_ground_truth
node_trainer -->|"reads record IDs"| node_cache
node_trainer -->|"writes model"| node_model
node_submitter -->|"provides files"| node_submission_files
node_validator -->|"checks format"| node_submission_files
node_validator -->|"checks source IDs"| node_challenge_data
node_submission_files -->|"submitted results"| node_scorer

click node_pipeline "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/run_pipeline.py"
click node_prep "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/prep.py"
click node_normalize "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/normalize.py"
click node_keys "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/keys.py"
click node_candidates "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/candidates.py"
click node_recall "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/eval_recall.py"
click node_blobs "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/blobs.py"
click node_feature_builder "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/build_features.py"
click node_fast_features "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/feats_fast.py"
click node_feature_defs "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/features.py"
click node_trainer "https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/train.py"
click node_validator "https://github.com/kumarharshzzz04/amazon-ml/blob/master/student_resource/student_resource/utils/validate_submission.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_prep toneBlue
class node_normalize,node_keys,node_candidates,node_recall toneAmber
class node_blobs,node_feature_builder,node_fast_features,node_feature_defs,node_trainer toneMint
class node_validator toneRose
class node_challenge_data,node_ground_truth,node_cache,node_key_data,node_candidate_data,node_blob_data,node_feature_data,node_model,node_submission_files,node_submitter,node_scorer toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline driver](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/run_pipeline.py): `code/business_entity_resolution/run_pipeline.py`
- [Record preparation](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/prep.py): `code/business_entity_resolution/src/prep.py`

### Candidate generation

- [Text normalization](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/normalize.py): `code/business_entity_resolution/src/normalize.py`
- [Blocking keys](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/keys.py): `code/business_entity_resolution/src/keys.py`
- [Candidate ranking](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/candidates.py): `code/business_entity_resolution/src/candidates.py`
- [Recall evaluation](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/eval_recall.py): `code/business_entity_resolution/src/eval_recall.py`

### Feature matching

- [Feature text blobs](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/blobs.py): `code/business_entity_resolution/src/blobs.py`
- [Feature builder](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/build_features.py): `code/business_entity_resolution/src/build_features.py`
- [Fast pair features](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/feats_fast.py): `code/business_entity_resolution/src/feats_fast.py`
- [Feature definitions](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/features.py): `code/business_entity_resolution/src/features.py`
- [Matcher training](https://github.com/kumarharshzzz04/amazon-ml/blob/master/code/business_entity_resolution/src/train.py): `code/business_entity_resolution/src/train.py`

### Challenge submission

- [Submission validator](https://github.com/kumarharshzzz04/amazon-ml/blob/master/student_resource/student_resource/utils/validate_submission.py): `student_resource/student_resource/utils/validate_submission.py`

### Other components

- **Challenge records**
- **Training labels**
- **Prepared records**
- **Encoded keys**
- **Candidate pairs**
- **Blob records**
- **Pair feature matrix**
- **Trained matcher**
- **Submission files**
- **Challenge participant**
- **Challenge scorer**

## Connections

- Challenge records → Record preparation: reads records
- Pipeline driver → Record preparation: dispatches stage
- Record preparation → Prepared records: writes prepared data
- Pipeline driver → Blocking keys: dispatches stage
- Blocking keys → Prepared records: reads records
- Blocking keys → Text normalization: normalizes text
- Blocking keys → Encoded keys: writes keys
- Pipeline driver → Candidate ranking: dispatches stage
- Candidate ranking → Encoded keys: reads keys
- Candidate ranking → Candidate pairs: writes ranked pairs
- Pipeline driver → Recall evaluation: dispatches stage
- Recall evaluation → Prepared records: reads records
- Recall evaluation → Candidate pairs: checks candidates
- Recall evaluation → Training labels: compares labels
- Pipeline driver → Feature text blobs: dispatches stage
- Feature text blobs → Prepared records: reads records
- Feature text blobs → Encoded keys: reads keys
- Feature text blobs → Blob records: writes text blobs
- Pipeline driver → Feature builder: dispatches stage
- Feature builder → Candidate pairs: reads candidates
- Feature builder → Blob records: reads blob records
- Feature builder → Fast pair features: computes chunks
- Fast pair features → Text normalization: tokenizes text
- Feature builder → Pair feature matrix: writes features
- Pipeline driver → Matcher training: dispatches stage
- Matcher training → Pair feature matrix: reads features
- Matcher training → Candidate pairs: reads pair indices
- Matcher training → Training labels: builds labels
- Matcher training → Prepared records: reads record IDs
- Matcher training → Trained matcher: writes model
- Challenge participant → Submission files: provides files
- Submission validator → Submission files: checks format
- Submission validator → Challenge records: checks source IDs
- Submission files → Challenge scorer: submitted results