# premliorate/amazon-ml-challenge architecture

> Architecture diagram of the GitHub repository premliorate/amazon-ml-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/premliorate/amazon-ml-challenge
- Repository: https://github.com/premliorate/amazon-ml-challenge

## Overview

This repository is an Amazon entity-resolution ML task: it generates candidate businesses for each source-1 record, scores candidate pairs, and writes match and candidate TSVs. The README identifies several runnable pipeline variants and mini/train/inference modes; excerpts show the ultimate pipeline’s blocking, pair expansion, feature generation, prediction, and output behavior. Training scripts, threshold analysis, smart inference, and submission validation are distinct supporting capabilities. Several internals and wiring paths are unsampled, so the graph omits uncertain links rather than inferring calls.

Main entry points: [`pipeline_ultimate.py`](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py) (Ultimate pipeline), [`student_resource/utils/validate_submission.py`](https://github.com/premliorate/amazon-ml-challenge/blob/main/student_resource/utils/validate_submission.py) (Submission validator), [`student_resource/dataset`](https://github.com/premliorate/amazon-ml-challenge/tree/main/student_resource/dataset) (Task datasets), [`pipeline_power.py`](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_power.py) (Power pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_run["Pipelines"]
  node_ultimate["Ultimate pipeline"]
  node_power["Power pipeline<br/>[pipeline_power.py]"]
  node_fast["Fast pipeline<br/>[pipeline_fast.py]"]
  node_baseline["Full pipeline<br/>[pipeline.py]"]
end

subgraph group_match["Matching stages"]
  node_blocking["Candidate blocking<br/>[blocking_v2.py]"]
  node_pairfeatures["Pair features"]
  node_matchoutput["Match outputs"]
end

subgraph group_model["Model development"]
  node_trainer1["Initial trainer<br/>[train_model.py]"]
  node_trainer2["V2 trainer<br/>[train_model_v2.py]"]
  node_trainer3["V3 trainer<br/>[train_model_v3.py]"]
  node_thresholds["Threshold analysis"]
  node_smart["Smart inference<br/>[inference_smart.py]"]
  node_groundtruth["Ground-truth analysis<br/>[analyze_gt.py]"]
  node_exploration["Data exploration<br/>[eda.py]"]
  node_minidata["Mini dataset creation"]
end

subgraph group_quality["Submission quality"]
  node_validator["Submission validator"]
  node_dataset[("Task datasets")]
end

node_operator(("Task operator"))

node_operator -->|"runs mode"| node_ultimate
node_operator -->|"runs pipeline"| node_power
node_operator -->|"runs pipeline"| node_fast
node_operator -->|"runs pipeline"| node_baseline
node_ultimate -.->|"produces candidates"| node_blocking
node_ultimate -->|"generates features"| node_pairfeatures
node_ultimate -->|"writes TSVs"| node_matchoutput
node_operator -.->|"runs training"| node_trainer1
node_operator -.->|"runs training"| node_trainer2
node_operator -.->|"runs training"| node_trainer3
node_operator -.->|"runs analysis"| node_thresholds
node_operator -->|"runs inference"| node_smart
node_operator -.->|"runs analysis"| node_groundtruth
node_operator -.->|"runs analysis"| node_exploration
node_operator -.->|"creates mini data"| node_minidata
node_operator -->|"validates submission"| node_validator
node_validator -.->|"checks result format"| node_matchoutput
node_ultimate -.->|"reads task data"| node_dataset
node_validator -->|"checks entity IDs"| node_dataset

click node_ultimate "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py"
click node_power "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_power.py"
click node_fast "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_fast.py"
click node_baseline "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline.py"
click node_blocking "https://github.com/premliorate/amazon-ml-challenge/blob/main/blocking_v2.py"
click node_pairfeatures "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py"
click node_matchoutput "https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py"
click node_trainer1 "https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model.py"
click node_trainer2 "https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model_v2.py"
click node_trainer3 "https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model_v3.py"
click node_thresholds "https://github.com/premliorate/amazon-ml-challenge/blob/main/learn_thresholds.py"
click node_smart "https://github.com/premliorate/amazon-ml-challenge/blob/main/inference_smart.py"
click node_groundtruth "https://github.com/premliorate/amazon-ml-challenge/blob/main/analyze_gt.py"
click node_exploration "https://github.com/premliorate/amazon-ml-challenge/blob/main/eda.py"
click node_minidata "https://github.com/premliorate/amazon-ml-challenge/blob/main/create_mini_dataset.py"
click node_validator "https://github.com/premliorate/amazon-ml-challenge/blob/main/student_resource/utils/validate_submission.py"
click node_dataset "https://github.com/premliorate/amazon-ml-challenge/tree/main/student_resource/dataset"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ultimate,node_power,node_fast,node_baseline toneBlue
class node_blocking,node_pairfeatures,node_matchoutput toneAmber
class node_trainer1,node_trainer2,node_trainer3,node_thresholds,node_smart,node_groundtruth,node_exploration,node_minidata toneMint
class node_validator,node_dataset,node_operator toneRose
```

## Components

### Pipelines

- [Ultimate pipeline](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py): `pipeline_ultimate.py`
- [Power pipeline](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_power.py): `pipeline_power.py`
- [Fast pipeline](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_fast.py): `pipeline_fast.py`
- [Full pipeline](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline.py): `pipeline.py`

### Matching stages

- [Candidate blocking](https://github.com/premliorate/amazon-ml-challenge/blob/main/blocking_v2.py): `blocking_v2.py`
- [Pair features](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py): `pipeline_ultimate.py`
- [Match outputs](https://github.com/premliorate/amazon-ml-challenge/blob/main/pipeline_ultimate.py): `pipeline_ultimate.py`

### Model development

- [Initial trainer](https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model.py): `train_model.py`
- [V2 trainer](https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model_v2.py): `train_model_v2.py`
- [V3 trainer](https://github.com/premliorate/amazon-ml-challenge/blob/main/train_model_v3.py): `train_model_v3.py`
- [Threshold analysis](https://github.com/premliorate/amazon-ml-challenge/blob/main/learn_thresholds.py): `learn_thresholds.py`
- [Smart inference](https://github.com/premliorate/amazon-ml-challenge/blob/main/inference_smart.py): `inference_smart.py`
- [Ground-truth analysis](https://github.com/premliorate/amazon-ml-challenge/blob/main/analyze_gt.py): `analyze_gt.py`
- [Data exploration](https://github.com/premliorate/amazon-ml-challenge/blob/main/eda.py): `eda.py`
- [Mini dataset creation](https://github.com/premliorate/amazon-ml-challenge/blob/main/create_mini_dataset.py): `create_mini_dataset.py`

### Submission quality

- [Submission validator](https://github.com/premliorate/amazon-ml-challenge/blob/main/student_resource/utils/validate_submission.py): `student_resource/utils/validate_submission.py`
- [Task datasets](https://github.com/premliorate/amazon-ml-challenge/tree/main/student_resource/dataset): `student_resource/dataset`

### Other components

- **Task operator**

## Connections

- Task operator → Ultimate pipeline: runs mode (evidence: `README.md`)
- Task operator → Power pipeline: runs pipeline (evidence: `README.md`)
- Task operator → Fast pipeline: runs pipeline (evidence: `README.md`)
- Task operator → Full pipeline: runs pipeline (evidence: `README.md`)
- Ultimate pipeline → Candidate blocking: produces candidates
- Ultimate pipeline → Pair features: generates features (evidence: `pipeline_ultimate.py`)
- Ultimate pipeline → Match outputs: writes TSVs (evidence: `pipeline_ultimate.py`)
- Task operator → Initial trainer: runs training
- Task operator → V2 trainer: runs training
- Task operator → V3 trainer: runs training
- Task operator → Threshold analysis: runs analysis
- Task operator → Smart inference: runs inference (evidence: `inference_smart.py`)
- Task operator → Ground-truth analysis: runs analysis
- Task operator → Data exploration: runs analysis
- Task operator → Mini dataset creation: creates mini data
- Task operator → Submission validator: validates submission (evidence: `student_resource/utils/validate_submission.py`)
- Submission validator → Match outputs: checks result format
- Ultimate pipeline → Task datasets: reads task data
- Submission validator → Task datasets: checks entity IDs (evidence: `student_resource/utils/validate_submission.py`)
