# nirmalkumar31/aml-evaluation-harness architecture

> Architecture diagram of the GitHub repository nirmalkumar31/aml-evaluation-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nirmalkumar31/aml-evaluation-harness
- Repository: https://github.com/nirmalkumar31/aml-evaluation-harness
- Video tour (about a minute): https://gitdiagram.com/nirmalkumar31/aml-evaluation-harness/video

## Overview

AML evaluation harness turns synthetic AMLworld transactions into reproducible, budget-constrained model evidence, then publishes verified aggregate results in an interactive site. The main workflow is ingest and label preparation, temporal/ring-aware splitting, feature construction, per-seed fitting and account-day evaluation, followed by provenance and publication checks. Separate diagnostic branches cover drift and leakage. The site exposes archived results and engineering explanations. Source excerpts and the index establish several module references; README statements support the broader pipeline. Unsampled subsystems and some data handoffs are shown only where documented.

Main entry points: [`aml-platform/src/aml/cli.py`](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/cli.py) (CLI orchestration), [`aml-platform/src/aml/models/train.py`](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/train.py) (Model fitting), [`aml-platform/src/aml/features/build.py`](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/features/build.py) (Feature construction), [`aml-platform/src/aml/io.py`](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/io.py) (Path and artifact I/O)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_cli["CLI orchestration<br/>[cli.py]"]
  node_schema["Data contract<br/>[schema.py]"]
  node_normalize["Transaction ingest<br/>[normalize.py]"]
  node_patterns["Pattern labels"]
  node_split["Ring-aware split<br/>[ring_aware.py]"]
  node_features["Feature construction<br/>[build.py]"]
  node_storage["Path and artifact I/O<br/>[io.py]"]
end

subgraph group_analysis["Evaluation"]
  node_model["Model fitting<br/>[train.py]"]
  node_config["Model configuration<br/>[config.py]"]
  node_evaluation["Budget metrics<br/>[metrics.py]"]
  node_evalrun["Evaluation runner<br/>[run.py]"]
  node_null["Random-ranker null<br/>[budget_null.py]"]
  node_stability["Seed stability<br/>[stability.py]"]
  node_leakplant["Leakage controls<br/>[plant.py]"]
  node_leaksweep["Leakage sweep<br/>[sweep.py]"]
  node_drift["Drift experiments"]
end

subgraph group_evidence["Evidence and publication"]
  node_manifest["Run provenance<br/>[manifest.py]"]
  node_tables["Aggregate reporting<br/>[make_tables.py]"]
  node_gate["Publication verification<br/>[release_facts.py]"]
end

subgraph group_website["Results website"]
  node_sitebuild["Site data and pages"]
  node_assets["Asset registry<br/>[assets.py]"]
  node_explorer["Results explorer<br/>[explorer.js]"]
  node_siteutil["UI utilities<br/>[util.js]"]
  node_sitepage["Explorer page<br/>[page-explorer.js]"]
  node_engineerpage["Engineering page"]
end

node_operator(("Harness user"))
node_visitor(("Site visitor"))

node_operator -->|"runs commands"| node_cli
node_cli -->|"dispatches ingest"| node_normalize
node_cli -->|"dispatches labels"| node_patterns
node_cli -->|"dispatches split"| node_split
node_cli -->|"dispatches features"| node_features
node_cli -->|"dispatches fit"| node_model
node_cli -->|"dispatches evaluation"| node_evalrun
node_normalize -->|"validates against"| node_schema
node_features -->|"uses contract"| node_schema
node_normalize -->|"uses I/O"| node_storage
node_features -->|"uses I/O"| node_storage
node_split -->|"uses I/O"| node_storage
node_patterns -->|"uses I/O"| node_storage
node_model -->|"uses feature set"| node_features
node_model -->|"uses configuration"| node_config
node_model -->|"calculates metrics"| node_evaluation
node_evalrun -->|"runs model"| node_model
node_evalrun -->|"evaluates scores"| node_evaluation
node_evalrun -->|"builds features"| node_features
node_null -->|"checks contract"| node_schema
node_null -->|"records provenance"| node_manifest
node_stability -->|"fits seeds"| node_model
node_stability -->|"measures variance"| node_evaluation
node_leaksweep -->|"uses controls"| node_leakplant
node_leaksweep -->|"evaluates arms"| node_evaluation
node_leaksweep -->|"uses features"| node_features
node_drift -->|"reads and writes"| node_storage
node_drift -->|"records runs"| node_manifest
node_manifest -->|"uses I/O"| node_storage
node_tables -->|"summarizes runs"| node_manifest
node_gate -->|"checks aggregates"| node_tables
node_tables -.->|"supplies published results"| node_sitebuild
node_sitebuild -->|"uses asset registry"| node_assets
node_sitepage -->|"initializes explorer"| node_explorer
node_sitepage -->|"uses UI helpers"| node_siteutil
node_explorer -->|"renders with"| node_siteutil
node_engineerpage -.->|"presents engineering content"| node_sitebuild
node_visitor -.->|"explores results"| node_sitepage

click node_cli "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/cli.py"
click node_schema "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/schema.py"
click node_normalize "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/ingest/normalize.py"
click node_patterns "https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/aml-platform/src/aml/patterns"
click node_split "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/splits/ring_aware.py"
click node_features "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/features/build.py"
click node_storage "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/io.py"
click node_model "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/train.py"
click node_config "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/config.py"
click node_evaluation "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/eval/metrics.py"
click node_evalrun "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/eval/run.py"
click node_null "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/budget_null.py"
click node_stability "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/stability.py"
click node_leakplant "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/leakproof/plant.py"
click node_leaksweep "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/leakproof/sweep.py"
click node_drift "https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/aml-platform/src/aml/drift"
click node_manifest "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/manifest.py"
click node_tables "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/make_tables.py"
click node_gate "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/release_facts.py"
click node_sitebuild "https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/site"
click node_assets "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/assets.py"
click node_explorer "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/explorer.js"
click node_siteutil "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/util.js"
click node_sitepage "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/page-explorer.js"
click node_engineerpage "https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/page-engineering.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_schema,node_normalize,node_patterns,node_split,node_features,node_storage,node_operator toneBlue
class node_model,node_config,node_evaluation,node_evalrun,node_null,node_stability,node_leakplant,node_leaksweep,node_drift toneAmber
class node_manifest,node_tables,node_gate toneMint
class node_sitebuild,node_assets,node_explorer,node_siteutil,node_sitepage,node_engineerpage toneRose
class node_visitor toneIndigo
```

## Components

### Data preparation

- [CLI orchestration](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/cli.py): `aml-platform/src/aml/cli.py`
- [Data contract](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/schema.py): `aml-platform/src/aml/schema.py`
- [Transaction ingest](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/ingest/normalize.py): `aml-platform/src/aml/ingest/normalize.py`
- [Pattern labels](https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/aml-platform/src/aml/patterns): `aml-platform/src/aml/patterns`
- [Ring-aware split](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/splits/ring_aware.py): `aml-platform/src/aml/splits/ring_aware.py`
- [Feature construction](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/features/build.py): `aml-platform/src/aml/features/build.py`
- [Path and artifact I/O](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/io.py): `aml-platform/src/aml/io.py`

### Evaluation

- [Model fitting](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/train.py): `aml-platform/src/aml/models/train.py`
- [Model configuration](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/config.py): `aml-platform/src/aml/models/config.py`
- [Budget metrics](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/eval/metrics.py): `aml-platform/src/aml/eval/metrics.py`
- [Evaluation runner](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/eval/run.py): `aml-platform/src/aml/eval/run.py`
- [Random-ranker null](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/budget_null.py): `aml-platform/scripts/budget_null.py`
- [Seed stability](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/models/stability.py): `aml-platform/src/aml/models/stability.py`
- [Leakage controls](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/leakproof/plant.py): `aml-platform/src/aml/leakproof/plant.py`
- [Leakage sweep](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/leakproof/sweep.py): `aml-platform/src/aml/leakproof/sweep.py`
- [Drift experiments](https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/aml-platform/src/aml/drift): `aml-platform/src/aml/drift`

### Evidence and publication

- [Run provenance](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/src/aml/manifest.py): `aml-platform/src/aml/manifest.py`
- [Aggregate reporting](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/make_tables.py): `aml-platform/scripts/make_tables.py`
- [Publication verification](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/aml-platform/scripts/release_facts.py): `aml-platform/scripts/release_facts.py`

### Results website

- [Site data and pages](https://github.com/nirmalkumar31/aml-evaluation-harness/tree/main/site): `site`
- [Asset registry](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/assets.py): `site/assets.py`
- [Results explorer](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/explorer.js): `site/js/explorer.js`
- [UI utilities](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/util.js): `site/js/util.js`
- [Explorer page](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/page-explorer.js): `site/js/page-explorer.js`
- [Engineering page](https://github.com/nirmalkumar31/aml-evaluation-harness/blob/main/site/js/page-engineering.js): `site/js/page-engineering.js`

### Other components

- **Harness user**
- **Site visitor**

## Connections

- Harness user → CLI orchestration: runs commands (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Transaction ingest: dispatches ingest (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Pattern labels: dispatches labels (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Ring-aware split: dispatches split (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Feature construction: dispatches features (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Model fitting: dispatches fit (evidence: `aml-platform/src/aml/cli.py`)
- CLI orchestration → Evaluation runner: dispatches evaluation (evidence: `aml-platform/src/aml/cli.py`)
- Transaction ingest → Data contract: validates against (evidence: `aml-platform/src/aml/ingest/normalize.py`)
- Feature construction → Data contract: uses contract (evidence: `aml-platform/src/aml/features/build.py`)
- Transaction ingest → Path and artifact I/O: uses I/O (evidence: `aml-platform/src/aml/ingest/normalize.py`)
- Feature construction → Path and artifact I/O: uses I/O (evidence: `aml-platform/src/aml/features/build.py`)
- Ring-aware split → Path and artifact I/O: uses I/O (evidence: `aml-platform/src/aml/splits/ring_aware.py`)
- Pattern labels → Path and artifact I/O: uses I/O (evidence: `aml-platform/src/aml/patterns/parse.py`)
- Model fitting → Feature construction: uses feature set (evidence: `aml-platform/src/aml/models/train.py`)
- Model fitting → Model configuration: uses configuration (evidence: `aml-platform/src/aml/models/train.py`)
- Model fitting → Budget metrics: calculates metrics (evidence: `aml-platform/src/aml/models/train.py`)
- Evaluation runner → Model fitting: runs model (evidence: `aml-platform/src/aml/eval/run.py`)
- Evaluation runner → Budget metrics: evaluates scores (evidence: `aml-platform/src/aml/eval/run.py`)
- Evaluation runner → Feature construction: builds features (evidence: `aml-platform/src/aml/eval/run.py`)
- Random-ranker null → Data contract: checks contract (evidence: `aml-platform/scripts/budget_null.py`)
- Random-ranker null → Run provenance: records provenance (evidence: `aml-platform/scripts/budget_null.py`)
- Seed stability → Model fitting: fits seeds (evidence: `aml-platform/src/aml/models/stability.py`)
- Seed stability → Budget metrics: measures variance (evidence: `aml-platform/src/aml/models/stability.py`)
- Leakage sweep → Leakage controls: uses controls (evidence: `aml-platform/src/aml/leakproof/sweep.py`)
- Leakage sweep → Budget metrics: evaluates arms (evidence: `aml-platform/src/aml/leakproof/sweep.py`)
- Leakage sweep → Feature construction: uses features (evidence: `aml-platform/src/aml/leakproof/sweep.py`)
- Drift experiments → Path and artifact I/O: reads and writes (evidence: `aml-platform/src/aml/drift/resample.py`)
- Drift experiments → Run provenance: records runs (evidence: `aml-platform/src/aml/drift/resample.py`)
- Run provenance → Path and artifact I/O: uses I/O (evidence: `aml-platform/src/aml/manifest.py`)
- Aggregate reporting → Run provenance: summarizes runs (evidence: `aml-platform/scripts/release_facts.py`)
- Publication verification → Aggregate reporting: checks aggregates (evidence: `aml-platform/scripts/release_facts.py`)
- Aggregate reporting → Site data and pages: supplies published results (evidence: `README.md`)
- Site data and pages → Asset registry: uses asset registry (evidence: `site/assemble_site.py`)
- Explorer page → Results explorer: initializes explorer (evidence: `site/js/page-explorer.js`)
- Explorer page → UI utilities: uses UI helpers (evidence: `site/js/page-explorer.js`)
- Results explorer → UI utilities: renders with (evidence: `site/js/explorer.js`)
- Engineering page → Site data and pages: presents engineering content (evidence: `site/js/page-engineering.js`)
- Site visitor → Explorer page: explores results
