How nirmalkumar31/aml-evaluation-harness works
AML evaluation harness turns synthetic AMLworld transactions into reproducible, budget-constrained model evidence, then publishes verified aggregate results in an interactive site. The main workflow is ingest and label preparation, temporal/ring-aware splitting, feature construction, per-seed fitting and account-day evaluation, followed by provenance and publication checks. Separate diagnostic branches cover drift and leakage. The site exposes archived results and engineering explanations. Source excerpts and the index establish several module references; README statements support the broader pipeline. Unsampled subsystems and some data handoffs are shown only where documented.
Main entry points: cli.py CLI orchestration · train.py Model fitting · build.py Feature construction · io.py Path and artifact I/O
Components 27
Data preparation
- CLI orchestration
aml-platform/src/aml/cli.py - Data contract
aml-platform/src/aml/schema.py - Transaction ingest
aml-platform/src/aml/ingest/normalize.py - Pattern labels
aml-platform/src/aml/patterns - Ring-aware split
aml-platform/src/aml/splits/ring_aware.py - Feature construction
aml-platform/src/aml/features/build.py - Path and artifact I/O
aml-platform/src/aml/io.py
Evaluation
- Model fitting
aml-platform/src/aml/models/train.py - Model configuration
aml-platform/src/aml/models/config.py - Budget metrics
aml-platform/src/aml/eval/metrics.py - Evaluation runner
aml-platform/src/aml/eval/run.py - Random-ranker null
aml-platform/scripts/budget_null.py - Seed stability
aml-platform/src/aml/models/stability.py - Leakage controls
aml-platform/src/aml/leakproof/plant.py - Leakage sweep
aml-platform/src/aml/leakproof/sweep.py - Drift experiments
aml-platform/src/aml/drift
Evidence and publication
- Run provenance
aml-platform/src/aml/manifest.py - Aggregate reporting
aml-platform/scripts/make_tables.py - Publication verification
aml-platform/scripts/release_facts.py
Results website
- Site data and pages
site - Asset registry
site/assets.py - Results explorer
site/js/explorer.js - UI utilities
site/js/util.js - Explorer page
site/js/page-explorer.js - Engineering page
site/js/page-engineering.js
Other components
- Harness user
- Site visitor
Connections 38
- Harness user → CLI orchestration: runs commands
aml-platform/src/aml/cli.py - CLI orchestration → Transaction ingest: dispatches ingest
aml-platform/src/aml/cli.py - CLI orchestration → Pattern labels: dispatches labels
aml-platform/src/aml/cli.py - CLI orchestration → Ring-aware split: dispatches split
aml-platform/src/aml/cli.py - CLI orchestration → Feature construction: dispatches features
aml-platform/src/aml/cli.py - CLI orchestration → Model fitting: dispatches fit
aml-platform/src/aml/cli.py - CLI orchestration → Evaluation runner: dispatches evaluation
aml-platform/src/aml/cli.py - Transaction ingest → Data contract: validates against
aml-platform/src/aml/ingest/normalize.py - Feature construction → Data contract: uses contract
aml-platform/src/aml/features/build.py - Transaction ingest → Path and artifact I/O: uses I/O
aml-platform/src/aml/ingest/normalize.py - Feature construction → Path and artifact I/O: uses I/O
aml-platform/src/aml/features/build.py - Ring-aware split → Path and artifact I/O: uses I/O
aml-platform/src/aml/splits/ring_aware.py - Pattern labels → Path and artifact I/O: uses I/O
aml-platform/src/aml/patterns/parse.py - Model fitting → Feature construction: uses feature set
aml-platform/src/aml/models/train.py - Model fitting → Model configuration: uses configuration
aml-platform/src/aml/models/train.py - Model fitting → Budget metrics: calculates metrics
aml-platform/src/aml/models/train.py - Evaluation runner → Model fitting: runs model
aml-platform/src/aml/eval/run.py - Evaluation runner → Budget metrics: evaluates scores
aml-platform/src/aml/eval/run.py - Evaluation runner → Feature construction: builds features
aml-platform/src/aml/eval/run.py - Random-ranker null → Data contract: checks contract
aml-platform/scripts/budget_null.py - Random-ranker null → Run provenance: records provenance
aml-platform/scripts/budget_null.py - Seed stability → Model fitting: fits seeds
aml-platform/src/aml/models/stability.py - Seed stability → Budget metrics: measures variance
aml-platform/src/aml/models/stability.py - Leakage sweep → Leakage controls: uses controls
aml-platform/src/aml/leakproof/sweep.py - Leakage sweep → Budget metrics: evaluates arms
aml-platform/src/aml/leakproof/sweep.py - Leakage sweep → Feature construction: uses features
aml-platform/src/aml/leakproof/sweep.py - Drift experiments → Path and artifact I/O: reads and writes
aml-platform/src/aml/drift/resample.py - Drift experiments → Run provenance: records runs
aml-platform/src/aml/drift/resample.py - Run provenance → Path and artifact I/O: uses I/O
aml-platform/src/aml/manifest.py - Aggregate reporting → Run provenance: summarizes runs
aml-platform/scripts/release_facts.py - Publication verification → Aggregate reporting: checks aggregates
aml-platform/scripts/release_facts.py - Aggregate reporting → Site data and pages: supplies published results
README.md - Site data and pages → Asset registry: uses asset registry
site/assemble_site.py - Explorer page → Results explorer: initializes explorer
site/js/page-explorer.js - Explorer page → UI utilities: uses UI helpers
site/js/page-explorer.js - Results explorer → UI utilities: renders with
site/js/explorer.js - Engineering page → Site data and pages: presents engineering content
site/js/page-engineering.js - Site visitor → Explorer page: explores results