How boryoungkim/sdv-bmt works
SDV-BMT is a model-level accelerator benchmarking system: it validates a planned run, executes a workload against a device under test, records append-only evidence, then computes metrics and validity results from that evidence. The Python control plane coordinates plans, inputs, workloads and evaluation; the C++ native plane records realtime facts, with fake and NVIDIA backends. The README documents the separation and major capabilities; some internal links below are therefore documented rather than verified from sampled implementation.
Main entry points: runner.py Benchmark Runner · main.py CLI · native.py Native DUT Adapter · bundle.py Evidence Bundle
Components 21
Plan and Control
- CLI
src/sdv_bmt/cli/main.py - Plan Validation
src/sdv_bmt/runner/plan.py - Run Contracts
src/sdv_bmt/contracts/specs.py
Run Execution
- Input Fixtures
src/sdv_bmt/input/dataset_fixture.py - Benchmark Runner
src/sdv_bmt/runner/runner.py - Request Schedule
src/sdv_bmt/runner/schedule.py - Stress Control
src/sdv_bmt/stress/lifecycle.py
Native Measurement
- Fake DUT
src/sdv_bmt/dut/fake.py - Native DUT Adapter
src/sdv_bmt/dut/native.py - NVIDIA Adapter
src/sdv_bmt/dut/nvidia/adapter.py - Native Measurement
native/src/common/loop.hpp - Output Pipelines
native/src/common/output_pipeline.hpp
Evidence and Evaluation
- Evidence Bundle
src/sdv_bmt/evidence/bundle.py - Bundle Reader
src/sdv_bmt/observation/reader.py - Metric Evaluation
src/sdv_bmt/metrics/metric_set_v1.py - Task Quality
src/sdv_bmt/metrics/quality_segmentation.py - Evidence Validity
src/sdv_bmt/evidence/validity.py
Workloads
- Model Workloads
src/sdv_bmt/workload/yolox.py - Segmentation Workload
src/sdv_bmt/workload/deeplab.py
Other components
- Benchmark Engineer
- NVIDIA GPU
Connections 21
- Benchmark Engineer → CLI: runs commands
- CLI → Plan Validation: validates plan
src/sdv_bmt/cli/main.py - Plan Validation → Run Contracts: parses specs
src/sdv_bmt/runner/plan.py - CLI → Benchmark Runner: starts run
src/sdv_bmt/cli/main.py - Benchmark Runner → Input Fixtures: loads items
src/sdv_bmt/runner/runner.py - Benchmark Runner → Request Schedule: issues requests
src/sdv_bmt/runner/runner.py - Benchmark Runner → Stress Control: controls lifecycle
src/sdv_bmt/runner/runner.py - Benchmark Runner → Model Workloads: runs workload
- Benchmark Runner → Segmentation Workload: runs workload
- Benchmark Runner → Fake DUT: uses adapter
- Benchmark Runner → Native DUT Adapter: uses adapter
- Native DUT Adapter → Native Measurement: invokes measurement
- Native Measurement → Output Pipelines: encodes outputs
native/src/common/output_pipeline.hpp - Native DUT Adapter → NVIDIA Adapter: selects backend
- NVIDIA Adapter → NVIDIA GPU: measures on
- Benchmark Runner → Evidence Bundle: writes evidence
src/sdv_bmt/runner/runner.py - CLI → Evidence Bundle: evaluates run
src/sdv_bmt/cli/main.py - Evidence Bundle → Bundle Reader: reads bundle
- Bundle Reader → Metric Evaluation: supplies observations
src/sdv_bmt/metrics/metric_set_v1.py - Metric Evaluation → Task Quality: computes quality
src/sdv_bmt/metrics/metric_set_v1.py - CLI → Evidence Validity: checks evidence