How krai/kilt-mlperf works
Project Type and Purpose
• This is an open-source C++ inference library and benchmarking framework (not a web app).
• Core purpose: provide a flexible, high-performance ML inference engine (KILT) with pluggable devices, models, data sources, configuration tools, and benchmarks for MLPerf.
File Structure
Top‐level directories:
– benchmarks/… : benchmarking harnesses (network & standalone) for various ML tasks (BERT, classification, object-detection)
– config/… : configuration abstractions and parsers (env, JSON, translation tables)
– devices/… : device back ends (dummy, ONNX) implementing the device interface
– plugins/… : optional algorithmic extensions (e.g., NMS-ABP)
Top‐level interfaces and core:
– iconfig.h, idatasource.h, idevice.h, imodel.h : abstract interfaces
– kilt.h / kilt_impl.h / kilt_utils.h : core API and implementation
– affinity.h, tracing.h : cross-cutting utilities (thread affinity, instrumentation)
README
• Confirms it’s a C++ inference library for MLPerf Inference.
• MIT-licensed, intended for integration into larger toolchains.
Main Components & Their Relationships
A. Core API Layer
– kilt.h / kilt_impl.h: Public API entrypoints (initialize, load model, run inference)
– Utilities: affinity.h, tracing.h
B. Interfaces / Abstractions
– IDevice: device plug-in contract
– IModel: model loader/executor contract
– IDataSource: pre/post-processing contract
– IConfig: configuration loader contract
C. Config Subsystem
– config_tools (env, JSON, translation) → implements IConfig
– config/kilt_config.h: high-level library settings
D. Device Subsystem
– devices/dummy and devices/onnx → implementations of IDevice
– Each has device_config, translation tables, device.h
E. Plugin Subsystem
– plugins/nms-abp: implements inference post-processing outside core (implements IModel or IDevice hooks)
F. Benchmarking Harness
– benchmarks/…/standalone and benchmarks/…/network
– Harness code uses Core API, loads devices/models/config, runs timing loops, reports results
Architectural Patterns & Principles
• Interface-based design for extensibility (IDevice, IModel, IDataSource).
• Plugin architecture: dynamic or static registration of device back ends and algorithms.
• Separation of concerns: config, core, device, model, data, benchmarks split into their own subsystems.
• Dependency inversion: core depends on interfaces, concrete implementations bound at runtime.
– Notes: annotate critical interactions (e.g., “IDevice::executeInference()”)