How facebookresearch/dcperf works
Project Type and Purpose
• DCPerf is a CLI-driven, plugin-based performance benchmark suite for datacenter workloads.
• It provides a “driver” (Benchpress) in Python to install, orchestrate, execute, monitor, parse, and report results of various real-world-style benchmarks (Mediawiki, FeedSim, TaoBench, etc.).
Main Architectural Components
A. CLI Layer
• Entry point: benchpress_cli.py
• Command definitions under benchpress/cli (commands: list, install, run, report, system_check, etc.)
• Uses Click for dispatching subcommands.
B. Core Framework
• benchpress/lib – implements core abstractions: Job, Factory, Hooks, Parsers, Reporter, Utilities
• benchpress/config – YAML job/benchmark definitions (benchmarks.yml, jobs.yml, toolchain.yml)
• benchpress/logging_config and benchpress/version
C. Plugin Subsystem
• Hooks (benchpress/plugins/hooks) – instrumentation and monitoring plugins (e.g., perf, cpu_limit, memstat)
• Parsers (benchpress/plugins/parsers) – parsers for raw benchmark output into structured metrics
• Each plugin is discovered and instantiated via HookFactory and ParserFactory
D. Benchmark Packages
• packages/… – per-benchmark install/build/run scripts and source (e.g., packages/feedsim, packages/django_workload, packages/tao_bench)
• Each contains shell scripts (install/run/cleanup), patches, templates, and third-party code
E. External Environment
• System shell, Python interpreter, OS packages (dnf/apt), compilers, third-party libraries (folly, FBThrift, Django, Cassandra, etc.)
• GitHub CI workflows and provisioning scripts under .github/
Relationships and Data Flow
• User invokes benchpress_cli.py → Click dispatches to Command class
• Command loads job definitions from benchpress/config/*.yml
• For install: calls the install script in packages/<job> via subprocess
• For run:
1) Command instantiates Job object (benchpress/lib/job.py)
2) Job object consults HookFactory to attach monitoring hooks
3) Job runs benchmark executables (from packages) under Hook wrappers
4) Raw stdout/log files are fed to ParserFactory
5) Parser produces structured JSON metrics
6) Reporter aggregates metrics, writes results to benchmark_metrics_<run_id>
• For report: Command invokes Reporter to compute per-benchmark and overall DCPerf scores
• system_check and health_check are special Jobs that collect system specs and micro-benchmarks via lib/sys_specs.py and health_check package
Architectural Patterns and Principles
• CLI Command Pattern (Click) for pluggable commands
• Plugin Architecture with Factory pattern: HookFactory, ParserFactory, ReporterFactory
• Separation of Concerns:
• CLI layer only parses args & dispatches
• Core library contains business logic
• Plugins encapsulate extensibility (monitoring, parsing)
• Packages isolate benchmark definitions and build scripts
• Config-driven: YAML files define what jobs exist and their roles
Technologies and Frameworks
• Python 3.x, Click, PyYAML, Pandas, Tabulate
• Bash shell scripts for environment setup & package installs
• C++ toolchains for FeedSim, TaoBench, VideoTranscodeBench
• Third-party libs: Folly, FBThrift, OpenSSL, Cassandra, Django, HHVM, Memcached, Spark, ffmpeg, etc.