How litz-lab/scarab-infra works
Core System Components
• Experiment Management:
• The entry point (run.sh) along with various command-line flags (e.g., --simulation, --status, --kill, --run) orchestrates experiment lifecycle management.
• JSON descriptors (in the json/ directory) specify experiment configuration details that the run.sh script uses.
• Docker & Container Infrastructure:
• Multiple Dockerfiles (in common/ and each workloads subfolder) define different container images. Workloads are containerized; images are pulled or built to run specific simulation workloads.
• The “common” directory holds Dockerfiles for different settings (oss, tailbench, etc.) and scripts for container entry initialization.
• Workload & Simulation Components:
• The workloads directory contains various pre-configured workloads (e.g., autoware, dacapo, spec2017) each having its own Dockerfile and sometimes additional source code.
• The screenshot shows that simpoints and execution traces are key: there are scripts (e.g., run_simpoint_trace.py) and JSON configuration files (exp.json, perf.json, trace.json) that control simulation details.
• Analysis & Post-Processing:
• The scarab_stats folder contains scripts and a Jupyter notebook for analyzing simulation statistics.
• Additional Python/R scripts in the common/scripts and scripts directories handle trace processing (e.g., plot_simpoints.py, run_trace_post_processing.sh).
Relationships and Interactions
• The user interacts with the system by executing commands (via run.sh) that initiate experiments.
• run.sh, along with auxiliary scripts in “scripts/”, reads and applies the experiment configuration from the JSON files, and then:
• Either pulls a pre-built Docker image from a container registry (with images tagged per git hash) or builds one using the Dockerfiles in “common/”.
• Launches the simulation inside Docker containers, potentially scheduled via Slurm if configured.
• Once the container is running, the simulation executes the Scarab binary (or related scripts in the “scarab_stats” and “fingerprint_src”) processing instructions according to the SimPoint method.
• After simulation completion, post-processing scripts analyze simulation traces and gather performance data.
• The simulation’s runtime results are then available for further evaluation or debugging via an interactive shell (for example, using “run.sh --run”).
Architectural Patterns and Technology Stack
• Containerization: Docker is used extensively to ensure consistency across simulation environments.
• Distributed Job Management: The integration with Slurm indicates design for parallel and distributed experiment execution.
• Modular Design: Distinct directories for workloads, common utilities/scripts, and analysis promote separation of concerns.
• Experiment Descriptor-Driven Flow: JSON configuration files decouple the experiment parameters from the execution scripts.
• Workflow Automation: GitHub Actions workflows (in .github/workflows) hint at continuous integration for image validation and possibly testing.