# litz-lab/scarab-infra architecture

> Architecture diagram of the GitHub repository litz-lab/scarab-infra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/litz-lab/scarab-infra
- Repository: https://github.com/litz-lab/scarab-infra

## Overview

##### Core System Components
- Experiment Management:
- The entry point (run.sh) along with various command-line flags (e.g., --simulation, --status, --kill, --run) orchestrates experiment lifecycle management.
- JSON descriptors (in the json/ directory) specify experiment configuration details that the run.sh script uses.
- Docker & Container Infrastructure:
- Multiple Dockerfiles (in common/ and each workloads subfolder) define different container images. Workloads are containerized; images are pulled or built to run specific simulation workloads.
- The “common” directory holds Dockerfiles for different settings (oss, tailbench, etc.) and scripts for container entry initialization.
- Workload & Simulation Components:
- The workloads directory contains various pre-configured workloads (e.g., autoware, dacapo, spec2017) each having its own Dockerfile and sometimes additional source code.
- The screenshot shows that simpoints and execution traces are key: there are scripts (e.g., run_simpoint_trace.py) and JSON configuration files (exp.json, perf.json, trace.json) that control simulation details.
- Analysis & Post-Processing:
- The scarab_stats folder contains scripts and a Jupyter notebook for analyzing simulation statistics.
- Additional Python/R scripts in the common/scripts and scripts directories handle trace processing (e.g., plot_simpoints.py, run_trace_post_processing.sh).

##### Relationships and Interactions
- The user interacts with the system by executing commands (via run.sh) that initiate experiments.
- run.sh, along with auxiliary scripts in “scripts/”, reads and applies the experiment configuration from the JSON files, and then:
- Either pulls a pre-built Docker image from a container registry (with images tagged per git hash) or builds one using the Dockerfiles in “common/”.
- Launches the simulation inside Docker containers, potentially scheduled via Slurm if configured.
- Once the container is running, the simulation executes the Scarab binary (or related scripts in the “scarab_stats” and “fingerprint_src”) processing instructions according to the SimPoint method.
- After simulation completion, post-processing scripts analyze simulation traces and gather performance data.
- The simulation’s runtime results are then available for further evaluation or debugging via an interactive shell (for example, using “run.sh --run”).

##### Architectural Patterns and Technology Stack
- Containerization: Docker is used extensively to ensure consistency across simulation environments.
- Distributed Job Management: The integration with Slurm indicates design for parallel and distributed experiment execution.
- Modular Design: Distinct directories for workloads, common utilities/scripts, and analysis promote separation of concerns.
- Experiment Descriptor-Driven Flow: JSON configuration files decouple the experiment parameters from the execution scripts.
- Workflow Automation: GitHub Actions workflows (in .github/workflows) hint at continuous integration for image validation and possibly testing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["User CLI (run.sh)"]:::cli
    B["Experiment Manager (JSON Config & run.sh)"]:::manager
    C["Docker Container Manager (common/ & workloads/)"]:::container
    D["Job Scheduler (scripts/slurm_runner.py)"]:::scheduler
    E["Simulation Execution (fingerprint_src/ & Workload Sim)"]:::simulation
    F["Results & Analysis (scarab_stats/ & post-processing scripts)"]:::analysis

    A -->|"initiates"| B
    B -->|"triggers"| C
    B -->|"schedules"| D
    D -->|"orchestrates"| C
    C -->|"runs simulation"| E
    E -->|"produces results"| F

    click A "https://github.com/litz-lab/scarab-infra/blob/main/run.sh"
    click B "https://github.com/litz-lab/scarab-infra/tree/main/json/"
    click C "https://github.com/litz-lab/scarab-infra/tree/main/common/"
    click D "https://github.com/litz-lab/scarab-infra/blob/main/scripts/slurm_runner.py"
    click E "https://github.com/litz-lab/scarab-infra/tree/main/fingerprint_src/"
    click F "https://github.com/litz-lab/scarab-infra/tree/main/scarab_stats/"

    classDef cli fill:#f9c,stroke:#333,stroke-width:2px,color:#000;
    classDef manager fill:#bbf,stroke:#333,stroke-width:2px,color:#000;
    classDef container fill:#cfc,stroke:#333,stroke-width:2px,color:#000;
    classDef scheduler fill:#fc9,stroke:#333,stroke-width:2px,color:#000;
    classDef simulation fill:#cff,stroke:#333,stroke-width:2px,color:#000;
    classDef analysis fill:#fdd,stroke:#333,stroke-width:2px,color:#000;
```
