# gonzalorodrigo/scsfworkload architecture

> Architecture diagram of the GitHub repository gonzalorodrigo/scsfworkload, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gonzalorodrigo/scsfworkload
- Repository: https://github.com/gonzalorodrigo/scsfworkload

## Overview

##### Core Project Components
- Experiment Definition & Orchestration:
– The orchestration module (orchestration/) defines experiments, including single and group experiments.
– The experiment definition class (found in orchestration/ExperimentDefinition and related methods) drives the lifecycle of an experiment (fresh, pre_sim, simulating, completed, pre_analysis, analysis done, etc.).
- Workload & System Modeling:
– The machines/ folder holds the machine models (e.g., Edison2015 or similar) that define HPC system characteristics.
– The generate/ package is used for workload generation and even for generating specific workflow or special cases.
- Simulation Execution:
– The bin/ directory includes various scripts to run simulations (run_sim_exp.py) and control worker processes (workers_launch.sh, workers_kill.sh, etc.).
– The slurm/ module integrates Slurm configuration and trace generation.
- Data Management & Utility Libraries:
– The commonLib/ includes critical utilities like DBManager, Logging, and task handling, which indicate the centrality of a MySQL-based database for storing experiment state and results.
- Analysis and Plotting:
– The analysis/ and stats/ directories contain modules for analyzing simulation results.
– The plot/ folder along with many bin/plot_* scripts produce visualizations of experiment metrics.
- Workflow Support:
– The workflows/ folder plus workflow-related manifests in bin/manifests and XML/JSON converters (bin/xml2json.py) suggest a dedicated sub-system for workflow scheduling experiments.
- Testing & Tools:
– The test/ folder ensures integrity, while tools/ (with SSH utilities) hints at operational and connectivity considerations (e.g., SSH/worker management).

##### Key Interactions and Data Flow
- A “Workload Generator/Machine Model” component then uses these definitions to create or load models (using generate/ and machines/ modules) that represent the system’s jobs and hardware.
- The “Simulation Engine” (primarily driven by scripts in bin/ and the slurm/ module) receives these parameters, executes the scheduling simulation, possibly on remote workers coordinated via scripts in bin/workers_*.
- As simulations run, output trace files and experiment metrics are fed into a MySQL Database (scripted in bin/sql_createdb.sh, bin/sql_populate_db.py, and connected via commonLib/DBManager).
- Post-simulation, the “Analysis Engine” (modules in analysis/ and stats/) pulls results from the database, processes them (computing histograms, CDFs, throughput, slowdown, etc.), and prepares data for visualization.
- Finally, the “Visualization” component (plot/ and several plot_examples scripts in bin/plot_examples) generates comparative plots, histograms, and charts for the user.
- Additionally, note that the orchestration of experiment redos (delta, grouped experiments, and second-pass analysis) implies a cyclic workflow that revisits simulation results for deeper insights.

##### Important Architectural Patterns and Design Principles
- Modular Design: Separate packages for workload generation, simulation execution, analysis, and plotting indicate a clear separation of concerns.
- Pipeline Flow: The experiment lifecycle—from definition through simulation to analysis—is sequential yet loosely coupled, enabling independent testing and development.
- Worker Coordination: The use of SSH commands and dedicated worker management scripts shows distributed execution, important for scaling experiments.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: Client/Configuration
    subgraph "Client/Configuration"
        O1["Experiment Definition"]:::orchestration
        O2["Experiment Running"]:::orchestration
        O3["Experiment Analyzing"]:::orchestration
    end

    %% Middle Layer: Simulation Engine and Worker Management
    subgraph "Simulation Engine Layer"
        subgraph "Workload & System Modeling"
            MM["Machine Models"]:::simulation
            WG["Workload Generation"]:::simulation
        end
        subgraph "Simulation Execution"
            SR["Simulation Runner"]:::simulation
            SI["Slurm Integration"]:::simulation
        end
        subgraph "Worker Management"
            WMG["Worker Manager"]:::simulation
            TEST["Test Suite"]:::testing
        end
    end

    %% Lower Layer: Data Persistence and Analytics
    subgraph "Data Persistence and Analytics"
        DB["Results Database"]:::database
        AE["Analysis Engine"]:::analytics
        VM["Visualization Module"]:::analytics
        WF["Workflow Support"]:::analytics
    end

    %% Flow Connections
    O1 -->|"defines"| MM
    O1 -->|"defines"| WG
    O2 -->|"triggers"| SR
    SR -->|"generatesData"| DB
    SI -->|"providesTraces"| DB
    DB -->|"feedsData"| AE
    AE -->|"analyzes"| VM
    O1 -->|"configures"| WF
    SR -->|"managesWorkers"| WMG

    %% Click Events
    click O1 "https://github.com/gonzalorodrigo/scsfworkload/blob/master/orchestration/definition.py"
    click O2 "https://github.com/gonzalorodrigo/scsfworkload/blob/master/orchestration/running.py"
    click O3 "https://github.com/gonzalorodrigo/scsfworkload/blob/master/orchestration/analyzing.py"
    click MM "https://github.com/gonzalorodrigo/scsfworkload/tree/master/machines/"
    click WG "https://github.com/gonzalorodrigo/scsfworkload/tree/master/generate/"
    click SR "https://github.com/gonzalorodrigo/scsfworkload/blob/master/bin/run_sim_exp.py"
    click SI "https://github.com/gonzalorodrigo/scsfworkload/blob/master/slurm/trace_gen.py"
    click DB "https://github.com/gonzalorodrigo/scsfworkload/blob/master/commonLib/DBManager.py"
    click AE "https://github.com/gonzalorodrigo/scsfworkload/blob/master/analysis/jobAnalysis.py"
    click VM "https://github.com/gonzalorodrigo/scsfworkload/blob/master/plot/print_data.py"
    click WF "https://github.com/gonzalorodrigo/scsfworkload/tree/master/workflows/"
    click WMG "https://github.com/gonzalorodrigo/scsfworkload/blob/master/tools/ssh.py"
    click TEST "https://github.com/gonzalorodrigo/scsfworkload/tree/master/test/"

    %% Styles for nodes
    classDef orchestration fill:#AED6F1,stroke:#21618C,stroke-width:2px;
    classDef simulation fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef database fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef analytics fill:#FAD7A0,stroke:#CA6F1E,stroke-width:2px;
    classDef testing fill:#EBDEF0,stroke:#8E44AD,stroke-width:2px;
```
