How gonzalorodrigo/scsfworkload works
Core Project Components
• Experiment Definition & Orchestration:
– The orchestration module (orchestration/) defines experiments, including single and group experiments.
– The experiment definition class (found in orchestration/ExperimentDefinition and related methods) drives the lifecycle of an experiment (fresh, pre_sim, simulating, completed, pre_analysis, analysis done, etc.).
• Workload & System Modeling:
– The machines/ folder holds the machine models (e.g., Edison2015 or similar) that define HPC system characteristics.
– The generate/ package is used for workload generation and even for generating specific workflow or special cases.
• Simulation Execution:
– The bin/ directory includes various scripts to run simulations (run_sim_exp.py) and control worker processes (workers_launch.sh, workers_kill.sh, etc.).
– The slurm/ module integrates Slurm configuration and trace generation.
• Data Management & Utility Libraries:
– The commonLib/ includes critical utilities like DBManager, Logging, and task handling, which indicate the centrality of a MySQL-based database for storing experiment state and results.
• Analysis and Plotting:
– The analysis/ and stats/ directories contain modules for analyzing simulation results.
– The plot/ folder along with many bin/plot_* scripts produce visualizations of experiment metrics.
• Workflow Support:
– The workflows/ folder plus workflow-related manifests in bin/manifests and XML/JSON converters (bin/xml2json.py) suggest a dedicated sub-system for workflow scheduling experiments.
• Testing & Tools:
– The test/ folder ensures integrity, while tools/ (with SSH utilities) hints at operational and connectivity considerations (e.g., SSH/worker management).
Key Interactions and Data Flow
• A “Workload Generator/Machine Model” component then uses these definitions to create or load models (using generate/ and machines/ modules) that represent the system’s jobs and hardware.
• The “Simulation Engine” (primarily driven by scripts in bin/ and the slurm/ module) receives these parameters, executes the scheduling simulation, possibly on remote workers coordinated via scripts in bin/workers_*.
• As simulations run, output trace files and experiment metrics are fed into a MySQL Database (scripted in bin/sql_createdb.sh, bin/sql_populate_db.py, and connected via commonLib/DBManager).
• Post-simulation, the “Analysis Engine” (modules in analysis/ and stats/) pulls results from the database, processes them (computing histograms, CDFs, throughput, slowdown, etc.), and prepares data for visualization.
• Finally, the “Visualization” component (plot/ and several plot_examples scripts in bin/plot_examples) generates comparative plots, histograms, and charts for the user.
• Additionally, note that the orchestration of experiment redos (delta, grouped experiments, and second-pass analysis) implies a cyclic workflow that revisits simulation results for deeper insights.
Important Architectural Patterns and Design Principles
• Modular Design: Separate packages for workload generation, simulation execution, analysis, and plotting indicate a clear separation of concerns.
• Pipeline Flow: The experiment lifecycle—from definition through simulation to analysis—is sequential yet loosely coupled, enabling independent testing and development.
• Worker Coordination: The use of SSH commands and dedicated worker management scripts shows distributed execution, important for scaling experiments.