How michaelzhiluo/starburst works
Project Type and Purpose
• The project is a cost-aware scheduler for “cloud bursting” that carries out both simulation experiments and real-world experiments using Kubernetes. It is designed to both replicate simulation results (using job trace inputs, waiting policies, and MILP optimizations) and run real experiments on a cluster with a Kubernetes scheduling plugin (Chakra).
• The README describes experiments (with figures and ablation studies) that validate cost savings and scheduler performance. Therefore, the project is a hybrid system that supports simulation as well as orchestration of jobs on a physical Kubernetes cluster.
File Structure
• Separate simulation components from real system components:
• The “skyburst” directory contains the simulation code (e.g., simulator.py, cluster.py, waiting_policy.py) and numerous notebooks/experiments illustrating simulation results.
• The “starburst” directory implements the real-world scheduler, job submission, event sources, sweep experiments, and cluster manager modules that interact with Kubernetes (evident from files under starburst/cluster_managers and drivers).
• There are also distinct directories for “real_scripts” (managing real Kubernetes-based experiments) and “simulator_scripts” (for running simulation experiments generating logs for plotting).
• Other supplementary files include task scripts for various ML experiments (under tasks) and configuration files (like sweep examples) that illustrate job generation parameters.
Main Components and Their Relationships
a. Simulation Module (Skyburst):
• Core simulation engine (skyburst/simulator.py) that models job arrivals, scheduling, waiting policies, and execution.
• Notebooks and plotting modules (skyburst/notebooks, skyburst/plot.py) to analyze simulation results and generate figures.
• Simulator scripts (simulator_scripts/) that run experiments and populate logs into designated locations (e.g., gs://starburst_bucket/logs).
b. Real System Module (Starburst):
• Scheduler Core: starburst/scheduler/starburst_scheduler.py where scheduling logic is implemented.
• Cluster Managers: Components in starburst/cluster_managers (e.g., kubernetes_manager.py, skypilot_manager.py) that abstract interactions with various clusters such as a Kubernetes on-prem cluster and possibly external cloud resources.
• Event Sources: starburst/event_sources (including a GRPC based event source) to manage message passing and job submission events.
• Job Drivers & Templates: Present in starburst/drivers and starburst/job_templates for launching the scheduler and constructing job configurations.
• Sweep Experiments: Modules under starburst/sweep that manage long-run experiments, log processing, and post-run log analysis.
• Plotting & Log Analysis: starburst/plots directory for visualizing results from real experiments and comparing them to simulation logs.
c. External and Supporting Components:
• Kubernetes Cluster: Both the simulation and real system experiments reference a Kubernetes cluster (configured via real_scripts and starburst/cluster_managers); the README provides detailed instructions for setting up and pre-pulling container images.
• Cloud Storage and Logging: Use of Google Cloud Storage (gs://starburst_bucket) for storing logs and sweep results.
• Optimization Component: The use of Gurobi for mixed integer linear programming (MILP) experiments (referenced in the notebooks and README).
• Additional Tools: The integration of Chakra as a scheduling plugin for Kubernetes and various scripts to cache images and control experiment flow.
Important Architectural Patterns and Design Principles
• Separation of Concerns: Distinct separation between simulation (skyburst) and the real-world scheduler orchestration (starburst) allows for independent evaluation and comparison.
• Modular Design: Component modules for cluster management, event handling, job generation, and plotting are clearly isolated in separate directories.
• Scripted Experiments: Use of bash scripts (both in simulator_scripts and real_scripts) to automate experiments and log generation, ensuring reproducibility.
• External Integration: Tight integration with Kubernetes, Cloud Storage, and optimization solvers (Gurobi) indicates a design that leverages external services robustly.
Final Recommendations
• Don’t oversimplify; capture as many components as possible:
• Simulation and real experiment branches.
• Flow from experiment start (via bash scripts) to logging and visualization.
• Significant architectural pillars (e.g., waiting policy decision module, job generation, cluster management logic).
• Provide annotations if needed – for instance, a note on the real-world experiments using a 4 node Kubernetes cluster and the equivalence of 1 GPU=3 CPUs.
• That each component (even if small or serving a utility purpose) is clearly labeled with its corresponding directory or functionality as observed in the file tree.