# michaelzhiluo/starburst architecture

> Architecture diagram of the GitHub repository michaelzhiluo/starburst, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/michaelzhiluo/starburst
- Repository: https://github.com/michaelzhiluo/starburst

## Overview

##### Project Type and Purpose
- The project is a cost-aware scheduler for “cloud bursting” that carries out both simulation experiments and real-world experiments using Kubernetes. It is designed to both replicate simulation results (using job trace inputs, waiting policies, and MILP optimizations) and run real experiments on a cluster with a Kubernetes scheduling plugin (Chakra).
- The README describes experiments (with figures and ablation studies) that validate cost savings and scheduler performance. Therefore, the project is a hybrid system that supports simulation as well as orchestration of jobs on a physical Kubernetes cluster.

##### File Structure
- Separate simulation components from real system components:
- The “skyburst” directory contains the simulation code (e.g., simulator.py, cluster.py, waiting_policy.py) and numerous notebooks/experiments illustrating simulation results.
- The “starburst” directory implements the real-world scheduler, job submission, event sources, sweep experiments, and cluster manager modules that interact with Kubernetes (evident from files under starburst/cluster_managers and drivers).
- There are also distinct directories for “real_scripts” (managing real Kubernetes-based experiments) and “simulator_scripts” (for running simulation experiments generating logs for plotting).
- Other supplementary files include task scripts for various ML experiments (under tasks) and configuration files (like sweep examples) that illustrate job generation parameters.

##### Main Components and Their Relationships
a. Simulation Module (Skyburst):
- Core simulation engine (skyburst/simulator.py) that models job arrivals, scheduling, waiting policies, and execution.
- Notebooks and plotting modules (skyburst/notebooks, skyburst/plot.py) to analyze simulation results and generate figures.
- Simulator scripts (simulator_scripts/) that run experiments and populate logs into designated locations (e.g., gs://starburst_bucket/logs).

b. Real System Module (Starburst):
- Scheduler Core: starburst/scheduler/starburst_scheduler.py where scheduling logic is implemented.
- Cluster Managers: Components in starburst/cluster_managers (e.g., kubernetes_manager.py, skypilot_manager.py) that abstract interactions with various clusters such as a Kubernetes on-prem cluster and possibly external cloud resources.
- Event Sources: starburst/event_sources (including a GRPC based event source) to manage message passing and job submission events.
- Job Drivers & Templates: Present in starburst/drivers and starburst/job_templates for launching the scheduler and constructing job configurations.
- Sweep Experiments: Modules under starburst/sweep that manage long-run experiments, log processing, and post-run log analysis.
- Plotting & Log Analysis: starburst/plots directory for visualizing results from real experiments and comparing them to simulation logs.

c. External and Supporting Components:
- Kubernetes Cluster: Both the simulation and real system experiments reference a Kubernetes cluster (configured via real_scripts and starburst/cluster_managers); the README provides detailed instructions for setting up and pre-pulling container images.
- Cloud Storage and Logging: Use of Google Cloud Storage (gs://starburst_bucket) for storing logs and sweep results.
- Optimization Component: The use of Gurobi for mixed integer linear programming (MILP) experiments (referenced in the notebooks and README).
- Additional Tools: The integration of Chakra as a scheduling plugin for Kubernetes and various scripts to cache images and control experiment flow.

##### Important Architectural Patterns and Design Principles
- Separation of Concerns: Distinct separation between simulation (skyburst) and the real-world scheduler orchestration (starburst) allows for independent evaluation and comparison.
- Modular Design: Component modules for cluster management, event handling, job generation, and plotting are clearly isolated in separate directories.
- Scripted Experiments: Use of bash scripts (both in simulator_scripts and real_scripts) to automate experiments and log generation, ensuring reproducibility.
- External Integration: Tight integration with Kubernetes, Cloud Storage, and optimization solvers (Gurobi) indicates a design that leverages external services robustly.

##### Final Recommendations
- Don’t oversimplify; capture as many components as possible:
- Simulation and real experiment branches.
- Flow from experiment start (via bash scripts) to logging and visualization.
- Significant architectural pillars (e.g., waiting policy decision module, job generation, cluster management logic).
- Provide annotations if needed – for instance, a note on the real-world experiments using a 4 node Kubernetes cluster and the equivalence of 1 GPU=3 CPUs.
- That each component (even if small or serving a utility purpose) is clearly labeled with its corresponding directory or functionality as observed in the file tree.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External nodes
    CloudStorage["Cloud Storage (Logs)"]:::external
    Gurobi["Gurobi Optimizer"]:::external
    Chakra["Chakra Plugin"]:::external
    KubeCluster["Kubernetes Cluster"]:::external

    %% Simulation Environment Subgraph
    subgraph "Simulation Environment"
        direction TB
        JobGen["Job Generator"]:::simulation
        WaitPolicy["Waiting Policy"]:::simulation
        SimCore["Simulation Core"]:::simulation
        SimScripts["Simulation Scripts"]:::simulation
        SimNotebooks["Simulation Notebooks"]:::simulation
        SimPlot["Simulation Plotting Module"]:::simulation
    end

    %% Real-World System Subgraph
    subgraph "Real-World System"
        direction TB
        EventSources["Event Sources"]:::real
        SchedulerCore["Scheduler Core"]:::real
        ClusterManagers["Cluster Managers"]:::real
        JobDrivers["Job Drivers"]:::real
        JobTemplates["Job Templates"]:::real
        SweepExpts["Sweep Experiments"]:::real
        RealPlots["Real Experiment Plots"]:::real
        RealScripts["Real Experiment Scripts"]:::real
    end

    %% Simulation Environment Relationships
    JobGen -->|"feeds"| SimCore
    WaitPolicy -->|"influences"| SimCore
    SimCore -->|"triggers"| SimScripts
    SimScripts -->|"saves logs to"| CloudStorage
    CloudStorage -->|"feeds data to"| SimNotebooks
    CloudStorage -->|"provides data to"| SimPlot
    SimCore -->|"optimizes via"| Gurobi

    %% Real-World System Relationships
    EventSources -->|"sends events to"| SchedulerCore
    SchedulerCore -->|"coordinates with"| ClusterManagers
    SchedulerCore -->|"receives from"| JobDrivers
    JobDrivers -->|"submits jobs to"| KubeCluster
    ClusterManagers -->|"manages"| KubeCluster
    RealScripts -->|"executes experiments on"| KubeCluster
    KubeCluster -->|"integrates with"| Chakra
    SweepExpts -->|"processes logs for"| RealPlots

    %% Styles for clearer differentiation
    classDef simulation fill:#ffcccc,stroke:#990000,stroke-width:2px;
    classDef real fill:#ccffcc,stroke:#008800,stroke-width:2px;
    classDef external fill:#cce5ff,stroke:#0044cc,stroke-width:2px;

    %% Click Events for Simulation Components
    click SimCore "https://github.com/michaelzhiluo/starburst/blob/main/skyburst/simulator.py"
    click SimNotebooks "https://github.com/michaelzhiluo/starburst/tree/main/skyburst/notebooks"
    click SimPlot "https://github.com/michaelzhiluo/starburst/blob/main/skyburst/plot.py"
    click JobGen "https://github.com/michaelzhiluo/starburst/blob/main/skyburst/job_generator.py"
    click WaitPolicy "https://github.com/michaelzhiluo/starburst/blob/main/skyburst/waiting_policy.py"
    click SimScripts "https://github.com/michaelzhiluo/starburst/tree/main/simulator_scripts"

    %% Click Events for Real-World System Components
    click SchedulerCore "https://github.com/michaelzhiluo/starburst/blob/main/starburst/scheduler/starburst_scheduler.py"
    click ClusterManagers "https://github.com/michaelzhiluo/starburst/tree/main/starburst/cluster_managers"
    click EventSources "https://github.com/michaelzhiluo/starburst/tree/main/starburst/event_sources"
    click JobDrivers "https://github.com/michaelzhiluo/starburst/tree/main/starburst/drivers"
    click JobTemplates "https://github.com/michaelzhiluo/starburst/tree/main/starburst/job_templates"
    click SweepExpts "https://github.com/michaelzhiluo/starburst/tree/main/starburst/sweep"
    click RealPlots "https://github.com/michaelzhiluo/starburst/tree/main/starburst/plots"
    click RealScripts "https://github.com/michaelzhiluo/starburst/tree/main/real_scripts"

    %% Click Event for External Plugin
    click Chakra "https://github.com/michaelzhiluo/starburst/tree/main/chakra"
```
