How volcano-sh/volcano works
Project Type and Purpose
• Volcano is a Kubernetes-native batch scheduling system (an “operator” plus custom scheduler) that extends kube-scheduler via CustomResourceDefinitions (CRDs) and admission webhooks to manage high-performance, AI/ML, Big Data, and HPC workloads at scale.
File‐Structure Analysis
• Top‐level “cmd/…”: entrypoints for each microservice
• cmd/agent – node-level agent running on each worker
• cmd/controller-manager – controllers watching CRDs (Job, PodGroup, Queue, JobFlow, JobTemplate)
• cmd/scheduler – custom scheduler implementing plugin framework
• cmd/webhook-manager – admission webhook server for mutation/validation of CRs
• cmd/cli – kubectl-style CLI (vcctl, vsub, vjobs, etc.)
• “config/crd”: CRD YAML definitions (bases & v1beta1) for Job, PodGroup, Queue, JobFlow, JobTemplate
• “pkg/…”: implementation packages for each component:
• pkg/agent, pkg/networkqos, pkg/resourceusage, pkg/metriccollect – node-side metrics, QoS, resource reporting
• pkg/controllers – reconciliation loops (operator controllers)
• pkg/scheduler – scheduling framework, plugins, cache, actions
• pkg/webhooks – admission mutations & validations
• pkg/cli – API client, command logic
• “installer/helm” & YAML: deployment artifacts (namespace, Deployments, Services) and Helm charts
• “docs/images/volcano-architecture.png”: high-level architecture reference
Key Components & Their Interactions
A. Client Layer
• CLI (vcctl/vsub/vjobs…)
↔ Kubernetes API Server (reads/writes Volcano CRs)
B. Kubernetes Control Plane
• API Server with CRDs (Job, PodGroup, Queue, JobFlow, JobTemplate)
• Admission Webhook (volcano-admission) intercepts CR creations/updates for validation & defaulting
C. Volcano Control Plane Services (in “volcano-system” namespace)
1. Controller-Manager
• Watches Volcano CRs
• Reconciles desired state: creates/deletes Pods, PodGroups, JobFlow steps, manages Queue status
2. Custom Scheduler
• Watches unscheduled Pods with schedulerName=volcano
• Uses plugin framework (enqueue, allocate, preempt, backfill, reclaim, shuffle, etc.)
• Binds Pods to Nodes via API Server
D. Node Agent
• Runs on each worker node
• Collects node metrics, QoS events, oversubscription information
• Reports to API Server/Controller side via Status fields
E. External Systems
• etcd (Kubernetes datastore)
• Prometheus & Grafana (optional monitoring stack installed via installer)
• Benchmark scripts and e2e tests (under test/ & benchmark/)
Architectural Patterns & Design Principles
• Operator Pattern: custom controllers reconciling CR state
• Microservices: each “cmd/…” component is a separate Deployment
• Plugin Framework: scheduler loaded with independent “actions” & “plugins” (binpack, drf, capacity, cdp, nodegroup, task-topology, etc.)
• Admission-Control: validate & mutate Volcano custom objects at creation time
• Event-Driven: controllers and scheduler driven by informers/list-watch on CRs and core objects
• Separation of Concerns:
• CLI: presentation & client
• Admission: request shaping
• Controllers: control loops
• Scheduler: placement logic
• Agent: node internals