How secretflow/kuscia works
High-level areas of the system
• Frontend/API Layer: Although not a full-fledged “frontend” in the traditional UI sense, the API components—implemented via the “kusciaapi”, “gateway”, and “transport” modules (with their GRPC/HTTP handlers found in proto definitions and related packages)—act as primary points of interaction.
• Orchestration Core/Controllers: The core “cmd/kuscia” binary and multiple “pkg/controllers” modules implement the orchestration logic. These components handle the lifecycle of secure computing tasks, schedule deployments, manage pods, and update CRD states.
• Agent and Runtime Components: The “pkg/agent” module and its various subpackages (e.g., container management and local/runtime implementations) provide the infrastructure for managing containerized tasks (including resource management, garbage collection, and probing).
• Custom Resource Definitions (CRDs): The “crds” directory clearly defines CRDs (for domains, datamesh, interconnection, tasks, etc.) that extend Kubernetes’ API and enable custom orchestration constructs.
• Configuration and Integration: “pkg/confmanager” and “pkg/datamesh” modules deal with configuration management and data service orchestration. They provide dynamic configuration handling for inter-domain scheduling, data access, and secure interconnection.
• Security & Interconnection: Components under “pkg/interconn”, “pkg/gateway”, and “pkg/kusciaapi” (which include certificate handling, domain route management, and secure handshake protocols) implement secure, multi-domain communication.
• Auxiliary Components: The scheduling (in “pkg/scheduler”), diagnostics (“pkg/diagnose”), logging/monitoring (“pkg/metricexporter” and “pkg/reporter”), and web interfaces (under “pkg/web”) provide supplementary infrastructure for ensuring health, metrics, and system integrity.
Relationships and interactions
• APIs (both HTTP and GRPC) expose endpoints for external systems to submit tasks, configurations, and queries. These endpoints communicate with the core orchestrator.
• The orchestrator (controllers) monitors and reconciles desired state using CRDs. It interacts with various agents deployed on each node.
• Agents manage the local container runtime (using container handling, scheduling, monitoring, and logging) and interact with the orchestrator via the CRDs and API calls.
• The configuration manager loads and propagates secure parameters and policies for tasks and clusters.
• Data-related tasks and datamesh components interlink with the interconnection modules to support cross-domain communications and privacy-preserving operations.
• Gateway components, along with proxy and interconnection modules, facilitate secure network communication and manage port reuse, thus ensuring that minimal public networking resources are required.
Important architectural patterns
• Modular Microservices: The project is split into multiple modules communicating through well-defined interfaces (HTTP/GRPC).
• Operator Pattern in Kubernetes: Use of CRDs along with controllers reflects the Kubernetes operator design pattern.
• Secure Inter-domain Communication: Security and privacy aspects have been emphasized with certificate management, handshake protocols, and gateway routing.
• Plugin/Extensibility: Middleware components (e.g., under “pkg/agent/middleware/plugins”) indicate that extensibility is baked into the system design for additional functionality (like image security, environment import, and config rendering).
Relevant technologies and frameworks
• Kubernetes (K3s) is the underlying orchestration platform.
• GRPC and HTTP for API endpoints.
• Bazel for build and dependency management (evidenced by BUILD.bazel, MODULE.bazel, and other build scripts).
• Go (with test files, extensive pkg structure, and a robust CLI under cmd/) as the language of implementation.
• CI/CD and deployment scripts (using CircleCI, GitHub actions, shell scripts).
• Dockerfiles and Kubernetes deployment templates are used for containerization and cluster deployment.