How buildbarn/bb-remote-execution works
Main Components
a. Core Services:
• bb_scheduler: Central scheduling service
• bb_worker: Build action orchestrator
• bb_runner: Command execution service
• bb_storage (external dependency): Storage daemon
b. Supporting Components:
• Build Queue
• CAS (Content Addressable Storage)
• File System Handlers
• Authentication/Authorization Layer
• Protocol Buffers Interface Layer
Component Relationships
Client Tools (Bazel/BuildStream/recc)
• ↓
[bb_storage] ←→ [bb_scheduler]
• ↓
• [bb_worker] ←→ [CAS/Blob Storage]
• ↓
• [bb_runner]
• ↓
• [Build Environment]
Key Architectural Elements
a. Service Boundaries:
• Clear separation between privileged and unprivileged components
• Network boundaries between components
b. Data Flow:
• Build request flow from client to execution
• File/artifact movement between storage and workers
• Results propagation back to clients
Important Subsystems to Highlight
a. Worker Components:
• Input file caching system
• Output file handling
• Build action orchestration
• Resource management
b. Runner Components:
• Execution environment isolation
• Command execution
• Resource constraints
• File system access
Integration Points
• GRPC interfaces between components
• Filesystem mounting points
• Authentication boundaries
• Monitoring/metrics collection points
Specific Implementation Details
• Volume mounting points for bb_runner_installer
• Relationship with external container images
• Points where caching occurs
Notes
Annotations for:
• Scalability points
• Security considerations
• Performance optimization points
• Critical paths
Remember to highlight the pluggable nature of the runner component and the ability to use different execution environments through container images.
This system design should clearly communicate both the high-level architecture and the important implementation details that make this remote execution system robust and secure.