# sbairoliya/buildgrid architecture

> Architecture diagram of the GitHub repository sbairoliya/buildgrid, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sbairoliya/buildgrid
- Repository: https://github.com/sbairoliya/buildgrid

## Overview

##### Main components
- External Clients: Represent the build and execution clients (Bazel, BuildStream, RECC, etc.) that initiate requests using the Remote Execution API.
- API & gRPC Interface: Highlight the gRPC endpoints defined by the extensive set of protobufs (found under buildgrid/_protos and protos/src) that establish the Remote Execution and Worker APIs.
- Server Core: This is the hub of BuildGrid and is implemented in the “buildgrid/server” directory. Identify the following subcomponents:
– App/CLI: The entry point that starts and configures the server (buildgrid/server/app).
– Scheduling: The scheduler and notifier modules (buildgrid/server/scheduler) that allocate jobs to workers.
– Execution and Client Logic: Components for handling the remote execution service (buildgrid/server/execution), client proxy (buildgrid/server/client), and the overall orchestration (buildgrid/server/controller and servicer).
– Authentication & Authorization: The auth module (buildgrid/server/auth) that manages tokens and secure access.
– Data Storage & Caching: Components for content-addressable storage (CAS) (buildgrid/server/cas) with different backend implementations (e.g., disk, redis, s3, etc.), along with the action cache, indexing, and mirrored/sharded caches (found under buildgrid/server/actioncache and buildgrid/server/cas/storage).
– Build Events & Introspection: Modules that track build event streams (buildgrid/server/build_events) and introspection (buildgrid/server/introspection).
– Operations and Monitoring: The operations module (buildgrid/server/operations) for filtering and managing operations, and the monitoring components (buildgrid/server/monitoring) that track system health and status.
– External Integrations: Components such as bots (buildgrid/server/bots), capabilities (buildgrid/server/capabilities), and cleanup/janitor (buildgrid/server/cleanup) which manage auxiliary functions.

##### Additional supporting pieces
- Configuration & Settings: Various YAML configuration files are stored under data/config and within buildgrid/server/app/settings. These define deployment details (e.g., storage backends, Redis, S3, SQL) and can be noted as “Configuration Layer.”
- Protobuf Definitions: The project contains an extensive directory tree for protobufs (both under buildgrid/_protos and the separate protos directory) that specify the remote execution, asset, BuildGrid-specific APIs, and various gRPC services – include these as the “API Definitions” that drive communication both within internal modules and between external clients and the server.

##### Relate the interactions
- How requests then funnel into the “Server Core” modules for scheduling, authentication, dispatching to the execution module, and finally result aggregation.
- Role of the scheduler and notifier in assigning jobs to “Remote Workers” (which can be represented externally).

##### Architectural patterns and technologies
- Microservice- or modular-style design within the BuildGrid server where distinct functionalities (e.g., CAS, authentication, scheduling) are well separated.
- Use of gRPC and protocol buffers for communication both internally and with external nodes.
- Configurable components (multiple storage backends, replicated caches, sharded action caches) as part of the extensibility of the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External and API Layers
    EC["External Clients"]:::external
    API["API & gRPC Interface"]:::api
    PD["Protobuf Definitions"]:::api

    %% Configuration & Settings subgraph
    subgraph "Configuration & Settings"
        conf1["YAML Configurations"]:::config
        conf2["Settings Parsing"]:::config
    end

    %% Server Core subgraph
    subgraph "Server Core"
        direction TB
        cli["App/CLI"]:::core
        sched["Scheduling Module"]:::scheduling
        exec["Execution Engine"]:::execution
        client["Client Proxy"]:::execution
        ctrl["Controller"]:::orchestration
        servicer["Servicer"]:::orchestration
        auth["Authentication Module"]:::auth
        cas["CAS Module"]:::storage
        acache["Action Cache"]:::storage
        bevents["Build Events"]:::events
        intro["Introspection"]:::events
        ops["Operations Module"]:::ops
        mon["Monitoring"]:::ops
        bots["Bots Integration"]:::integrations
        caps["Capabilities Integration"]:::integrations
        cleanup["Cleanup/Janitor"]:::integrations
    end

    %% Data Flow Connections
    EC -->|"calls_via_gRPC"| API
    API -->|"uses_protos"| PD
    API -->|"routes_requests"| cli
    conf1 -->|"load_config"| cli
    conf2 -->|"parse_settings"| cli

    cli -->|"init_scheduling"| sched
    cli -->|"start_execution"| exec
    cli -->|"init_client_proxy"| client
    cli -->|"orchestrate_tasks"| ctrl
    cli -->|"provide_services"| servicer
    cli -->|"validate_access"| auth

    sched -->|"assign_jobs"| exec

    client -->|"read/write_CAS"| cas
    client -->|"access_cache"| acache

    exec -->|"store_artifacts"| cas
    exec -->|"cache_results"| acache
    exec -->|"emit_events"| bevents
    exec -->|"emit_introspection"| intro
    exec -->|"trigger_bots"| bots
    exec -->|"check_capabilities"| caps
    exec -->|"invoke_cleanup"| cleanup

    ctrl -->|"monitor_ops"| ops
    servicer -->|"monitor_ops"| ops
    ops -->|"update_health"| mon
    mon -->|"system_status"| EC

    bevents -->|"report_events"| EC
    intro -->|"report_introspection"| EC

    %% Class Definitions for styling
    classDef external fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef api fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px;
    classDef core fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef scheduling fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
    classDef execution fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef orchestration fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
    classDef auth fill:#f0b27a,stroke:#e67e22,stroke-width:2px;
    classDef storage fill:#d7bde2,stroke:#af7ac5,stroke-width:2px;
    classDef events fill:#d1f2eb,stroke:#1abc9c,stroke-width:2px;
    classDef ops fill:#f9e79f,stroke:#f39c12,stroke-width:2px;
    classDef integrations fill:#f5b7b1,stroke:#cd6155,stroke-width:2px;
    classDef config fill:#e8f8f5,stroke:#1abc9c,stroke-width:2px;

    %% Click Events
    click API "https://github.com/sbairoliya/buildgrid/tree/master/buildgrid/_protos"
    click PD "https://github.com/sbairoliya/buildgrid/tree/master/protos/src"
    click cli "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/app/cli.py"
    click sched "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/scheduler/assigner.py"
    click exec "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/execution/service.py"
    click client "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/client/actioncache.py"
    click ctrl "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/controller.py"
    click servicer "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/servicer.py"
    click auth "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/auth/manager.py"
    click cas "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/cas/instance.py"
    click acache "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/actioncache/service.py"
    click bevents "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/build_events/service.py"
    click intro "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/introspection/service.py"
    click ops "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/operations/service.py"
    click mon "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/monitoring.py"
    click bots "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/bots/service.py"
    click caps "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/capabilities/service.py"
    click cleanup "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/cleanup/cleanup.py"
    click conf1 "https://github.com/sbairoliya/buildgrid/tree/master/data/config"
    click conf2 "https://github.com/sbairoliya/buildgrid/blob/master/buildgrid/server/app/settings/config.py"
```
