# bacalhau-project/bacalhau architecture

> Architecture diagram of the GitHub repository bacalhau-project/bacalhau, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bacalhau-project/bacalhau
- Repository: https://github.com/bacalhau-project/bacalhau

## Overview

##### Project Type & Purpose
- Bacalhau is an open-source, full-stack distributed compute orchestration framework (“Compute Over Data”) written primarily in Go, with a TypeScript/Next.js web UI and Python client SDK.
- It implements an Orchestrator-Compute model: a control plane (orchestrator & API layer) schedules and dispatches jobs, while a fleet of compute nodes executes tasks close to the data.

##### High-Level File Structure
- **cmd/**: CLI entry-points (`bacalhau agent`, `bacalhau auth`, `bacalhau job`, etc.).
- **pkg/**: Core libraries subdivided by domain:
- `orchestrator/`—job scheduling, state transitions, planners & schedulers
- `compute/`—node capacity, executors, log streaming
- `publicapi/`—HTTP server, OpenAPI models, middleware
- `nats/`, `transport/`—message-bus abstractions & protocol handlers
- `storage/`, `downloader/`, `publisher/`—pluggable I/O backends (S3, IPFS, HTTP, local)
- plus cross-cutting concerns: authn/authz, config, telemetry, caching, jobstore (BoltDB)
- **webui/**: Next.js React front-end for job & node dashboards
- **clients/python/**: Python SDK auto-generated from OpenAPI
- **docker- & docker-compose-deployment/**: Container images and local multi-node setups
- **integration/**, **test_integration/**: end-to-end tests & examples
- **ops/** & **scripts/**: Terraform, metrics stacks (Prometheus, Grafana), CI/CD, cert management

##### Architectural Components
a. **User Interfaces**
- CLI (`cmd/cli`) and Web UI (`webui/`)
- Python SDK (`clients/python/`) or HTTP clients via OpenAPI
b. **API Layer**
- `publicapi/server.go` exposes REST/gRPC endpoints
- AuthN/AuthZ middleware (`pkg/authn/`, `pkg/authz/`)
- Binds into orchestrator control plane via internal Go APIs
c. **Control Plane (Orchestrator)**
- Receives job submissions, applies admission control, stores definitions in JobStore (BoltDB)
- Planner & Scheduler (`pkg/orchestrator/planner`, `pkg/orchestrator/scheduler`) decide node assignments
- Housekeeping, retry logic, state machine transitions
- Publishes dispatch messages over NATS (`pkg/nats/`, `pkg/transport/`)
d. **Data Plane (Compute Nodes)**
- Agent daemon (`cmd/cli/agent`) subscribes to job messages via NATS
- Capacity tracking & resource discovery (`pkg/compute/capacity/*`)
- Executor plugins: Docker (`pkg/executor/docker`), WASM (`pkg/executor/wasm`), No-Op
- Downloaders fetch input via HTTP, S3, IPFS (`pkg/downloader/*`)
- Publishers emit outputs to storage backends (`pkg/publisher/*`)
- Stream logs back to orchestrator via NATS logstream topics
e. **Message Bus & Transport**
- NATS Core for pub/sub (`pkg/nats/pubsub`) and streaming (`pkg/nats/stream`)
- bprotocol & nclprotocol implementations (`pkg/transport/*`)
f. **Storage & Metadata**
- BoltDB stores job metadata and node states in both orchestrator and compute (“devstack”)
- External systems: S3 buckets, IPFS nodes, HTTP servers, local volumes
g. **Observability & Metrics**
- Telemetry (`pkg/telemetry/*`) emits metrics to Prometheus
- Grafana dashboards configured under `ops/metrics/grafana`
- OpenTelemetry collector via config in `ops/metrics/otel-collector-config.yaml`

##### Relationships & Data Flow
1. **User → CLI/Web UI**: submit job (YAML or CLI flags)
2. **CLI/Web UI → Public API**: HTTP(gRPC) call → authn/authz → handler
3. **Public API → Orchestrator**: create job record in JobStore → return JobID
4. **Orchestrator → Message Bus**: Planner emits “NewJob” events on NATS
5. **Compute Agents → Message Bus**: subscribe to NewJob topics
6. **Compute Agents**:
- Fetch input (S3/IPFS/HTTP)
- Invoke executor (Docker/WASM)
- Stream logs/results via NATS back to orchestrator
7. **Orchestrator**: updates job status, triggers resume/cleanup
8. **Publishers**: write final outputs (S3/local)
9. **User → CLI/Web UI**: query status & logs via API

##### Architectural Patterns & Principles
- **Modular plugin architecture** for executors, storage, downloaders, publishers
- **Message-driven control/data planes** decoupled via NATS topics
- **Single-binary model** (config-driven Go process can run in “agent,” “orchestrator,” or “client” mode)
- **Event sourcing / state machine** in orchestrator and compute logging
- **Hexagonal / Clean architecture**: pkg modules define interfaces, cmd/webui/clients are adapters

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Swimlanes
    subgraph "User Interfaces"
        CLI["CLI"]:::ui
        WEBUI["Web UI"]:::ui
        PYSDK["Python SDK"]:::ui
    end

    subgraph "API Layer & Control Plane"
        API["Public API\n(REST/gRPC)"]:::goService
        AUTHN["AuthN"]:::goService
        AUTHZ["AuthZ"]:::goService
        ORCH["Orchestrator"]:::goService
        PLANNER["Planner"]:::goService
        SCHED["Scheduler"]:::goService
        JOBSTORE["JobStore\n(BoltDB)"]:::db
    end

    subgraph "Message Bus & Transport"
        NATS["NATS\n(pub/sub & stream)"]:::message
        TRANSPORT["Transport\nbprotocol/nclprotocol"]:::goService
    end

    subgraph "Compute Node Data Plane"
        AGENT["Compute Agent"]:::goService
        CAP["Capacity Tracker"]:::goService
        DL["Downloader"]:::goService
        DOCKER["Docker Executor"]:::plugin
        WASM["WASM Executor"]:::plugin
        NOOP["No-Op Executor"]:::plugin
        PUB["Publisher"]:::goService
    end

    subgraph "Storage & Observability"
        S3["S3/IPFS/HTTP\nStorage"]:::db
        METRICS["Telemetry & Metrics"]:::obs
        GRAFANA["Grafana Dashboards"]:::obs
    end

    %% Connections
    CLI -->|HTTP/gRPC| API
    WEBUI -->|HTTP/gRPC| API
    PYSDK -->|OpenAPI| API

    API -->|authenticate| AUTHN
    API -->|authorize| AUTHZ
    API -->|create job| ORCH

    ORCH -->|plan| PLANNER
    ORCH -->|schedule| SCHED
    ORCH -->|persist| JOBSTORE
    ORCH -->|publish NewJob| NATS

    NATS -->|subscribe| AGENT
    AGENT -->|report capacity| CAP
    AGENT -->|fetch input| DL
    DL -->|data| AGENT

    AGENT -->|execute| DOCKER
    AGENT -->|execute| WASM
    AGENT -->|execute| NOOP

    AGENT -->|publish logs/results| PUB
    PUB -->|store outputs| S3

    PUB -->|stream logs| NATS
    NATS -->|job status| ORCH
    ORCH -->|update status| JOBSTORE

    ORCH -->|emit metrics| METRICS
    AGENT -->|emit metrics| METRICS
    METRICS -->|visualize| GRAFANA

    %% Click Events
    click CLI "https://github.com/bacalhau-project/bacalhau/tree/main/cmd/cli/"
    click WEBUI "https://github.com/bacalhau-project/bacalhau/tree/main/webui/"
    click PYSDK "https://github.com/bacalhau-project/bacalhau/tree/main/clients/python/"
    click API "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/publicapi/"
    click AUTHN "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/authn/"
    click AUTHZ "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/authz/"
    click ORCH "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/orchestrator/"
    click PLANNER "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/orchestrator/planner/"
    click SCHED "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/orchestrator/scheduler/"
    click NATS "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/nats/pubsub/"
    click TRANSPORT "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/transport/bprotocol/"
    click AGENT "https://github.com/bacalhau-project/bacalhau/tree/main/cmd/cli/agent/"
    click CAP "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/compute/capacity/"
    click DL "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/downloader/"
    click DOCKER "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/executor/docker/"
    click WASM "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/executor/wasm/"
    click NOOP "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/executor/noop/"
    click PUB "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/publisher/"
    click JOBSTORE "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/jobstore/boltdb/"
    click METRICS "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/telemetry/"
    click GRAFANA "https://github.com/bacalhau-project/bacalhau/tree/main/ops/metrics/"
    click S3 "https://github.com/bacalhau-project/bacalhau/tree/main/pkg/storage/"

    %% Styles
    classDef ui fill:#f0e68c,stroke:#333
    classDef goService fill:#cce5ff,stroke:#333
    classDef db fill:#c2f0c2,stroke:#333
    classDef message fill:#ffe5cc,stroke:#333
    classDef plugin fill:#e5ccff,stroke:#333
    classDef obs fill:#d9d9d9,stroke:#333
```
