How bacalhau-project/bacalhau works
Project Type & Purpose
• Bacalhau is an open-source, full-stack distributed compute orchestration framework (“Compute Over Data”) written primarily in Go, with a TypeScript/Next.js web UI and Python client SDK.
• It implements an Orchestrator-Compute model: a control plane (orchestrator & API layer) schedules and dispatches jobs, while a fleet of compute nodes executes tasks close to the data.
High-Level File Structure
• cmd/: CLI entry-points (bacalhau agent, bacalhau auth, bacalhau job, etc.).
• pkg/: Core libraries subdivided by domain:
• orchestrator/—job scheduling, state transitions, planners & schedulers
• compute/—node capacity, executors, log streaming
• publicapi/—HTTP server, OpenAPI models, middleware
• nats/, transport/—message-bus abstractions & protocol handlers
• storage/, downloader/, publisher/—pluggable I/O backends (S3, IPFS, HTTP, local)
• plus cross-cutting concerns: authn/authz, config, telemetry, caching, jobstore (BoltDB)
• webui/: Next.js React front-end for job & node dashboards
• clients/python/: Python SDK auto-generated from OpenAPI
• docker- & docker-compose-deployment/: Container images and local multi-node setups
• integration/, test_integration/: end-to-end tests & examples
• ops/ & scripts/: Terraform, metrics stacks (Prometheus, Grafana), CI/CD, cert management
Architectural Components
a. User Interfaces
• CLI (cmd/cli) and Web UI (webui/)
• Python SDK (clients/python/) or HTTP clients via OpenAPI
b. API Layer
• publicapi/server.go exposes REST/gRPC endpoints
• AuthN/AuthZ middleware (pkg/authn/, pkg/authz/)
• Binds into orchestrator control plane via internal Go APIs
c. Control Plane (Orchestrator)
• Receives job submissions, applies admission control, stores definitions in JobStore (BoltDB)
• Planner & Scheduler (pkg/orchestrator/planner, pkg/orchestrator/scheduler) decide node assignments
• Housekeeping, retry logic, state machine transitions
• Publishes dispatch messages over NATS (pkg/nats/, pkg/transport/)
d. Data Plane (Compute Nodes)
• Agent daemon (cmd/cli/agent) subscribes to job messages via NATS
• Capacity tracking & resource discovery (pkg/compute/capacity/*)
• Executor plugins: Docker (pkg/executor/docker), WASM (pkg/executor/wasm), No-Op
• Downloaders fetch input via HTTP, S3, IPFS (pkg/downloader/*)
• Publishers emit outputs to storage backends (pkg/publisher/*)
• Stream logs back to orchestrator via NATS logstream topics
e. Message Bus & Transport
• NATS Core for pub/sub (pkg/nats/pubsub) and streaming (pkg/nats/stream)
• bprotocol & nclprotocol implementations (pkg/transport/*)
f. Storage & Metadata
• BoltDB stores job metadata and node states in both orchestrator and compute (“devstack”)
• External systems: S3 buckets, IPFS nodes, HTTP servers, local volumes
g. Observability & Metrics
• Telemetry (pkg/telemetry/*) emits metrics to Prometheus
• Grafana dashboards configured under ops/metrics/grafana
• OpenTelemetry collector via config in ops/metrics/otel-collector-config.yaml
Relationships & Data Flow
1. User → CLI/Web UI: submit job (YAML or CLI flags)
2. CLI/Web UI → Public API: HTTP(gRPC) call → authn/authz → handler
3. Public API → Orchestrator: create job record in JobStore → return JobID
4. Orchestrator → Message Bus: Planner emits “NewJob” events on NATS
5. Compute Agents → Message Bus: subscribe to NewJob topics
6. Compute Agents:
• Fetch input (S3/IPFS/HTTP)
• Invoke executor (Docker/WASM)
• Stream logs/results via NATS back to orchestrator
7. Orchestrator: updates job status, triggers resume/cleanup
8. Publishers: write final outputs (S3/local)
9. User → CLI/Web UI: query status & logs via API
Architectural Patterns & Principles
• Modular plugin architecture for executors, storage, downloaders, publishers
• Message-driven control/data planes decoupled via NATS topics
• Single-binary model (config-driven Go process can run in “agent,” “orchestrator,” or “client” mode)
• Event sourcing / state machine in orchestrator and compute logging
• Hexagonal / Clean architecture: pkg modules define interfaces, cmd/webui/clients are adapters