# uber/kraken architecture

> Architecture diagram of the GitHub repository uber/kraken, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uber/kraken
- Repository: https://github.com/uber/kraken

## Overview

Kraken is a P2P-powered Docker registry: clients push images through a proxy and pull them from agents, which fetch blobs from peers while the tracker coordinates peer discovery. Dedicated origins seed content and bridge local files to pluggable blob backends; build-index resolves tags and supports asynchronous cross-cluster replication. The README establishes the overall component roles; sampled handlers and the source index show selected calls. Some internal wiring and unsampled subsystems are therefore represented only where documented.

Main entry points: [`lib/backend/manager.go`](https://github.com/uber/kraken/blob/master/lib/backend/manager.go) (Backend manager), [`proxy/proxyserver/server.go`](https://github.com/uber/kraken/blob/master/proxy/proxyserver/server.go) (Registry proxy), [`build-index/tagserver/server.go`](https://github.com/uber/kraken/blob/master/build-index/tagserver/server.go) (Build-index server), [`agent/agentserver/server.go`](https://github.com/uber/kraken/blob/master/agent/agentserver/server.go) (Host agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_registry["Registry access"]
  node_proxy["Registry proxy<br/>[server.go]"]
  node_override["Registry override<br/>[server.go]"]
  node_agent["Host agent<br/>[server.go]"]
  node_storage_driver["Registry driver<br/>[storage_driver.go]"]
end

subgraph group_p2p["Peer distribution"]
  node_tracker["Peer tracker<br/>[server.go]"]
  node_announce["Peer announcements<br/>[client.go]"]
  node_scheduler["Peer scheduler<br/>[scheduler.go]"]
  node_piece_requests["Piece requests<br/>[manager.go]"]
  node_torrent_storage["Torrent storage<br/>[storage.go]"]
end

subgraph group_origin["Origin storage"]
  node_origin_server["Origin server<br/>[server.go]"]
  node_blob_client["Blob client<br/>[client.go]"]
  node_cluster_client["Origin cluster<br/>[cluster_client.go]"]
end

subgraph group_index["Tags and replication"]
  node_build_index["Build-index server<br/>[server.go]"]
  node_tag_client["Tag client<br/>[client.go]"]
  node_tag_store[("Tag store<br/>[store.go]")]
  node_retry["Replication retries<br/>[manager.go]"]
end

subgraph group_backend["Storage backends"]
  node_backend_manager["Backend manager<br/>[manager.go]"]
  node_gcs["GCS backend<br/>[client.go]"]
  node_hdfs["HDFS backend<br/>[client.go]"]
  node_s3["S3 backend<br/>[client.go]"]
  node_shadow["Shadow backend<br/>[client.go]"]
  node_sql["SQL backend<br/>[client.go]"]
end

node_docker_client(("Docker client"))
node_external_registry(("Remote registry"))

node_docker_client -->|"pushes images"| node_proxy
node_docker_client -->|"pulls images"| node_agent
node_proxy -->|"uploads blobs"| node_origin_server
node_proxy -->|"uploads tags"| node_build_index
node_proxy -->|"preheats blobs"| node_blob_client
node_proxy -->|"resolves tags"| node_tag_client
node_override -->|"lists tags"| node_tag_client
node_agent -->|"looks up tags"| node_tag_client
node_agent -->|"downloads via P2P"| node_scheduler
node_agent -->|"announces content"| node_announce
node_scheduler -->|"requests peers"| node_tracker
node_scheduler -.->|"dispatches pieces"| node_piece_requests
node_scheduler -.->|"stores pieces"| node_torrent_storage
node_tracker -->|"checks origin readiness"| node_cluster_client
node_tracker -->|"returns ordered peers"| node_external_registry
node_origin_server -->|"distributes blob load"| node_cluster_client
node_origin_server -->|"reads and writes blobs"| node_backend_manager
node_cluster_client -.->|"uses configured stores"| node_backend_manager
node_backend_manager -->|"creates client"| node_gcs
node_backend_manager -->|"creates client"| node_hdfs
node_backend_manager -->|"creates client"| node_s3
node_backend_manager -->|"creates client"| node_shadow
node_backend_manager -->|"creates client"| node_sql
node_build_index -->|"stores tag mappings"| node_tag_store
node_tag_client -->|"calls tag API"| node_build_index
node_build_index -->|"replicates tags"| node_external_registry
node_build_index -.->|"queues replication retries"| node_retry
node_external_registry -.->|"receives replicated tags"| node_external_registry

click node_proxy "https://github.com/uber/kraken/blob/master/proxy/proxyserver/server.go"
click node_override "https://github.com/uber/kraken/blob/master/proxy/registryoverride/server.go"
click node_agent "https://github.com/uber/kraken/blob/master/agent/agentserver/server.go"
click node_storage_driver "https://github.com/uber/kraken/blob/master/lib/dockerregistry/storage_driver.go"
click node_tracker "https://github.com/uber/kraken/blob/master/tracker/trackerserver/server.go"
click node_announce "https://github.com/uber/kraken/blob/master/tracker/announceclient/client.go"
click node_scheduler "https://github.com/uber/kraken/blob/master/lib/torrent/scheduler/scheduler.go"
click node_piece_requests "https://github.com/uber/kraken/blob/master/lib/torrent/scheduler/dispatch/piecerequest/manager.go"
click node_torrent_storage "https://github.com/uber/kraken/blob/master/lib/torrent/storage/storage.go"
click node_origin_server "https://github.com/uber/kraken/blob/master/origin/blobserver/server.go"
click node_blob_client "https://github.com/uber/kraken/blob/master/origin/blobclient/client.go"
click node_cluster_client "https://github.com/uber/kraken/blob/master/origin/blobclient/cluster_client.go"
click node_backend_manager "https://github.com/uber/kraken/blob/master/lib/backend/manager.go"
click node_gcs "https://github.com/uber/kraken/blob/master/lib/backend/gcsbackend/client.go"
click node_hdfs "https://github.com/uber/kraken/blob/master/lib/backend/hdfsbackend/client.go"
click node_s3 "https://github.com/uber/kraken/blob/master/lib/backend/s3backend/client.go"
click node_shadow "https://github.com/uber/kraken/blob/master/lib/backend/shadowbackend/client.go"
click node_sql "https://github.com/uber/kraken/blob/master/lib/backend/sqlbackend/client.go"
click node_build_index "https://github.com/uber/kraken/blob/master/build-index/tagserver/server.go"
click node_tag_client "https://github.com/uber/kraken/blob/master/build-index/tagclient/client.go"
click node_tag_store "https://github.com/uber/kraken/blob/master/build-index/tagstore/store.go"
click node_retry "https://github.com/uber/kraken/blob/master/lib/persistedretry/manager.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_proxy,node_override,node_agent,node_storage_driver,node_docker_client toneBlue
class node_tracker,node_announce,node_scheduler,node_piece_requests,node_torrent_storage toneAmber
class node_origin_server,node_blob_client,node_cluster_client toneMint
class node_build_index,node_tag_client,node_tag_store,node_retry toneRose
class node_backend_manager,node_gcs,node_hdfs,node_s3,node_shadow,node_sql,node_external_registry toneIndigo
```

## Components

### Registry access

- [Registry proxy](https://github.com/uber/kraken/blob/master/proxy/proxyserver/server.go): `proxy/proxyserver/server.go`
- [Registry override](https://github.com/uber/kraken/blob/master/proxy/registryoverride/server.go): `proxy/registryoverride/server.go`
- [Host agent](https://github.com/uber/kraken/blob/master/agent/agentserver/server.go): `agent/agentserver/server.go`
- [Registry driver](https://github.com/uber/kraken/blob/master/lib/dockerregistry/storage_driver.go): `lib/dockerregistry/storage_driver.go`

### Peer distribution

- [Peer tracker](https://github.com/uber/kraken/blob/master/tracker/trackerserver/server.go): `tracker/trackerserver/server.go`
- [Peer announcements](https://github.com/uber/kraken/blob/master/tracker/announceclient/client.go): `tracker/announceclient/client.go`
- [Peer scheduler](https://github.com/uber/kraken/blob/master/lib/torrent/scheduler/scheduler.go): `lib/torrent/scheduler/scheduler.go`
- [Piece requests](https://github.com/uber/kraken/blob/master/lib/torrent/scheduler/dispatch/piecerequest/manager.go): `lib/torrent/scheduler/dispatch/piecerequest/manager.go`
- [Torrent storage](https://github.com/uber/kraken/blob/master/lib/torrent/storage/storage.go): `lib/torrent/storage/storage.go`

### Origin storage

- [Origin server](https://github.com/uber/kraken/blob/master/origin/blobserver/server.go): `origin/blobserver/server.go`
- [Blob client](https://github.com/uber/kraken/blob/master/origin/blobclient/client.go): `origin/blobclient/client.go`
- [Origin cluster](https://github.com/uber/kraken/blob/master/origin/blobclient/cluster_client.go): `origin/blobclient/cluster_client.go`

### Tags and replication

- [Build-index server](https://github.com/uber/kraken/blob/master/build-index/tagserver/server.go): `build-index/tagserver/server.go`
- [Tag client](https://github.com/uber/kraken/blob/master/build-index/tagclient/client.go): `build-index/tagclient/client.go`
- [Tag store](https://github.com/uber/kraken/blob/master/build-index/tagstore/store.go): `build-index/tagstore/store.go`
- [Replication retries](https://github.com/uber/kraken/blob/master/lib/persistedretry/manager.go): `lib/persistedretry/manager.go`

### Storage backends

- [Backend manager](https://github.com/uber/kraken/blob/master/lib/backend/manager.go): `lib/backend/manager.go`
- [GCS backend](https://github.com/uber/kraken/blob/master/lib/backend/gcsbackend/client.go): `lib/backend/gcsbackend/client.go`
- [HDFS backend](https://github.com/uber/kraken/blob/master/lib/backend/hdfsbackend/client.go): `lib/backend/hdfsbackend/client.go`
- [S3 backend](https://github.com/uber/kraken/blob/master/lib/backend/s3backend/client.go): `lib/backend/s3backend/client.go`
- [Shadow backend](https://github.com/uber/kraken/blob/master/lib/backend/shadowbackend/client.go): `lib/backend/shadowbackend/client.go`
- [SQL backend](https://github.com/uber/kraken/blob/master/lib/backend/sqlbackend/client.go): `lib/backend/sqlbackend/client.go`

### Other components

- **Docker client**
- **Remote registry**

## Connections

- Docker client → Registry proxy: pushes images
- Docker client → Host agent: pulls images
- Registry proxy → Origin server: uploads blobs
- Registry proxy → Build-index server: uploads tags
- Registry proxy → Blob client: preheats blobs (evidence: `proxy/proxyserver/server.go`)
- Registry proxy → Tag client: resolves tags (evidence: `proxy/proxyserver/server.go`)
- Registry override → Tag client: lists tags (evidence: `proxy/registryoverride/server.go`)
- Host agent → Tag client: looks up tags (evidence: `agent/agentserver/server.go`)
- Host agent → Peer scheduler: downloads via P2P (evidence: `agent/agentserver/server.go`)
- Host agent → Peer announcements: announces content
- Peer scheduler → Peer tracker: requests peers
- Peer scheduler → Piece requests: dispatches pieces
- Peer scheduler → Torrent storage: stores pieces
- Peer tracker → Origin cluster: checks origin readiness (evidence: `tracker/trackerserver/server.go`)
- Peer tracker → Remote registry: returns ordered peers
- Origin server → Origin cluster: distributes blob load
- Origin server → Backend manager: reads and writes blobs (evidence: `origin/blobserver/server.go`)
- Origin cluster → Backend manager: uses configured stores
- Backend manager → GCS backend: creates client (evidence: `lib/backend/manager.go`)
- Backend manager → HDFS backend: creates client (evidence: `lib/backend/manager.go`)
- Backend manager → S3 backend: creates client (evidence: `lib/backend/manager.go`)
- Backend manager → Shadow backend: creates client (evidence: `lib/backend/manager.go`)
- Backend manager → SQL backend: creates client (evidence: `lib/backend/manager.go`)
- Build-index server → Tag store: stores tag mappings (evidence: `build-index/tagserver/server.go`)
- Tag client → Build-index server: calls tag API (evidence: `build-index/tagclient/client.go`)
- Build-index server → Remote registry: replicates tags
- Build-index server → Replication retries: queues replication retries
- Remote registry → Remote registry: receives replicated tags
