# bluesky-social/jetstream architecture

> Architecture diagram of the GitHub repository bluesky-social/jetstream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bluesky-social/jetstream
- Repository: https://github.com/bluesky-social/jetstream

## Overview

Jetstream is an atproto full-network archive, replay, and streaming service. Its principal workflow ingests repository history through backfill and live events, writes archive data, and exposes it through subscriptions and archive APIs; operators can also inspect status and export repository data. The repository documents both local and PostgreSQL/S3 disaggregated storage, plus a development network simulator. Edges reflect README statements or relationships visible in the supplied source index and excerpts; unsampled implementation details are not inferred.

Main entry points: [`internal/jetstreamd/runtime.go`](https://github.com/bluesky-social/jetstream/blob/main/internal/jetstreamd/runtime.go) (Service runtime), [`internal/subscribe/handler.go`](https://github.com/bluesky-social/jetstream/blob/main/internal/subscribe/handler.go) (WebSocket subscriptions), [`internal/catalog/scripts.go`](https://github.com/bluesky-social/jetstream/blob/main/internal/catalog/scripts.go) (Archive catalog), [`internal/xrpcapi/server.go`](https://github.com/bluesky-social/jetstream/blob/main/internal/xrpcapi/server.go) (Archive XRPC)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and archive"]
  node_backfill["Repo backfill<br/>[handler.go]"]
  node_writer["Ingest writer<br/>[writer.go]"]
  node_orchestrator["Ingest orchestrator<br/>[orchestrator.go]"]
end

subgraph group_storage["Storage and coordination"]
  node_manifest["Archive manifest<br/>[manifest.go]"]
  node_segments["Segment files<br/>[reader.go]"]
  node_catalog[("Archive catalog<br/>[scripts.go]")]
  node_objects[("S3 object store<br/>[blob.go]")]
end

subgraph group_serving["Client-facing serving"]
  node_session["Client sessions<br/>[session.go]"]
  node_subscribe["WebSocket subscriptions<br/>[handler.go]"]
  node_xrpc["Archive XRPC<br/>[server.go]"]
  node_api["Jetstream API client<br/>[client_core.go]"]
  node_web["Status web page<br/>[handler.go]"]
  node_export["Repo export<br/>[reconstruct.go]"]
end

subgraph group_operations["Operations and tooling"]
  node_service["Jetstream process<br/>[main.go]"]
  node_runtime["Service runtime<br/>[runtime.go]"]
  node_status["Status snapshot<br/>[collect.go]"]
  node_httpserver["HTTP listeners<br/>[server.go]"]
end

subgraph group_simulator["Development simulator"]
  node_simcmd["Simulator command<br/>[main.go]"]
  node_simhttp["Simulated atproto HTTP<br/>[handler.go]"]
  node_world["Simulated network<br/>[world.go]"]
end

node_network(("atproto network"))
node_operator(("Operator"))
node_client(("Jetstream client"))
node_pg[("PostgreSQL")]

node_network -.->|"supplies history"| node_backfill
node_backfill -->|"emits events"| node_writer
node_writer -.->|"feeds ingestion"| node_orchestrator
node_orchestrator -->|"coordinates archive"| node_catalog
node_manifest -->|"tracks segments"| node_segments
node_catalog -->|"references segments"| node_segments
node_runtime -.->|"runs ingestion"| node_orchestrator
node_service -->|"starts runtime"| node_runtime
node_runtime -.->|"creates sessions"| node_session
node_session -->|"configures"| node_subscribe
node_subscribe -->|"reads archive"| node_catalog
node_subscribe -->|"reads archive view"| node_manifest
node_subscribe -->|"reads segment data"| node_segments
node_client -.->|"streams events"| node_subscribe
node_client -->|"uses client library"| node_api
node_api -->|"requests archive"| node_xrpc
node_xrpc -->|"serves archive view"| node_manifest
node_xrpc -->|"opens archive objects"| node_objects
node_web -->|"loads snapshot"| node_status
node_web -->|"verifies repository"| node_export
node_status -->|"collects archive stats"| node_catalog
node_runtime -.->|"serves HTTP"| node_httpserver
node_runtime -.->|"uses database"| node_pg
node_runtime -.->|"uses object store"| node_objects
node_operator -->|"runs service"| node_service
node_operator -.->|"views status"| node_web
node_operator -->|"initializes storage"| node_service
node_simcmd -->|"starts HTTP services"| node_simhttp
node_simcmd -->|"runs simulated world"| node_world
node_simhttp -->|"serves network state"| node_world
node_network -->|"emulated locally"| node_simhttp

click node_service "https://github.com/bluesky-social/jetstream/blob/main/cmd/jetstream/main.go"
click node_runtime "https://github.com/bluesky-social/jetstream/blob/main/internal/jetstreamd/runtime.go"
click node_backfill "https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/backfill/handler.go"
click node_writer "https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/writer.go"
click node_orchestrator "https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/orchestrator/orchestrator.go"
click node_manifest "https://github.com/bluesky-social/jetstream/blob/main/internal/manifest/manifest.go"
click node_segments "https://github.com/bluesky-social/jetstream/blob/main/segment/reader.go"
click node_catalog "https://github.com/bluesky-social/jetstream/blob/main/internal/catalog/scripts.go"
click node_objects "https://github.com/bluesky-social/jetstream/blob/main/internal/objstore/s3/blob.go"
click node_session "https://github.com/bluesky-social/jetstream/blob/main/internal/jetstreamd/session.go"
click node_subscribe "https://github.com/bluesky-social/jetstream/blob/main/internal/subscribe/handler.go"
click node_xrpc "https://github.com/bluesky-social/jetstream/blob/main/internal/xrpcapi/server.go"
click node_api "https://github.com/bluesky-social/jetstream/blob/main/client_core.go"
click node_web "https://github.com/bluesky-social/jetstream/blob/main/internal/web/handler.go"
click node_status "https://github.com/bluesky-social/jetstream/blob/main/internal/status/collect.go"
click node_export "https://github.com/bluesky-social/jetstream/blob/main/internal/repoexport/reconstruct.go"
click node_httpserver "https://github.com/bluesky-social/jetstream/blob/main/internal/server/server.go"
click node_simcmd "https://github.com/bluesky-social/jetstream/blob/main/cmd/simulator/main.go"
click node_simhttp "https://github.com/bluesky-social/jetstream/blob/main/internal/simulator/http/handler.go"
click node_world "https://github.com/bluesky-social/jetstream/blob/main/internal/simulator/world/world.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_backfill,node_writer,node_orchestrator,node_client toneBlue
class node_manifest,node_segments,node_catalog,node_objects,node_pg toneAmber
class node_session,node_subscribe,node_xrpc,node_api,node_web,node_export toneMint
class node_service,node_runtime,node_status,node_httpserver toneRose
class node_simcmd,node_simhttp,node_world,node_network,node_operator toneIndigo
```

## Components

### Ingestion and archive

- [Repo backfill](https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/backfill/handler.go): `internal/ingest/backfill/handler.go`
- [Ingest writer](https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/writer.go): `internal/ingest/writer.go`
- [Ingest orchestrator](https://github.com/bluesky-social/jetstream/blob/main/internal/ingest/orchestrator/orchestrator.go): `internal/ingest/orchestrator/orchestrator.go`

### Storage and coordination

- [Archive manifest](https://github.com/bluesky-social/jetstream/blob/main/internal/manifest/manifest.go): `internal/manifest/manifest.go`
- [Segment files](https://github.com/bluesky-social/jetstream/blob/main/segment/reader.go): `segment/reader.go`
- [Archive catalog](https://github.com/bluesky-social/jetstream/blob/main/internal/catalog/scripts.go): `internal/catalog/scripts.go`
- [S3 object store](https://github.com/bluesky-social/jetstream/blob/main/internal/objstore/s3/blob.go): `internal/objstore/s3/blob.go`

### Client-facing serving

- [Client sessions](https://github.com/bluesky-social/jetstream/blob/main/internal/jetstreamd/session.go): `internal/jetstreamd/session.go`
- [WebSocket subscriptions](https://github.com/bluesky-social/jetstream/blob/main/internal/subscribe/handler.go): `internal/subscribe/handler.go`
- [Archive XRPC](https://github.com/bluesky-social/jetstream/blob/main/internal/xrpcapi/server.go): `internal/xrpcapi/server.go`
- [Jetstream API client](https://github.com/bluesky-social/jetstream/blob/main/client_core.go): `client_core.go`
- [Status web page](https://github.com/bluesky-social/jetstream/blob/main/internal/web/handler.go): `internal/web/handler.go`
- [Repo export](https://github.com/bluesky-social/jetstream/blob/main/internal/repoexport/reconstruct.go): `internal/repoexport/reconstruct.go`

### Operations and tooling

- [Jetstream process](https://github.com/bluesky-social/jetstream/blob/main/cmd/jetstream/main.go): `cmd/jetstream/main.go`
- [Service runtime](https://github.com/bluesky-social/jetstream/blob/main/internal/jetstreamd/runtime.go): `internal/jetstreamd/runtime.go`
- [Status snapshot](https://github.com/bluesky-social/jetstream/blob/main/internal/status/collect.go): `internal/status/collect.go`
- [HTTP listeners](https://github.com/bluesky-social/jetstream/blob/main/internal/server/server.go): `internal/server/server.go`

### Development simulator

- [Simulator command](https://github.com/bluesky-social/jetstream/blob/main/cmd/simulator/main.go): `cmd/simulator/main.go`
- [Simulated atproto HTTP](https://github.com/bluesky-social/jetstream/blob/main/internal/simulator/http/handler.go): `internal/simulator/http/handler.go`
- [Simulated network](https://github.com/bluesky-social/jetstream/blob/main/internal/simulator/world/world.go): `internal/simulator/world/world.go`

### Other components

- **atproto network**
- **Operator**
- **Jetstream client**
- **PostgreSQL**

## Connections

- atproto network → Repo backfill: supplies history (evidence: `README.md`)
- Repo backfill → Ingest writer: emits events (evidence: `internal/ingest/backfill/handler.go`)
- Ingest writer → Ingest orchestrator: feeds ingestion
- Ingest orchestrator → Archive catalog: coordinates archive (evidence: `internal/ingest/orchestrator/orchestrator.go`)
- Archive manifest → Segment files: tracks segments (evidence: `internal/manifest/manifest.go`)
- Archive catalog → Segment files: references segments (evidence: `internal/catalog/scripts.go`)
- Service runtime → Ingest orchestrator: runs ingestion
- Jetstream process → Service runtime: starts runtime (evidence: `cmd/jetstream/main.go`)
- Service runtime → Client sessions: creates sessions
- Client sessions → WebSocket subscriptions: configures (evidence: `internal/jetstreamd/session.go`)
- WebSocket subscriptions → Archive catalog: reads archive (evidence: `internal/subscribe/handler.go`)
- WebSocket subscriptions → Archive manifest: reads archive view (evidence: `internal/subscribe/handler.go`)
- WebSocket subscriptions → Segment files: reads segment data (evidence: `internal/subscribe/handler.go`)
- Jetstream client → WebSocket subscriptions: streams events (evidence: `README.md`)
- Jetstream client → Jetstream API client: uses client library (evidence: `client_core.go`)
- Jetstream API client → Archive XRPC: requests archive (evidence: `client_core.go`)
- Archive XRPC → Archive manifest: serves archive view (evidence: `internal/xrpcapi/server.go`)
- Archive XRPC → S3 object store: opens archive objects (evidence: `internal/xrpcapi/server.go`)
- Status web page → Status snapshot: loads snapshot (evidence: `internal/web/handler.go`)
- Status web page → Repo export: verifies repository (evidence: `internal/web/handler.go`)
- Status snapshot → Archive catalog: collects archive stats (evidence: `internal/status/collect.go`)
- Service runtime → HTTP listeners: serves HTTP
- Service runtime → PostgreSQL: uses database
- Service runtime → S3 object store: uses object store
- Operator → Jetstream process: runs service (evidence: `README.md`)
- Operator → Status web page: views status
- Operator → Jetstream process: initializes storage (evidence: `cmd/jetstream/main.go`)
- Simulator command → Simulated atproto HTTP: starts HTTP services (evidence: `cmd/simulator/main.go`)
- Simulator command → Simulated network: runs simulated world (evidence: `cmd/simulator/main.go`)
- Simulated atproto HTTP → Simulated network: serves network state (evidence: `internal/simulator/http/handler.go`)
- atproto network → Simulated atproto HTTP: emulated locally (evidence: `README.md`)
