How bluesky-social/jetstream works
Jetstream is an atproto full-network archive, replay, and streaming service. Its principal workflow ingests repository history through backfill and live events, writes archive data, and exposes it through subscriptions and archive APIs; operators can also inspect status and export repository data. The repository documents both local and PostgreSQL/S3 disaggregated storage, plus a development network simulator. Edges reflect README statements or relationships visible in the supplied source index and excerpts; unsampled implementation details are not inferred.
Main entry points: runtime.go Service runtime · handler.go WebSocket subscriptions · scripts.go Archive catalog · server.go Archive XRPC
Components 24
Ingestion and archive
- Repo backfill
internal/ingest/backfill/handler.go - Ingest writer
internal/ingest/writer.go - Ingest orchestrator
internal/ingest/orchestrator/orchestrator.go
Storage and coordination
- Archive manifest
internal/manifest/manifest.go - Segment files
segment/reader.go - Archive catalog
internal/catalog/scripts.go - S3 object store
internal/objstore/s3/blob.go
Client-facing serving
- Client sessions
internal/jetstreamd/session.go - WebSocket subscriptions
internal/subscribe/handler.go - Archive XRPC
internal/xrpcapi/server.go - Jetstream API client
client_core.go - Status web page
internal/web/handler.go - Repo export
internal/repoexport/reconstruct.go
Operations and tooling
- Jetstream process
cmd/jetstream/main.go - Service runtime
internal/jetstreamd/runtime.go - Status snapshot
internal/status/collect.go - HTTP listeners
internal/server/server.go
Development simulator
- Simulator command
cmd/simulator/main.go - Simulated atproto HTTP
internal/simulator/http/handler.go - Simulated network
internal/simulator/world/world.go
Other components
- atproto network
- Operator
- Jetstream client
- PostgreSQL
Connections 31
- atproto network → Repo backfill: supplies history
README.md - Repo backfill → Ingest writer: emits events
internal/ingest/backfill/handler.go - Ingest writer → Ingest orchestrator: feeds ingestion
- Ingest orchestrator → Archive catalog: coordinates archive
internal/ingest/orchestrator/orchestrator.go - Archive manifest → Segment files: tracks segments
internal/manifest/manifest.go - Archive catalog → Segment files: references segments
internal/catalog/scripts.go - Service runtime → Ingest orchestrator: runs ingestion
- Jetstream process → Service runtime: starts runtime
cmd/jetstream/main.go - Service runtime → Client sessions: creates sessions
- Client sessions → WebSocket subscriptions: configures
internal/jetstreamd/session.go - WebSocket subscriptions → Archive catalog: reads archive
internal/subscribe/handler.go - WebSocket subscriptions → Archive manifest: reads archive view
internal/subscribe/handler.go - WebSocket subscriptions → Segment files: reads segment data
internal/subscribe/handler.go - Jetstream client → WebSocket subscriptions: streams events
README.md - Jetstream client → Jetstream API client: uses client library
client_core.go - Jetstream API client → Archive XRPC: requests archive
client_core.go - Archive XRPC → Archive manifest: serves archive view
internal/xrpcapi/server.go - Archive XRPC → S3 object store: opens archive objects
internal/xrpcapi/server.go - Status web page → Status snapshot: loads snapshot
internal/web/handler.go - Status web page → Repo export: verifies repository
internal/web/handler.go - Status snapshot → Archive catalog: collects archive stats
internal/status/collect.go - Service runtime → HTTP listeners: serves HTTP
- Service runtime → PostgreSQL: uses database
- Service runtime → S3 object store: uses object store
- Operator → Jetstream process: runs service
README.md - Operator → Status web page: views status
- Operator → Jetstream process: initializes storage
cmd/jetstream/main.go - Simulator command → Simulated atproto HTTP: starts HTTP services
cmd/simulator/main.go - Simulator command → Simulated network: runs simulated world
cmd/simulator/main.go - Simulated atproto HTTP → Simulated network: serves network state
internal/simulator/http/handler.go - atproto network → Simulated atproto HTTP: emulated locally
README.md