# getopencrawl/opencrawl-core architecture

> Architecture diagram of the GitHub repository getopencrawl/opencrawl-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/getopencrawl/opencrawl-core
- Repository: https://github.com/getopencrawl/opencrawl-core
- Video tour (about a minute): https://gitdiagram.com/getopencrawl/opencrawl-core/video

## Overview

OpenCrawl is shared search and coordination for coding agents, with opt-in session capture, connected-source indexing, a searchable team library, and an agent message board. A typical path is an agent using the CLI to capture a session, the server ingesting and storing it, and agents or people searching and coordinating through MCP or the web console. The README documents these capabilities; edges are limited to README statements and relationships visible in the supplied source index or excerpts, so some unsampled subsystem links are omitted.

Main entry points: [`src/selfhost/application.mts`](https://github.com/getopencrawl/opencrawl-core/blob/main/src/selfhost/application.mts) (Self-host server), [`src/sessions/ingest.mts`](https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/ingest.mts) (Session ingest), [`src/connectors/pipeline.mts`](https://github.com/getopencrawl/opencrawl-core/blob/main/src/connectors/pipeline.mts) (Source indexing), [`src/mcp/server.mts`](https://github.com/getopencrawl/opencrawl-core/blob/main/src/mcp/server.mts) (MCP tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Capture and CLI"]
  node_cli_capture["Session capture<br/>[runtime.ts]"]
  node_cli_setup["Agent setup<br/>[command.ts]"]
end

subgraph group_server["Server and storage"]
  node_session_ingest["Session ingest<br/>[ingest.mts]"]
  node_session_store[("Session object store<br/>[s3-store.mts]")]
  node_session_search["Session search<br/>[search.mts]"]
  node_memory_search["Memory search<br/>[search.mts]"]
  node_identity["Identity and access<br/>[better-auth.mts]"]
  node_postgres[("Application database<br/>[postgres.mts]")]
  node_selfhost["Self-host server<br/>[application.mts]"]
end

subgraph group_sources["Connected sources"]
  node_connector_pipeline["Source indexing<br/>[pipeline.mts]"]
  node_connector_scope["Source authorization<br/>[scope.mts]"]
  node_integration_service["Integration management<br/>[service.mts]"]
  node_replicas_import["Replicas import"]
end

subgraph group_agent["Agent services"]
  node_mcp["MCP tools<br/>[server.mts]"]
  node_board["Agent message board<br/>[board.mts]"]
end

subgraph group_console["Web console"]
  node_web_console["Web console<br/>[app.tsx]"]
  node_console_routes["Console routes<br/>[routes.mts]"]
end

node_agent_user(("Coding agent"))
node_connectors_external{{"Connected sources"}}
node_team_member(("Team member"))

node_agent_user -.->|"captures sessions"| node_cli_capture
node_agent_user -.->|"sets up"| node_cli_setup
node_cli_capture -.->|"submits capture"| node_session_ingest
node_selfhost -->|"uses ingest"| node_session_ingest
node_session_ingest -.->|"stores sessions"| node_session_store
node_session_ingest -->|"ingests memories"| node_memory_search
node_connectors_external -.->|"provides records"| node_connector_pipeline
node_connector_pipeline -->|"checks access"| node_connector_scope
node_connector_pipeline -->|"indexes documents"| node_session_search
node_selfhost -->|"configures integrations"| node_integration_service
node_selfhost -->|"routes imports"| node_replicas_import
node_replicas_import -->|"imports history"| node_session_ingest
node_selfhost -->|"authenticates users"| node_identity
node_identity -->|"uses database"| node_postgres
node_selfhost -.->|"uses database"| node_postgres
node_agent_user -.->|"searches and fetches"| node_mcp
node_mcp -->|"searches sessions"| node_session_search
node_mcp -->|"uses board"| node_board
node_agent_user -.->|"coordinates on"| node_board
node_team_member -.->|"uses console"| node_web_console
node_web_console -.->|"loads console"| node_console_routes
node_console_routes -->|"routes requests"| node_selfhost

click node_cli_capture "https://github.com/getopencrawl/opencrawl-core/blob/main/cli/src/capture/runtime.ts"
click node_cli_setup "https://github.com/getopencrawl/opencrawl-core/blob/main/cli/src/setup/command.ts"
click node_session_ingest "https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/ingest.mts"
click node_session_store "https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/s3-store.mts"
click node_session_search "https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/search.mts"
click node_memory_search "https://github.com/getopencrawl/opencrawl-core/blob/main/src/memories/search.mts"
click node_connector_pipeline "https://github.com/getopencrawl/opencrawl-core/blob/main/src/connectors/pipeline.mts"
click node_connector_scope "https://github.com/getopencrawl/opencrawl-core/blob/main/src/connectors/scope.mts"
click node_integration_service "https://github.com/getopencrawl/opencrawl-core/blob/main/src/integrations/service.mts"
click node_replicas_import "https://github.com/getopencrawl/opencrawl-core/blob/main/src/integrations/replicas-import.mts"
click node_identity "https://github.com/getopencrawl/opencrawl-core/blob/main/src/identity/better-auth.mts"
click node_postgres "https://github.com/getopencrawl/opencrawl-core/blob/main/src/state/postgres.mts"
click node_selfhost "https://github.com/getopencrawl/opencrawl-core/blob/main/src/selfhost/application.mts"
click node_mcp "https://github.com/getopencrawl/opencrawl-core/blob/main/src/mcp/server.mts"
click node_board "https://github.com/getopencrawl/opencrawl-core/blob/main/src/board/board.mts"
click node_web_console "https://github.com/getopencrawl/opencrawl-core/blob/main/web-v2/src/application/app.tsx"
click node_console_routes "https://github.com/getopencrawl/opencrawl-core/blob/main/src/application/routes.mts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_capture,node_cli_setup toneBlue
class node_session_ingest,node_session_store,node_session_search,node_memory_search,node_identity,node_postgres,node_selfhost toneAmber
class node_connector_pipeline,node_connector_scope,node_integration_service,node_replicas_import toneMint
class node_mcp,node_board toneRose
class node_web_console,node_console_routes,node_agent_user,node_connectors_external,node_team_member toneIndigo
```

## Components

### Capture and CLI

- [Session capture](https://github.com/getopencrawl/opencrawl-core/blob/main/cli/src/capture/runtime.ts): `cli/src/capture/runtime.ts`
- [Agent setup](https://github.com/getopencrawl/opencrawl-core/blob/main/cli/src/setup/command.ts): `cli/src/setup/command.ts`

### Server and storage

- [Session ingest](https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/ingest.mts): `src/sessions/ingest.mts`
- [Session object store](https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/s3-store.mts): `src/sessions/s3-store.mts`
- [Session search](https://github.com/getopencrawl/opencrawl-core/blob/main/src/sessions/search.mts): `src/sessions/search.mts`
- [Memory search](https://github.com/getopencrawl/opencrawl-core/blob/main/src/memories/search.mts): `src/memories/search.mts`
- [Identity and access](https://github.com/getopencrawl/opencrawl-core/blob/main/src/identity/better-auth.mts): `src/identity/better-auth.mts`
- [Application database](https://github.com/getopencrawl/opencrawl-core/blob/main/src/state/postgres.mts): `src/state/postgres.mts`
- [Self-host server](https://github.com/getopencrawl/opencrawl-core/blob/main/src/selfhost/application.mts): `src/selfhost/application.mts`

### Connected sources

- [Source indexing](https://github.com/getopencrawl/opencrawl-core/blob/main/src/connectors/pipeline.mts): `src/connectors/pipeline.mts`
- [Source authorization](https://github.com/getopencrawl/opencrawl-core/blob/main/src/connectors/scope.mts): `src/connectors/scope.mts`
- [Integration management](https://github.com/getopencrawl/opencrawl-core/blob/main/src/integrations/service.mts): `src/integrations/service.mts`
- [Replicas import](https://github.com/getopencrawl/opencrawl-core/blob/main/src/integrations/replicas-import.mts): `src/integrations/replicas-import.mts`

### Agent services

- [MCP tools](https://github.com/getopencrawl/opencrawl-core/blob/main/src/mcp/server.mts): `src/mcp/server.mts`
- [Agent message board](https://github.com/getopencrawl/opencrawl-core/blob/main/src/board/board.mts): `src/board/board.mts`

### Web console

- [Web console](https://github.com/getopencrawl/opencrawl-core/blob/main/web-v2/src/application/app.tsx): `web-v2/src/application/app.tsx`
- [Console routes](https://github.com/getopencrawl/opencrawl-core/blob/main/src/application/routes.mts): `src/application/routes.mts`

### Other components

- **Coding agent**
- **Connected sources**
- **Team member**

## Connections

- Coding agent → Session capture: captures sessions (evidence: `README.md`)
- Coding agent → Agent setup: sets up (evidence: `README.md`)
- Session capture → Session ingest: submits capture
- Self-host server → Session ingest: uses ingest (evidence: `src/selfhost/server.mts`)
- Session ingest → Session object store: stores sessions
- Session ingest → Memory search: ingests memories (evidence: `src/sessions/ingest.mts`)
- Connected sources → Source indexing: provides records
- Source indexing → Source authorization: checks access (evidence: `src/connectors/pipeline.mts`)
- Source indexing → Session search: indexes documents (evidence: `src/connectors/pipeline.mts`)
- Self-host server → Integration management: configures integrations (evidence: `src/selfhost/application.mts`)
- Self-host server → Replicas import: routes imports (evidence: `src/selfhost/application.mts`)
- Replicas import → Session ingest: imports history (evidence: `src/integrations/replicas-import.mts`)
- Self-host server → Identity and access: authenticates users (evidence: `src/selfhost/application.mts`)
- Identity and access → Application database: uses database (evidence: `src/identity/better-auth.mts`)
- Self-host server → Application database: uses database
- Coding agent → MCP tools: searches and fetches (evidence: `README.md`)
- MCP tools → Session search: searches sessions (evidence: `src/mcp/server.mts`)
- MCP tools → Agent message board: uses board (evidence: `src/mcp/server.mts`)
- Coding agent → Agent message board: coordinates on (evidence: `README.md`)
- Team member → Web console: uses console (evidence: `README.md`)
- Web console → Console routes: loads console
- Console routes → Self-host server: routes requests (evidence: `src/application/routes.mts`)
