# sourcegraph/zoekt architecture

> Architecture diagram of the GitHub repository sourcegraph/zoekt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sourcegraph/zoekt
- Repository: https://github.com/sourcegraph/zoekt

## Overview

##### Main System Components

The system can be broken down into three major subsystems:

a) Indexing Subsystem
- Multiple indexers for different sources (git, archive, repo)
- Index builder and storage
- CTags integration for symbol indexing

b) Search Subsystem
- Web server
- gRPC server
- Query processing engine
- Shard management

c) Service Management
- Index server for repository management
- Mirror services for different platforms (GitHub, GitLab, Gitiles, etc.)

##### Key Architectural Elements

a) Data Flow
- Repository ingestion → Index creation → Search queries → Results
- Both synchronous (search) and asynchronous (indexing) flows

b) Service Architecture
- Multi-tenant architecture (internal/tenant)
- Distributed search across shards
- gRPC communication layer

**[Source Repositories]**
- ↓
**[Mirror Services] → [Repository Cache]**
- ↓
**[Index Builder] → [Shard Storage]**
- ↓
**[Search Services]**
- ↓
**[Client Interface]**

##### Component Details

a) Source Repositories Layer:
- GitHub
- GitLab
- Gitiles
- Bitbucket
- Local Git

b) Mirror Services Layer (cmd/zoekt-mirror-*):
- Repository syncing
- Change detection
- Version control integration

c) Indexing Layer:
- CTags integration
- Sharding mechanism
- Index format handling
- Symbol extraction

d) Search Layer:
- Query parsing
- Shard scheduling
- Result ranking
- API endpoints (REST/gRPC)

##### Technical Specifications

Technical details:
- Go-based implementation
- gRPC communication protocol
- Universal CTags integration
- Protobuf for data serialization
- OpenTelemetry for tracing
- Prometheus metrics

##### Integration Points

Connections with:
- Source code hosts
- CTags service
- Metrics collection
- Tracing system
- Client applications

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef external fill:#4CAF50,stroke:#388E3B,color:white
    classDef storage fill:#FFC107,stroke:#FFA000,color:black
    classDef critical fill:#F44336,stroke:#D32F2F,color:white
    classDef support fill:#9E9E9E,stroke:#616161,color:white

    %% Source Repositories Layer
    subgraph Sources
        GitHub["GitHub"]:::external
        GitLab["GitLab"]:::external
        Gitiles["Gitiles"]:::external
        Bitbucket["Bitbucket"]:::external
        LocalGit["Local Git"]:::external
    end

    %% Mirror Services Layer
    subgraph Mirrors
        GitHubMirror["GitHub Mirror"]:::core
        GitLabMirror["GitLab Mirror"]:::core
        GitilesMirror["Gitiles Mirror"]:::core
        BitbucketMirror["Bitbucket Mirror"]:::core
        GitIndexer["Git Indexer"]:::core
    end

    %% Repository Management
    RepoCache["Repository Cache"]:::storage
    
    %% Indexing Layer
    subgraph Indexing
        IndexBuilder["Index Builder"]:::core
        CTagsIntegration["CTags Integration"]:::external
        IndexServer["Index Server"]:::core
    end

    %% Storage Layer
    ShardStorage["Shard Storage"]:::storage
    
    %% Search Services Layer
    subgraph SearchServices
        WebServer["Web Server"]:::core
        GRPCServer["gRPC Server"]:::core
        QueryProcessor["Query Processor"]:::core
        ShardManager["Shard Manager"]:::core
    end

    %% Supporting Services
    subgraph Support
        MultiTenant["Multi-tenant Manager"]:::support
        TracingSystem["Tracing System"]:::support
    end

    %% Client Interface
    WebAPI["Web API"]:::core

    %% Connections
    GitHub --> GitHubMirror
    GitLab --> GitLabMirror
    Gitiles --> GitilesMirror
    Bitbucket --> BitbucketMirror
    LocalGit --> GitIndexer

    GitHubMirror & GitLabMirror & GitilesMirror & BitbucketMirror & GitIndexer --> RepoCache
    RepoCache --> IndexBuilder
    IndexBuilder --> ShardStorage
    CTagsIntegration --> IndexBuilder
    IndexServer --> IndexBuilder

    ShardStorage --> ShardManager
    ShardManager --> QueryProcessor
    QueryProcessor --> WebServer & GRPCServer
    WebServer & GRPCServer --> WebAPI

    MultiTenant -.-> SearchServices
    TracingSystem -.-> SearchServices

    %% Click events
    click IndexBuilder "https://github.com/sourcegraph/zoekt/blob/main/build/builder.go"
    click CTagsIntegration "https://github.com/sourcegraph/zoekt/blob/main/ctags/parser.go"
    click QueryProcessor "https://github.com/sourcegraph/zoekt/blob/main/query/query.go"
    click ShardManager "https://github.com/sourcegraph/zoekt/blob/main/shards/shards.go"
    click WebServer "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-webserver/main.go"
    click GRPCServer "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-webserver/grpc/server/server.go"
    click GitHubMirror "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-mirror-github/main.go"
    click GitLabMirror "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-mirror-gitlab/main.go"
    click GitilesMirror "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-mirror-gitiles/main.go"
    click BitbucketMirror "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-mirror-bitbucket-server/main.go"
    click GitIndexer "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-git-index/main.go"
    click IndexServer "https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-indexserver/main.go"
    click RepoCache "https://github.com/sourcegraph/zoekt/blob/main/gitindex/repocache.go"
    click MultiTenant "https://github.com/sourcegraph/zoekt/tree/main/internal/tenant"
    click TracingSystem "https://github.com/sourcegraph/zoekt/tree/main/internal/tracer"
    click WebAPI "https://github.com/sourcegraph/zoekt/blob/main/web/api.go"

    %% Legend
    subgraph Legend
        CoreService["Core Service"]:::core
        ExternalService["External Service"]:::external
        StorageComponent["Storage Component"]:::storage
        CriticalPath["Critical Path"]:::critical
        SupportService["Support Service"]:::support
    end
```
