How sourcegraph/zoekt works
Main System Components
The system can be broken down into three major subsystems:
a) Indexing Subsystem
• Multiple indexers for different sources (git, archive, repo)
• Index builder and storage
• CTags integration for symbol indexing
b) Search Subsystem
• Web server
• gRPC server
• Query processing engine
• Shard management
c) Service Management
• Index server for repository management
• Mirror services for different platforms (GitHub, GitLab, Gitiles, etc.)
Key Architectural Elements
a) Data Flow
• Repository ingestion → Index creation → Search queries → Results
• Both synchronous (search) and asynchronous (indexing) flows
b) Service Architecture
• Multi-tenant architecture (internal/tenant)
• Distributed search across shards
• gRPC communication layer
[Source Repositories]
• ↓
[Mirror Services] → [Repository Cache]
• ↓
[Index Builder] → [Shard Storage]
• ↓
[Search Services]
• ↓
[Client Interface]
Component Details
a) Source Repositories Layer:
• GitHub
• GitLab
• Gitiles
• Bitbucket
• Local Git
b) Mirror Services Layer (cmd/zoekt-mirror-*):
• Repository syncing
• Change detection
• Version control integration
c) Indexing Layer:
• CTags integration
• Sharding mechanism
• Index format handling
• Symbol extraction
d) Search Layer:
• Query parsing
• Shard scheduling
• Result ranking
• API endpoints (REST/gRPC)
Technical Specifications
Technical details:
• Go-based implementation
• gRPC communication protocol
• Universal CTags integration
• Protobuf for data serialization
• OpenTelemetry for tracing
• Prometheus metrics
Integration Points
Connections with:
• Source code hosts
• CTags service
• Metrics collection
• Tracing system
• Client applications