# theproject24/zynsearch architecture

> Architecture diagram of the GitHub repository theproject24/zynsearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theproject24/zynsearch
- Repository: https://github.com/theproject24/zynsearch

## Overview

ZynSearch is a Rust search engine exposed through a CLI, REST and gRPC server, and thin JS, Python, and Go SDKs. Documents are collected from local directories or S3, normalized and analyzed, then inserted into an inverted index; queries are coordinated across shards, retrieved and ranked, and formatted for callers. The graph preserves documented REST, gRPC, streaming, bulk-index, persistence, and SDK capabilities. Several source excerpts are partial or unsampled, so uncertain internal wiring is omitted.

Main entry points: [`crates/zynsearch-core/src/ingestion.rs`](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/ingestion.rs) (Source ingestion), [`crates/zynsearch-core/src/engine.rs`](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/engine.rs) (Search engine), [`crates/zynsearch-server/src/http.rs`](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-server/src/http.rs) (REST gateway), [`crates/zynsearch-core/src/query_pipeline.rs`](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/query_pipeline.rs) (Query coordinator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Clients"]
  node_cli["CLI<br/>[main.rs]"]
  node_sdk["Language SDKs"]
end

subgraph group_ingestion["Document ingestion"]
  node_ingestion["Source ingestion<br/>[ingestion.rs]"]
  node_normalization["Document normalization<br/>[document_ingest.rs]"]
  node_crawler["Directory crawler<br/>[crawler.rs]"]
end

subgraph group_core["Search core"]
  node_engine["Search engine<br/>[engine.rs]"]
  node_analyzer["Text analyzer<br/>[analyzer.rs]"]
  node_index[("Inverted index<br/>[index.rs]")]
  node_coordinator["Query coordinator<br/>[query_pipeline.rs]"]
  node_searcher["Retrieval engine<br/>[searcher.rs]"]
  node_bm25["BM25 ranking<br/>[bm25.rs]"]
  node_topk["Top-K collector<br/>[top_k.rs]"]
end

subgraph group_transport["Server transports"]
  node_server["Server runtime<br/>[main.rs]"]
  node_http["REST gateway<br/>[http.rs]"]
  node_grpc["gRPC contract<br/>[zynsearch.proto]"]
end

subgraph group_persistence["Persistence"]
  node_storage[("Index persistence<br/>[storage.rs]")]
end

node_user(("Search user"))
node_local(("Local corpus"))
node_s3[("S3 objects")]
node_cliuser(("CLI operator"))
node_caller(("API caller"))
node_result["Search results"]

node_cliuser -->|"runs"| node_cli
node_cli -->|"starts ingestion"| node_ingestion
node_local -->|"supplies files"| node_ingestion
node_s3 -.->|"supplies objects"| node_ingestion
node_ingestion -->|"crawls directory"| node_crawler
node_ingestion -->|"normalizes content"| node_normalization
node_ingestion -->|"indexes documents"| node_engine
node_normalization -->|"provides text"| node_analyzer
node_engine -->|"analyzes text"| node_analyzer
node_engine -->|"updates index"| node_index
node_engine -->|"executes search"| node_searcher
node_searcher -->|"analyzes query"| node_analyzer
node_searcher -->|"reads postings"| node_index
node_searcher -->|"scores matches"| node_bm25
node_coordinator -->|"dispatches shards"| node_engine
node_coordinator -->|"collects results"| node_topk
node_coordinator -->|"formats results"| node_result
node_cli -->|"submits query"| node_coordinator
node_cli -->|"loads or saves index"| node_storage
node_ingestion -->|"persists index"| node_storage
node_user -->|"uses client"| node_sdk
node_sdk -->|"sends REST requests"| node_http
node_caller -->|"calls API"| node_http
node_caller -->|"uses RPC contract"| node_grpc
node_server -->|"starts gateway"| node_http
node_server -->|"coordinates queries"| node_coordinator
node_http -->|"handles engine actions"| node_engine
node_http -->|"runs searches"| node_coordinator
node_grpc -->|"defines RPCs"| node_server
node_server -->|"hydrates or persists"| node_storage
node_http -->|"serializes response"| node_result
node_grpc -->|"defines result messages"| node_result

click node_cli "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-cli/src/main.rs"
click node_sdk "https://github.com/theproject24/zynsearch/tree/master/sdks"
click node_ingestion "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/ingestion.rs"
click node_normalization "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/document_ingest.rs"
click node_crawler "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/crawler.rs"
click node_engine "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/engine.rs"
click node_analyzer "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/analyzer.rs"
click node_index "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/index.rs"
click node_coordinator "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/query_pipeline.rs"
click node_searcher "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/searcher.rs"
click node_bm25 "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/bm25.rs"
click node_topk "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/top_k.rs"
click node_storage "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/storage.rs"
click node_server "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-server/src/main.rs"
click node_http "https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-server/src/http.rs"
click node_grpc "https://github.com/theproject24/zynsearch/blob/master/proto/zynsearch/v1/zynsearch.proto"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_sdk,node_user toneBlue
class node_ingestion,node_normalization,node_crawler,node_s3 toneAmber
class node_engine,node_analyzer,node_index,node_coordinator,node_searcher,node_bm25,node_topk,node_caller toneMint
class node_server,node_http,node_grpc toneRose
class node_storage,node_local,node_cliuser,node_result toneIndigo
```

## Components

### Clients

- [CLI](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-cli/src/main.rs): `crates/zynsearch-cli/src/main.rs`
- [Language SDKs](https://github.com/theproject24/zynsearch/tree/master/sdks): `sdks`

### Document ingestion

- [Source ingestion](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/ingestion.rs): `crates/zynsearch-core/src/ingestion.rs`
- [Document normalization](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/document_ingest.rs): `crates/zynsearch-core/src/document_ingest.rs`
- [Directory crawler](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/crawler.rs): `crates/zynsearch-core/src/crawler.rs`

### Search core

- [Search engine](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/engine.rs): `crates/zynsearch-core/src/engine.rs`
- [Text analyzer](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/analyzer.rs): `crates/zynsearch-core/src/analyzer.rs`
- [Inverted index](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/index.rs): `crates/zynsearch-core/src/index.rs`
- [Query coordinator](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/query_pipeline.rs): `crates/zynsearch-core/src/query_pipeline.rs`
- [Retrieval engine](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/searcher.rs): `crates/zynsearch-core/src/searcher.rs`
- [BM25 ranking](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/bm25.rs): `crates/zynsearch-core/src/bm25.rs`
- [Top-K collector](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/top_k.rs): `crates/zynsearch-core/src/top_k.rs`

### Server transports

- [Server runtime](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-server/src/main.rs): `crates/zynsearch-server/src/main.rs`
- [REST gateway](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-server/src/http.rs): `crates/zynsearch-server/src/http.rs`
- [gRPC contract](https://github.com/theproject24/zynsearch/blob/master/proto/zynsearch/v1/zynsearch.proto): `proto/zynsearch/v1/zynsearch.proto`

### Persistence

- [Index persistence](https://github.com/theproject24/zynsearch/blob/master/crates/zynsearch-core/src/storage.rs): `crates/zynsearch-core/src/storage.rs`

### Other components

- **Search user**
- **Local corpus**
- **S3 objects**
- **CLI operator**
- **API caller**
- **Search results**

## Connections

- CLI operator → CLI: runs
- CLI → Source ingestion: starts ingestion
- Local corpus → Source ingestion: supplies files
- S3 objects → Source ingestion: supplies objects
- Source ingestion → Directory crawler: crawls directory
- Source ingestion → Document normalization: normalizes content
- Source ingestion → Search engine: indexes documents
- Document normalization → Text analyzer: provides text
- Search engine → Text analyzer: analyzes text
- Search engine → Inverted index: updates index
- Search engine → Retrieval engine: executes search
- Retrieval engine → Text analyzer: analyzes query
- Retrieval engine → Inverted index: reads postings
- Retrieval engine → BM25 ranking: scores matches
- Query coordinator → Search engine: dispatches shards
- Query coordinator → Top-K collector: collects results
- Query coordinator → Search results: formats results
- CLI → Query coordinator: submits query
- CLI → Index persistence: loads or saves index
- Source ingestion → Index persistence: persists index
- Search user → Language SDKs: uses client
- Language SDKs → REST gateway: sends REST requests
- API caller → REST gateway: calls API
- API caller → gRPC contract: uses RPC contract
- Server runtime → REST gateway: starts gateway
- Server runtime → Query coordinator: coordinates queries
- REST gateway → Search engine: handles engine actions
- REST gateway → Query coordinator: runs searches
- gRPC contract → Server runtime: defines RPCs
- Server runtime → Index persistence: hydrates or persists
- REST gateway → Search results: serializes response
- gRPC contract → Search results: defines result messages