How yelp/nrtsearch works
nrtSearch is a Lucene-backed search server exposed primarily through gRPC, with optional REST APIs. The user workflow is to create and start an index, register its field schema, stream documents for indexing, and issue searches whose results are returned through the search API. Distinct operational paths include primary-to-replica near-real-time replication and remote backup/restore. The graph preserves major documented search capabilities and plugin extensibility. Some relationships rely on README documentation or SOURCE INDEX references; unsampled implementations and wiring are not asserted.
Main entry points: SearchHandler.java Search execution · handler Request handlers · NrtsearchServer.java gRPC server · SearchStreamHandler.java Streaming search
Components 22
Client API
- gRPC client
clientlib/src/main/java/com/yelp/nrtsearch/clientlib/grpc/NrtSearchClient.java - gRPC server
src/main/java/com/yelp/nrtsearch/server/grpc/NrtsearchServer.java - Request handlers
src/main/java/com/yelp/nrtsearch/server/handler
Indexing and Search
- Global index state
src/main/java/com/yelp/nrtsearch/server/state/GlobalState.java - Field definitions
src/main/java/com/yelp/nrtsearch/server/field/FieldDef.java - Field definition factory
src/main/java/com/yelp/nrtsearch/server/field/FieldDefCreator.java - Index state
src/main/java/com/yelp/nrtsearch/server/index/IndexState.java - Shard and writer
src/main/java/com/yelp/nrtsearch/server/index/ShardState.java - Streaming search
src/main/java/com/yelp/nrtsearch/server/handler/stream/SearchStreamHandler.java - Search execution
src/main/java/com/yelp/nrtsearch/server/handler/SearchHandler.java - Search context
src/main/java/com/yelp/nrtsearch/server/search/SearchContext.java - Query and retrieval
src/main/java/com/yelp/nrtsearch/server/query - Facets and collectors
src/main/java/com/yelp/nrtsearch/server/search/collectors - Highlighting
src/main/java/com/yelp/nrtsearch/server/highlights/HighlighterService.java
Replication and Durability
- Primary replication
src/main/java/com/yelp/nrtsearch/server/nrt/NRTPrimaryNode.java - Replication client
src/main/java/com/yelp/nrtsearch/server/grpc/ReplicationServerClient.java - NRT data manager
src/main/java/com/yelp/nrtsearch/server/nrt/NrtDataManager.java - Remote storage
src/main/java/com/yelp/nrtsearch/server/remote/RemoteBackend.java
Extensions and Operations
- Server configuration
src/main/java/com/yelp/nrtsearch/server/config/NrtsearchConfig.java - Plugin loading
src/main/java/com/yelp/nrtsearch/server/plugins/PluginsService.java - Monitoring
src/main/java/com/yelp/nrtsearch/server/monitoring
Other components
- Search client
Connections 20
- Search client → gRPC client: issues requests
- gRPC client → gRPC server: calls gRPC
- gRPC server → Request handlers: routes requests
- Request handlers → Global index state: accesses state
src/main/java/com/yelp/nrtsearch/server/handler/DeleteByQueryHandler.java - Request handlers → Index state: resolves index
src/main/java/com/yelp/nrtsearch/server/handler/DeleteByQueryHandler.java - Request handlers → Field definition factory: registers fields
- Field definition factory → Field definitions: creates definitions
src/main/java/com/yelp/nrtsearch/server/field/IndexableFieldDef.java - Index state → Shard and writer: owns shards
src/main/java/com/yelp/nrtsearch/server/index/IndexState.java - Streaming search → Search execution: delegates search
src/main/java/com/yelp/nrtsearch/server/handler/stream/SearchStreamHandler.java - Streaming search → Search context: builds context
src/main/java/com/yelp/nrtsearch/server/handler/stream/SearchStreamHandler.java - Search execution → Search context: uses context
src/main/java/com/yelp/nrtsearch/server/handler/SearchHandler.java - Search execution → Query and retrieval: executes queries
- Search execution → Facets and collectors: collects results
- Search execution → Highlighting: formats highlights
- Primary replication → Replication client: sends updates
- Primary replication → NRT data manager: publishes segments
src/main/java/com/yelp/nrtsearch/server/nrt/NrtDataManager.java - NRT data manager → Remote storage: uploads index files
src/main/java/com/yelp/nrtsearch/server/nrt/NrtDataManager.java - Server configuration → Plugin loading: configures plugins
src/main/java/com/yelp/nrtsearch/server/plugins/PluginsService.java - Server configuration → gRPC server: configures service
- Plugin loading → Monitoring: registers metrics
src/main/java/com/yelp/nrtsearch/server/plugins/PluginsService.java