How rajal-ui/minies works
MiniES is a single-binary log ingestion and search service. Producers submit validated records through HTTP into a bounded asynchronous pipeline; records are WAL-backed, token-indexed, and persisted as segments. Search parses boolean, wildcard, and fuzzy queries and resolves matches through the index and stored records. The README documents KMP matching, background compaction, and graceful shutdown; sampled code supports the main flow, but does not show compaction scheduling or KMP wiring. The metrics package exists, though the sampled stats handler reports pipeline statistics.
Main entry points: eventloop.go Event loop · executor.go Search executor · handler.go HTTP handlers · memindex.go Sharded index
Components 19
API and clients
- Interactive client
cmd/minies-search/main.go - MiniES server
cmd/minies/main.go - HTTP handlers
internal/ingest/handler.go
Ingestion pipeline
- Log records
pkg/logrecord/record.go - Event loop
internal/pipeline/eventloop.go - Bounded intake
internal/pipeline/backpressure.go - Token normalization
internal/tokenizer/tokenizer.go
Index and storage
- Write-ahead log
internal/storage/wal.go - Sharded index
internal/index/memindex.go - Index shards
internal/index/shard.go - Wildcard trie
internal/index/trie.go - Segment flusher
internal/storage/flush.go - Disk segments
internal/storage/segment.go - Segment compaction
internal/storage/merge.go
Query and observability
- Query parser
internal/search/parser.go - Search executor
internal/search/executor.go - String matching
internal/search - In-process metrics
internal/metrics/metrics.go
Other components
- Log producers
Connections 21
- Log producers → HTTP handlers: submit records
- Interactive client → HTTP handlers: HTTP requests
- HTTP handlers → Log records: decode and validate
- HTTP handlers → Event loop: enqueue records
- Event loop → Bounded intake: bounded intake
- Event loop → Write-ahead log: append and replay
- Event loop → Token normalization: normalize tokens
- Event loop → Sharded index: add postings
- Event loop → Wildcard trie: insert tokens
- Sharded index → Index shards: partition postings
- Event loop → Segment flusher: flush records
- Segment flusher → Disk segments: write segments
- Segment compaction → Disk segments: merge segments
- HTTP handlers → Search executor: run query
- Search executor → Query parser: parse query
- Search executor → Sharded index: lookup postings
- Search executor → Wildcard trie: prefix lookup
- Search executor → String matching: match terms
- Search executor → Disk segments: read records
- HTTP handlers → Event loop: read pipeline stats
- Event loop → Sharded index: report index stats