# rajal-ui/minies architecture

> Architecture diagram of the GitHub repository rajal-ui/minies, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajal-ui/minies
- Repository: https://github.com/rajal-ui/minies

## Overview

MiniES is a single-binary log ingestion and search service. Producers submit validated records through HTTP into a bounded asynchronous pipeline; records are WAL-backed, token-indexed, and persisted as segments. Search parses boolean, wildcard, and fuzzy queries and resolves matches through the index and stored records. The README documents KMP matching, background compaction, and graceful shutdown; sampled code supports the main flow, but does not show compaction scheduling or KMP wiring. The metrics package exists, though the sampled stats handler reports pipeline statistics.

Main entry points: [`internal/pipeline/eventloop.go`](https://github.com/rajal-ui/minies/blob/main/internal/pipeline/eventloop.go) (Event loop), [`internal/search/executor.go`](https://github.com/rajal-ui/minies/blob/main/internal/search/executor.go) (Search executor), [`internal/ingest/handler.go`](https://github.com/rajal-ui/minies/blob/main/internal/ingest/handler.go) (HTTP handlers), [`internal/index/memindex.go`](https://github.com/rajal-ui/minies/blob/main/internal/index/memindex.go) (Sharded index)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and clients"]
  node_search_client["Interactive client<br/>[main.go]"]
  node_server["MiniES server<br/>[main.go]"]
  node_http_api["HTTP handlers<br/>[handler.go]"]
end

subgraph group_ingest["Ingestion pipeline"]
  node_records["Log records<br/>[record.go]"]
  node_event_loop["Event loop<br/>[eventloop.go]"]
  node_backpressure["Bounded intake<br/>[backpressure.go]"]
  node_tokenizer["Token normalization<br/>[tokenizer.go]"]
end

subgraph group_indexstore["Index and storage"]
  node_wal[("Write-ahead log<br/>[wal.go]")]
  node_inverted_index[("Sharded index<br/>[memindex.go]")]
  node_shards["Index shards<br/>[shard.go]"]
  node_trie["Wildcard trie<br/>[trie.go]"]
  node_flusher["Segment flusher<br/>[flush.go]"]
  node_segments[("Disk segments<br/>[segment.go]")]
  node_compaction["Segment compaction<br/>[merge.go]"]
end

subgraph group_query["Query and observability"]
  node_parser["Query parser<br/>[parser.go]"]
  node_executor["Search executor<br/>[executor.go]"]
  node_matching["String matching"]
  node_metrics["In-process metrics<br/>[metrics.go]"]
end

node_producers(("Log producers"))

node_producers -->|"submit records"| node_http_api
node_search_client -->|"HTTP requests"| node_http_api
node_http_api -->|"decode and validate"| node_records
node_http_api -->|"enqueue records"| node_event_loop
node_event_loop -->|"bounded intake"| node_backpressure
node_event_loop -->|"append and replay"| node_wal
node_event_loop -->|"normalize tokens"| node_tokenizer
node_event_loop -->|"add postings"| node_inverted_index
node_event_loop -->|"insert tokens"| node_trie
node_inverted_index -->|"partition postings"| node_shards
node_event_loop -->|"flush records"| node_flusher
node_flusher -->|"write segments"| node_segments
node_compaction -->|"merge segments"| node_segments
node_http_api -->|"run query"| node_executor
node_executor -->|"parse query"| node_parser
node_executor -->|"lookup postings"| node_inverted_index
node_executor -->|"prefix lookup"| node_trie
node_executor -->|"match terms"| node_matching
node_executor -->|"read records"| node_segments
node_http_api -->|"read pipeline stats"| node_event_loop
node_event_loop -->|"report index stats"| node_inverted_index

click node_search_client "https://github.com/rajal-ui/minies/blob/main/cmd/minies-search/main.go"
click node_server "https://github.com/rajal-ui/minies/blob/main/cmd/minies/main.go"
click node_http_api "https://github.com/rajal-ui/minies/blob/main/internal/ingest/handler.go"
click node_records "https://github.com/rajal-ui/minies/blob/main/pkg/logrecord/record.go"
click node_event_loop "https://github.com/rajal-ui/minies/blob/main/internal/pipeline/eventloop.go"
click node_backpressure "https://github.com/rajal-ui/minies/blob/main/internal/pipeline/backpressure.go"
click node_tokenizer "https://github.com/rajal-ui/minies/blob/main/internal/tokenizer/tokenizer.go"
click node_wal "https://github.com/rajal-ui/minies/blob/main/internal/storage/wal.go"
click node_inverted_index "https://github.com/rajal-ui/minies/blob/main/internal/index/memindex.go"
click node_shards "https://github.com/rajal-ui/minies/blob/main/internal/index/shard.go"
click node_trie "https://github.com/rajal-ui/minies/blob/main/internal/index/trie.go"
click node_flusher "https://github.com/rajal-ui/minies/blob/main/internal/storage/flush.go"
click node_segments "https://github.com/rajal-ui/minies/blob/main/internal/storage/segment.go"
click node_compaction "https://github.com/rajal-ui/minies/blob/main/internal/storage/merge.go"
click node_parser "https://github.com/rajal-ui/minies/blob/main/internal/search/parser.go"
click node_executor "https://github.com/rajal-ui/minies/blob/main/internal/search/executor.go"
click node_matching "https://github.com/rajal-ui/minies/tree/main/internal/search"
click node_metrics "https://github.com/rajal-ui/minies/blob/main/internal/metrics/metrics.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_search_client,node_server,node_http_api toneBlue
class node_records,node_event_loop,node_backpressure,node_tokenizer toneAmber
class node_wal,node_inverted_index,node_shards,node_trie,node_flusher,node_segments,node_compaction toneMint
class node_parser,node_executor,node_matching,node_metrics toneRose
class node_producers toneIndigo
```

## Components

### API and clients

- [Interactive client](https://github.com/rajal-ui/minies/blob/main/cmd/minies-search/main.go): `cmd/minies-search/main.go`
- [MiniES server](https://github.com/rajal-ui/minies/blob/main/cmd/minies/main.go): `cmd/minies/main.go`
- [HTTP handlers](https://github.com/rajal-ui/minies/blob/main/internal/ingest/handler.go): `internal/ingest/handler.go`

### Ingestion pipeline

- [Log records](https://github.com/rajal-ui/minies/blob/main/pkg/logrecord/record.go): `pkg/logrecord/record.go`
- [Event loop](https://github.com/rajal-ui/minies/blob/main/internal/pipeline/eventloop.go): `internal/pipeline/eventloop.go`
- [Bounded intake](https://github.com/rajal-ui/minies/blob/main/internal/pipeline/backpressure.go): `internal/pipeline/backpressure.go`
- [Token normalization](https://github.com/rajal-ui/minies/blob/main/internal/tokenizer/tokenizer.go): `internal/tokenizer/tokenizer.go`

### Index and storage

- [Write-ahead log](https://github.com/rajal-ui/minies/blob/main/internal/storage/wal.go): `internal/storage/wal.go`
- [Sharded index](https://github.com/rajal-ui/minies/blob/main/internal/index/memindex.go): `internal/index/memindex.go`
- [Index shards](https://github.com/rajal-ui/minies/blob/main/internal/index/shard.go): `internal/index/shard.go`
- [Wildcard trie](https://github.com/rajal-ui/minies/blob/main/internal/index/trie.go): `internal/index/trie.go`
- [Segment flusher](https://github.com/rajal-ui/minies/blob/main/internal/storage/flush.go): `internal/storage/flush.go`
- [Disk segments](https://github.com/rajal-ui/minies/blob/main/internal/storage/segment.go): `internal/storage/segment.go`
- [Segment compaction](https://github.com/rajal-ui/minies/blob/main/internal/storage/merge.go): `internal/storage/merge.go`

### Query and observability

- [Query parser](https://github.com/rajal-ui/minies/blob/main/internal/search/parser.go): `internal/search/parser.go`
- [Search executor](https://github.com/rajal-ui/minies/blob/main/internal/search/executor.go): `internal/search/executor.go`
- [String matching](https://github.com/rajal-ui/minies/tree/main/internal/search): `internal/search`
- [In-process metrics](https://github.com/rajal-ui/minies/blob/main/internal/metrics/metrics.go): `internal/metrics/metrics.go`

### Other components

- **Log producers**

## Connections

- Log producers → HTTP handlers: submit records
- Interactive client → HTTP handlers: HTTP requests
- HTTP handlers → Log records: decode and validate
- HTTP handlers → Event loop: enqueue records
- Event loop → Bounded intake: bounded intake
- Event loop → Write-ahead log: append and replay
- Event loop → Token normalization: normalize tokens
- Event loop → Sharded index: add postings
- Event loop → Wildcard trie: insert tokens
- Sharded index → Index shards: partition postings
- Event loop → Segment flusher: flush records
- Segment flusher → Disk segments: write segments
- Segment compaction → Disk segments: merge segments
- HTTP handlers → Search executor: run query
- Search executor → Query parser: parse query
- Search executor → Sharded index: lookup postings
- Search executor → Wildcard trie: prefix lookup
- Search executor → String matching: match terms
- Search executor → Disk segments: read records
- HTTP handlers → Event loop: read pipeline stats
- Event loop → Sharded index: report index stats