# mush1e/indexstream-v2 architecture

> Architecture diagram of the GitHub repository mush1e/indexstream-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mush1e/indexstream-v2
- Repository: https://github.com/mush1e/indexstream-v2

## Overview

IndexStream is a local web-search engine: users search indexed pages or submit crawl requests through its HTTP interface. The visible processing stages include crawling pages, extracting and tokenizing text, building an inverted index, and ranking search results. The index uses a multi-layer cache, while the repository also defines database persistence helpers. The README is too sparse to establish more product behavior, and the excerpts do not show how crawl requests are dispatched or how database helpers are wired into the application.

Main entry points: [`internal/service/indexer.go`](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/indexer.go) (Inverted index), [`cmd/index-stream/main.go`](https://github.com/mush1e/indexstream-v2/blob/main/cmd/index-stream/main.go) (Application startup), [`internal/service/extractor.go`](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/extractor.go) (Text extractor), [`internal/server/server.go`](https://github.com/mush1e/indexstream-v2/blob/main/internal/server/server.go) (HTTP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_http["HTTP interface"]
  node_main["Application startup<br/>[main.go]"]
  node_server["HTTP server<br/>[server.go]"]
  node_middleware["Request logger<br/>[middleware.go]"]
  node_handler["HTTP handlers<br/>[handler.go]"]
end

subgraph group_crawl["Page ingestion"]
  node_crawler["Web crawler<br/>[crawler.go]"]
  node_extractor["Text extractor<br/>[extractor.go]"]
  node_htmlfiles["Local HTML files"]
end

subgraph group_search["Search and indexing"]
  node_tokenizer["Tokenizer<br/>[tokenizer.go]"]
  node_indexer["Inverted index<br/>[indexer.go]"]
  node_searcher["Enhanced search<br/>[searcher.go]"]
end

subgraph group_state["State and configuration"]
  node_cache["Multi-layer cache<br/>[cache.go]"]
  node_database["Database helpers<br/>[db.go]"]
  node_config["Runtime config<br/>[config.go]"]
end

node_user(("Search user"))
node_web(("Web pages"))

node_user -->|"sends requests"| node_server
node_server -->|"wraps requests"| node_middleware
node_server -->|"dispatches routes"| node_handler
node_handler -->|"searches index"| node_indexer
node_handler -->|"reads statistics"| node_indexer
node_web -->|"serves pages"| node_crawler
node_crawler -->|"writes pages"| node_htmlfiles
node_extractor -->|"reads pages"| node_htmlfiles
node_extractor -->|"tokenizes text"| node_tokenizer
node_extractor -->|"adds documents"| node_indexer
node_indexer -->|"tokenizes queries"| node_tokenizer
node_searcher -->|"tokenizes queries"| node_tokenizer
node_searcher -->|"queries index"| node_indexer
node_indexer -->|"reads and writes"| node_cache
node_main -->|"loads settings"| node_config
node_main -->|"starts server"| node_server
node_main -->|"starts extractor"| node_extractor
node_main -->|"prewarms cache"| node_indexer
node_crawler -->|"reads crawl settings"| node_config
node_database --> node_htmlfiles

click node_main "https://github.com/mush1e/indexstream-v2/blob/main/cmd/index-stream/main.go"
click node_server "https://github.com/mush1e/indexstream-v2/blob/main/internal/server/server.go"
click node_middleware "https://github.com/mush1e/indexstream-v2/blob/main/internal/middleware/middleware.go"
click node_handler "https://github.com/mush1e/indexstream-v2/blob/main/internal/handler/handler.go"
click node_crawler "https://github.com/mush1e/indexstream-v2/blob/main/internal/service/crawler.go"
click node_extractor "https://github.com/mush1e/indexstream-v2/blob/main/internal/service/extractor.go"
click node_tokenizer "https://github.com/mush1e/indexstream-v2/blob/main/internal/service/tokenizer.go"
click node_indexer "https://github.com/mush1e/indexstream-v2/blob/main/internal/service/indexer.go"
click node_searcher "https://github.com/mush1e/indexstream-v2/blob/main/internal/service/searcher.go"
click node_cache "https://github.com/mush1e/indexstream-v2/blob/main/internal/cache/cache.go"
click node_database "https://github.com/mush1e/indexstream-v2/blob/main/internal/database/db.go"
click node_config "https://github.com/mush1e/indexstream-v2/blob/main/config/config.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_server,node_middleware,node_handler,node_user toneBlue
class node_crawler,node_extractor,node_htmlfiles toneAmber
class node_tokenizer,node_indexer,node_searcher toneMint
class node_cache,node_database,node_config toneRose
class node_web toneIndigo
```

## Components

### HTTP interface

- [Application startup](https://github.com/mush1e/indexstream-v2/blob/main/cmd/index-stream/main.go): `cmd/index-stream/main.go`
- [HTTP server](https://github.com/mush1e/indexstream-v2/blob/main/internal/server/server.go): `internal/server/server.go`
- [Request logger](https://github.com/mush1e/indexstream-v2/blob/main/internal/middleware/middleware.go): `internal/middleware/middleware.go`
- [HTTP handlers](https://github.com/mush1e/indexstream-v2/blob/main/internal/handler/handler.go): `internal/handler/handler.go`

### Page ingestion

- [Web crawler](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/crawler.go): `internal/service/crawler.go`
- [Text extractor](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/extractor.go): `internal/service/extractor.go`
- **Local HTML files**

### Search and indexing

- [Tokenizer](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/tokenizer.go): `internal/service/tokenizer.go`
- [Inverted index](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/indexer.go): `internal/service/indexer.go`
- [Enhanced search](https://github.com/mush1e/indexstream-v2/blob/main/internal/service/searcher.go): `internal/service/searcher.go`

### State and configuration

- [Multi-layer cache](https://github.com/mush1e/indexstream-v2/blob/main/internal/cache/cache.go): `internal/cache/cache.go`
- [Database helpers](https://github.com/mush1e/indexstream-v2/blob/main/internal/database/db.go): `internal/database/db.go`
- [Runtime config](https://github.com/mush1e/indexstream-v2/blob/main/config/config.go): `config/config.go`

### Other components

- **Search user**
- **Web pages**

## Connections

- Search user → HTTP server: sends requests
- HTTP server → Request logger: wraps requests
- HTTP server → HTTP handlers: dispatches routes
- HTTP handlers → Inverted index: searches index
- HTTP handlers → Inverted index: reads statistics
- Web pages → Web crawler: serves pages
- Web crawler → Local HTML files: writes pages
- Text extractor → Local HTML files: reads pages
- Text extractor → Tokenizer: tokenizes text
- Text extractor → Inverted index: adds documents
- Inverted index → Tokenizer: tokenizes queries
- Enhanced search → Tokenizer: tokenizes queries
- Enhanced search → Inverted index: queries index
- Inverted index → Multi-layer cache: reads and writes
- Application startup → Runtime config: loads settings
- Application startup → HTTP server: starts server
- Application startup → Text extractor: starts extractor
- Application startup → Inverted index: prewarms cache
- Web crawler → Runtime config: reads crawl settings
- Database helpers → Local HTML files