How mush1e/indexstream-v2 works
IndexStream is a local web-search engine: users search indexed pages or submit crawl requests through its HTTP interface. The visible processing stages include crawling pages, extracting and tokenizing text, building an inverted index, and ranking search results. The index uses a multi-layer cache, while the repository also defines database persistence helpers. The README is too sparse to establish more product behavior, and the excerpts do not show how crawl requests are dispatched or how database helpers are wired into the application.
Main entry points: indexer.go Inverted index · main.go Application startup · extractor.go Text extractor · server.go HTTP server
Components 15
HTTP interface
- Application startup
cmd/index-stream/main.go - HTTP server
internal/server/server.go - Request logger
internal/middleware/middleware.go - HTTP handlers
internal/handler/handler.go
Page ingestion
- Web crawler
internal/service/crawler.go - Text extractor
internal/service/extractor.go - Local HTML files
Search and indexing
- Tokenizer
internal/service/tokenizer.go - Inverted index
internal/service/indexer.go - Enhanced search
internal/service/searcher.go
State and configuration
- Multi-layer cache
internal/cache/cache.go - Database helpers
internal/database/db.go - Runtime config
config/config.go
Other components
- Search user
- Web pages
Connections 20
- Search user → HTTP server: sends requests
- HTTP server → Request logger: wraps requests
- HTTP server → HTTP handlers: dispatches routes
- HTTP handlers → Inverted index: searches index
- HTTP handlers → Inverted index: reads statistics
- Web pages → Web crawler: serves pages
- Web crawler → Local HTML files: writes pages
- Text extractor → Local HTML files: reads pages
- Text extractor → Tokenizer: tokenizes text
- Text extractor → Inverted index: adds documents
- Inverted index → Tokenizer: tokenizes queries
- Enhanced search → Tokenizer: tokenizes queries
- Enhanced search → Inverted index: queries index
- Inverted index → Multi-layer cache: reads and writes
- Application startup → Runtime config: loads settings
- Application startup → HTTP server: starts server
- Application startup → Text extractor: starts extractor
- Application startup → Inverted index: prewarms cache
- Web crawler → Runtime config: reads crawl settings
- Database helpers → Local HTML files