How sjhorn/dbul works
Project type and purpose
• “dbul” is a Dart library implementing a full-text search engine via an inverted index. It provides tokenization, filtering, and index-building capabilities, and exposes an executable (bin/dbul.dart) as well as a reusable API (lib/dbul.dart, lib/tokenizer.dart).
File structure
• Top-level directories:
• bin/ – the CLI entry point (dbul.dart)
• lib/ – public API modules (dbul.dart) and internal modules (tokenizer.dart)
• test/ – automated tests (dbul_test.dart)
• Configs and metadata: pubspec.yaml, analysis options, README, LICENSE
README for insights
• Core components: document ingestion, tokenization, filtering, inverted index construction, query lookup
• External inspirations: Wikipedia’s inverted index example, Python/Lunr.js implementations
• Design intent: maximize recall/precision via configurable filters (stop-words, stemming, n-grams)
Main components
a. Document Source
• JSON files or any text corpus (e.g., movies.json, user-supplied docs)
b. Tokenizer Module
• lib/tokenizer.dart: splits raw text into tokens
c. Filter/Transform Pipeline
• operations like lowercasing, stop-word removal, stemming, n-gram generation
d. Index Builder
• lib/dbul.dart: consumes filtered tokens and builds the inverted index data structure (map of token → document IDs)
e. Index Storage
• in-memory data store (potential JSON serialization)
f. Query Interface / CLI
• bin/dbul.dart: exposes commands to index documents or perform lookup
g. Tests
• test/dbul_test.dart: verifies correctness of tokenization, filtering, and index outputs
Relationships and interactions
• Index Builder ↔ Index Storage (bidirectional for build & persist/serialize)
Architectural patterns & principles
• Pipeline pattern: clear stages (tokenize → filter → index)
• Separation of concerns: tokenizer, filter logic, indexer are decoupled
• Single-process, in-memory data store for low-volume corpora, with potential to extend persistence layer
• Open-closed principle: filter pipeline is configurable and extensible without modifying core indexer
Relevant technologies & frameworks
• Dart SDK (language & runtime)
• pubspec.yaml dependencies (none external, pure Dart)
• JSON handling (built-in Dart libraries)
• Testing: package:test