# sjhorn/dbul architecture

> Architecture diagram of the GitHub repository sjhorn/dbul, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sjhorn/dbul
- Repository: https://github.com/sjhorn/dbul

## Overview

##### Project type and purpose
- “dbul” is a Dart library implementing a full-text search engine via an inverted index. It provides tokenization, filtering, and index-building capabilities, and exposes an executable (`bin/dbul.dart`) as well as a reusable API (`lib/dbul.dart`, `lib/tokenizer.dart`).

##### File structure
- Top-level directories:
- bin/ – the CLI entry point (`dbul.dart`)
- lib/ – public API modules (`dbul.dart`) and internal modules (`tokenizer.dart`)
- test/ – automated tests (`dbul_test.dart`)
- Configs and metadata: `pubspec.yaml`, analysis options, README, LICENSE

##### README for insights
- Core components: document ingestion, tokenization, filtering, inverted index construction, query lookup
- External inspirations: Wikipedia’s inverted index example, Python/Lunr.js implementations
- Design intent: maximize recall/precision via configurable filters (stop-words, stemming, n-grams)

##### Main components
a. Document Source
- JSON files or any text corpus (e.g., `movies.json`, user-supplied docs)
b. Tokenizer Module
- `lib/tokenizer.dart`: splits raw text into tokens
c. Filter/Transform Pipeline
- operations like lowercasing, stop-word removal, stemming, n-gram generation
d. Index Builder
- `lib/dbul.dart`: consumes filtered tokens and builds the inverted index data structure (map of token → document IDs)
e. Index Storage
- in-memory data store (potential JSON serialization)
f. Query Interface / CLI
- `bin/dbul.dart`: exposes commands to index documents or perform lookup
g. Tests
- `test/dbul_test.dart`: verifies correctness of tokenization, filtering, and index outputs

##### Relationships and interactions
- Index Builder ↔ Index Storage (bidirectional for build & persist/serialize)

##### Architectural patterns & principles
- Pipeline pattern: clear stages (tokenize → filter → index)
- Separation of concerns: tokenizer, filter logic, indexer are decoupled
- Single-process, in-memory data store for low-volume corpora, with potential to extend persistence layer
- Open-closed principle: filter pipeline is configurable and extensible without modifying core indexer

##### Relevant technologies & frameworks
- Dart SDK (language & runtime)
- pubspec.yaml dependencies (none external, pure Dart)
- JSON handling (built-in Dart libraries)
- Testing: `package:test`

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Ingestion Layer"
        DS[/"Document Source\n(movies.json)"/]:::ingestion
    end

    subgraph "Library (lib/)"
        TM["Tokenizer Module"]:::processing
        FP["Filter/Transform Pipeline"]:::processing
        IB["Index Builder"]:::processing
        IS[(Index Storage)]:::storage
        API["Public API"]:::interface
    end

    subgraph "CLI Interface (bin/)"
        CLI["CLI / Query Interface"]:::interface
    end

    subgraph "Tests (test/)"
        TEST["dbul_test.dart"]:::test
    end

    DS -->|"raw text"| TM
    TM -->|"tokens"| FP
    FP -->|"filtered tokens"| IB
    IB <-->|"build & serialize / lookup"| IS
    CLI -->|"build index"| IB
    CLI -->|"search query"| IB
    API -->|"build index"| IB
    API -->|"search query"| IB
    TEST -.-> TM
    TEST -.-> FP
    TEST -.-> IB
    TEST -.-> IS

    click DS "https://github.com/sjhorn/dbul/blob/main/movies.json"
    click TM "https://github.com/sjhorn/dbul/blob/main/lib/tokenizer.dart"
    click FP "https://github.com/sjhorn/dbul/blob/main/lib/dbul.dart"
    click IB "https://github.com/sjhorn/dbul/blob/main/lib/dbul.dart"
    click IS "https://github.com/sjhorn/dbul/blob/main/lib/dbul.dart"
    click API "https://github.com/sjhorn/dbul/blob/main/lib/dbul.dart"
    click CLI "https://github.com/sjhorn/dbul/blob/main/bin/dbul.dart"
    click TEST "https://github.com/sjhorn/dbul/blob/main/test/dbul_test.dart"

    classDef ingestion fill:#a6cee3,stroke:#333,stroke-width:1px
    classDef processing fill:#b2df8a,stroke:#333,stroke-width:1px
    classDef storage fill:#fb9a99,stroke:#333,stroke-width:1px
    classDef interface fill:#cab2d6,stroke:#333,stroke-width:1px
    classDef test fill:#cccccc,stroke:#333,stroke-width:1px
```
