# quickwit-oss/tantivy architecture

> Architecture diagram of the GitHub repository quickwit-oss/tantivy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quickwit-oss/tantivy
- Repository: https://github.com/quickwit-oss/tantivy

## Overview

##### Main Components
- Core Library & Engine: Represented by the “src” directory and its submodules (e.g., core, index, query, schema, postings, directory, fastfield, aggregation, snippet, tokenizer). This is the heart of Tantivy.
- Indexing Components:
– Index Writer (part of the “indexer” module) which handles document ingestion (including deletion and incremental indexing).
– Segment Management: Handling segment creation, merging, and the writing process to disk.
- Query Processing:
– Query Parser (see the “query_parser” module), query classes (term, boolean, phrase, etc.), and scoring components.
- Tokenizer:
– Based on the “tokenizer” and “tokenizer-api” directories. This component is responsible for breaking text into tokens (including language-specific support for stemming and filtering).
- Directory & Storage Management:
– “directory” module which abstracts file system interactions (MmapDirectory, RamDirectory, etc.).
– “sstable” and “termdict” modules for on-disk storage structure and inverted index management.
- Auxiliary Components:
– Fast Field management (for numeric or lightweight field retrieval).
– Aggregation functions and collectors.
– Utility libraries such as “common”, “stacker” (for memory management), and “bitpacker” (for SIMD integer compression).
- External Interfaces & Tools:
– CLI (e.g., “columnar-cli”, “tokenizer-api”) for demo/search interfaces.
– Examples and benchmarks (found in “examples” and “benches”) to illustrate performance and usage.

##### Relationships and Interactions
- Inside the Index Writer, detail the process of creating segments and merging them. Include a component for “Merge Policy” (e.g., the “log_merge_policy” module).
- Link the Directory abstraction (with its components like “mmap_directory” and “ram_directory”) to storage modules (sstable, termdict) to show how data is persisted.
- That the Aggregation Collector and Fast Fields are supplementary components that provide statistical and secondary data look-ups during query execution.
- Optional: Represent ancillary libraries (like “stacker” for memory management or “common” for utility functions) as helper libraries that support the core workflow.

##### Architectural Patterns and Design Principles
- Modular Architecture: The system is clearly divided into distinct modules (indexing, query, tokenizer, storage) which interact through well-defined interfaces.
- Separation of Concerns: Tokenizing, indexing, query parsing, and storage are encapsulated in independent modules. This enables pluggable tokenizers or custom merging policies.
- Performance and Concurrency: The design supports multi-threaded indexing (as noted by benchmarks and “merge_policy”) and is optimized for speedy searches as well as low startup time.
- Use of Abstraction Layers: The “directory” abstraction decouples storage implementation from the rest of the system—allowing different backends (memory mapped or in-memory) to be swapped.

##### Specific Elements
- Each major component should have a clear label:
- “Tokenizer” (with subcomponents for filtering, stemming, and custom tokenizers).
- “Index Writer” and “Segment Manager” (showing the merge process and document handling).
- “Query Parser & Searcher” (indicating different query types: term, Boolean, phrase, etc.).
- “Directory & Storage” (including abstractions for file-based storage, MmapDirectory, and sstable/termdict storage engines).
- “Aggregation & Fast Fields” as auxiliary data retrieval components.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "External Inputs"
        DIn["Documents"]:::external
        QIn["Queries"]:::external
    end

    subgraph "Core Engine"
        CL["Core Library"]:::engine
        Token["Tokenizer"]:::engine
        IW["Index Writer (Document Ingestion)"]:::engine
        SM["Segment Manager"]:::engine
        QPS["Query Parser & Searcher"]:::engine
        PI["Inverted Index (Postings)"]:::engine
    end

    subgraph "Storage Layer"
        DS["Directory & Storage"]:::storage
        SST["SSTable"]:::storage
        TD["TermDict"]:::storage
    end

    subgraph "Auxiliary Components"
        FF["Fast Field Management"]:::aux
        AG["Aggregation Functions & Collectors"]:::aux
        CU["Common Utilities"]:::aux
        BP["SIMD Compression (Bitpacker)"]:::aux
        ST["Memory Management (Stacker)"]:::aux
    end

    subgraph "External Interfaces & Tools"
        CLI["CLI Tools"]:::external
        EX["Examples & Benchmarks"]:::external
    end

    %% Data Ingestion Flow
    DIn -->|"tokenizes"| Token
    Token -->|"processes"| IW
    IW -->|"creates segments"| SM
    SM -->|"persists segments"| DS
    DS -->|"stores data"| SST
    DS -->|"stores term dictionaries"| TD
    SM -->|"updates"| PI

    %% Query Processing Flow
    QIn -->|"parsedby"| QPS
    PI -->|"retrievedby"| QPS
    QPS -->|"fetches"| FF
    QPS -->|"aggregates"| AG

    %% Core Library relationships
    CL -->|"contains"| Token
    CL -->|"contains"| IW
    CL -->|"contains"| SM
    CL -->|"contains"| QPS
    CL -->|"contains"| PI

    %% Auxiliary Utilities support
    CU -.-> IW
    CU -.-> SM
    CU -.-> QPS
    BP -.-> IW
    BP -.-> SM
    ST -.-> IW
    ST -.-> SM

    %% External Interfaces interactions
    CLI -->|"submits"| QPS
    EX -->|"demonstrates"| QPS

    %% Click Events
    click CL "https://github.com/quickwit-oss/tantivy/tree/main/src/"
    click Token "https://github.com/quickwit-oss/tantivy/tree/main/src/tokenizer/"
    click IW "https://github.com/quickwit-oss/tantivy/blob/main/src/indexer/index_writer.rs"
    click SM "https://github.com/quickwit-oss/tantivy/blob/main/src/indexer/segment_manager.rs"
    click QPS "https://github.com/quickwit-oss/tantivy/tree/main/src/query/query_parser/"
    click DS "https://github.com/quickwit-oss/tantivy/tree/main/src/directory/"
    click FF "https://github.com/quickwit-oss/tantivy/tree/main/src/fastfield/"
    click AG "https://github.com/quickwit-oss/tantivy/tree/main/src/aggregation/"
    click CU "https://github.com/quickwit-oss/tantivy/tree/main/common/"
    click BP "https://github.com/quickwit-oss/tantivy/tree/main/bitpacker/"
    click ST "https://github.com/quickwit-oss/tantivy/tree/main/stacker/"
    click CLI "https://github.com/quickwit-oss/tantivy/tree/main/columnar/columnar-cli/"
    click EX "https://github.com/quickwit-oss/tantivy/tree/main/examples/"
    click TD "https://github.com/quickwit-oss/tantivy/tree/main/src/termdict/"
    click SST "https://github.com/quickwit-oss/tantivy/tree/main/sstable/"

    %% Styles
    classDef external fill:#F9E79F,stroke:#D4AC0D,stroke-width:2px;
    classDef engine fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef storage fill:#AED6F1,stroke:#2E86C1,stroke-width:2px;
    classDef aux fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
```
