# ddmitov/reteti architecture

> Architecture diagram of the GitHub repository ddmitov/reteti, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ddmitov/reteti
- Repository: https://github.com/ddmitov/reteti

## Overview

##### Main Components
- Indexer Module: Represented by components and scripts such as demo_indexer.py along with the core files (reteti_core.py, reteti_file.py, reteti_text.py). This module is responsible for:
- Splitting texts into words using normalizers and pre-tokenizers (from the Tokenizers Python module).
- Hashing these words.
- Partitioning and writing the hashed data (including positions) into segmented Parquet files.
- Searcher Module: Highlight demo_searcher.py which demonstrates how search requests are handled. This module:
- Processes search queries (converting query words into their hashes).
- Uses DuckDB SQL to search only within the relevant partitions (the Parquet files corresponding to the hash prefixes involved).
- External Object Storage: Although not a directory in the project, the README explains that the partitioned Parquet indices and texts are stored in an object storage system (e.g., managed by Tigris Data or cloud providers). This acts as the persistence layer.
- Execution/Deployment Environment: The project contains configuration files like Dockerfile, .dockerignore, fly.toml, etc., indicating that it is containerized and can be deployed in serverless environments or on platforms like Fly.io.

##### Relationships and Interactions
- The “Indexer” reads original texts, processes them using text normalization, and then outputs hashed values along with positional metadata to the partitioned index files in object storage.
- The “Searcher” receives a query, processes the search request (again using text tokenizers to convert the query into hashed words), and interacts with the object storage by selecting only the partitions (Parquet files) that might contain candidate matching data.
- DuckDB is the query engine invoked during search, and it interacts with these Parquet files to execute SQL queries for filtering and ranking.
- The core module files (reteti_core.py, reteti_file.py, reteti_text.py) provide low-level utilities that support both indexing and searching.
- The overall architecture emphasizes decoupling – the indexing process is separate from the search handling process, which fits into a serverless/scale-to-zero pattern.

##### Important Architectural Patterns / Design Principles
- Separation of Concerns: Clearly separated modules for indexing and search processing.
- Decoupled Storage and Compute: The design enables the compute processes (indexing and searching) to work independently from the storage, facilitating scalability and operational flexibility.
- Partitioned Data Indexing: The use of partitioning via hashed words stored as Parquet files ensures that only relevant data is loaded during search.
- Serverless and Scale-to-Zero Readiness: Configuration for serverless environments (such as in fly.toml and Dockerfile) indicates adaptability to modern cloud environments.

##### Relevant Technologies, Frameworks, or Libraries
- Python and Tokenizers: For text splitting and normalization.
- DuckDB: For SQL-based search on Parquet files.
- Parquet: As the storage format for partitioned indices.
- Docker, Dockerfile, and Fly.io deployment configuration (fly.toml): For containerization and cloud deployment.
- Object storage: Although abstract in the repository, it is crucial for persisting the index data.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Indexer Pipeline
    subgraph "Indexer Pipeline"
        IT1["Raw Texts"]:::input
        IT2["Text Processing (Normalization & Tokenization)"]:::process
        IT3["Hashing & Partitioning"]:::process
        IT4["Indexer Module (demo_indexer.py)"]:::module
        IT1 --> IT2
        IT2 --> IT3
        IT4 --> IT2
        IT3 --> OS["Object Storage (Parquet Files)"]:::storage
    end

    %% Shared Application Core Modules
    subgraph "Application Core"
        AC1["reteti_core.py"]:::utility
        AC2["reteti_file.py"]:::utility
        AC3["reteti_text.py"]:::utility
    end
    %% Utilization of Application Core by Indexer
    IT2 --- AC1

    %% Searcher Pipeline
    subgraph "Searcher Pipeline"
        ST1["Search Request (Query)"]:::input
        ST2["Query Tokenization"]:::process
        ST3["Selective Data Retrieval"]:::process
        ST4["DuckDB SQL Engine"]:::engine
        ST5["Searcher Module (demo_searcher.py)"]:::module
        ST1 --> ST2
        ST2 --> ST3
        ST3 --> ST4
        ST5 --> ST2
        OS --> ST3
    end

    %% Deployment & Configuration Assets
    subgraph "Deployment & Configuration"
        DEP_ENV["Deployment Environment (Containerized & Serverless)"]:::deployment
        CF1["Dockerfile"]:::deployment
        CF2[".dockerignore"]:::deployment
        CF3["fly.toml"]:::deployment
        ASSETS["Supporting Assets"]:::asset
        DEP_ENV --> CF1
        DEP_ENV --> CF2
        DEP_ENV --> CF3
    end

    %% Click Events for Mapped Components
    click IT4 "https://github.com/ddmitov/reteti/blob/main/demo_indexer.py"
    click AC1 "https://github.com/ddmitov/reteti/blob/main/reteti_core.py"
    click AC2 "https://github.com/ddmitov/reteti/blob/main/reteti_file.py"
    click AC3 "https://github.com/ddmitov/reteti/blob/main/reteti_text.py"
    click ST5 "https://github.com/ddmitov/reteti/blob/main/demo_searcher.py"
    click CF1 "https://github.com/ddmitov/reteti/tree/main/Dockerfile"
    click CF2 "https://github.com/ddmitov/reteti/blob/main/.dockerignore"
    click CF3 "https://github.com/ddmitov/reteti/blob/main/fly.toml"
    click ASSETS "https://github.com/ddmitov/reteti/tree/main/assets"

    %% Styles
    classDef input fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef process fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef storage fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef engine fill:#F5B7B1,stroke:#CD6155,stroke-width:2px;
    classDef module fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef utility fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef deployment fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef asset fill:#F1948A,stroke:#C0392B,stroke-width:2px;
```
