How ddmitov/reteti works
Main Components
• Indexer Module: Represented by components and scripts such as demo_indexer.py along with the core files (reteti_core.py, reteti_file.py, reteti_text.py). This module is responsible for:
• Splitting texts into words using normalizers and pre-tokenizers (from the Tokenizers Python module).
• Hashing these words.
• Partitioning and writing the hashed data (including positions) into segmented Parquet files.
• Searcher Module: Highlight demo_searcher.py which demonstrates how search requests are handled. This module:
• Processes search queries (converting query words into their hashes).
• Uses DuckDB SQL to search only within the relevant partitions (the Parquet files corresponding to the hash prefixes involved).
• External Object Storage: Although not a directory in the project, the README explains that the partitioned Parquet indices and texts are stored in an object storage system (e.g., managed by Tigris Data or cloud providers). This acts as the persistence layer.
• Execution/Deployment Environment: The project contains configuration files like Dockerfile, .dockerignore, fly.toml, etc., indicating that it is containerized and can be deployed in serverless environments or on platforms like Fly.io.
Relationships and Interactions
• The “Indexer” reads original texts, processes them using text normalization, and then outputs hashed values along with positional metadata to the partitioned index files in object storage.
• The “Searcher” receives a query, processes the search request (again using text tokenizers to convert the query into hashed words), and interacts with the object storage by selecting only the partitions (Parquet files) that might contain candidate matching data.
• DuckDB is the query engine invoked during search, and it interacts with these Parquet files to execute SQL queries for filtering and ranking.
• The core module files (reteti_core.py, reteti_file.py, reteti_text.py) provide low-level utilities that support both indexing and searching.
• The overall architecture emphasizes decoupling – the indexing process is separate from the search handling process, which fits into a serverless/scale-to-zero pattern.
Important Architectural Patterns / Design Principles
• Separation of Concerns: Clearly separated modules for indexing and search processing.
• Decoupled Storage and Compute: The design enables the compute processes (indexing and searching) to work independently from the storage, facilitating scalability and operational flexibility.
• Partitioned Data Indexing: The use of partitioning via hashed words stored as Parquet files ensures that only relevant data is loaded during search.
• Serverless and Scale-to-Zero Readiness: Configuration for serverless environments (such as in fly.toml and Dockerfile) indicates adaptability to modern cloud environments.
Relevant Technologies, Frameworks, or Libraries
• Python and Tokenizers: For text splitting and normalization.
• DuckDB: For SQL-based search on Parquet files.
• Parquet: As the storage format for partitioned indices.
• Docker, Dockerfile, and Fly.io deployment configuration (fly.toml): For containerization and cloud deployment.
• Object storage: Although abstract in the repository, it is crucial for persisting the index data.