How kioxiaamerica/aisaq-diskann works
Project Type and Purpose
• This is a high-performance approximate nearest neighbor search system that uses disk-based and in‐memory indices to perform efficient searches without requiring DRAM. It is a full-stack system that includes performance-critical C++ (and Rust) code, Python bindings for developer convenience, and REST/CLI interfaces for interaction.
• The project is built around a core engine (adapted from DiskANN) that manages index construction, search algorithms (including product quantization), and supports both disk-based and memory-based indices.
File Structure – High-Level Components
• Core C++/CMake Code (src, include)
– Contains the central index algorithms, data structures and utility components.
– Files in “include” (e.g., abstract_index, index_config, pq, restapi) and “src” (e.g., index, logger, distance) are essential parts of the core engine.
• Application Drivers (apps)
– Multiple standalone applications are provided for different tasks: building disk or memory indices, stitching indices, and supporting a REST API.
– Subdirectories under “apps” specifically tailor to different use cases (command line tools, REST API server).
• Python Bindings and Modules (python, apps/python)
– Python directory holds CMake build scripts, the module source code (in python/src), and also a dedicated REST and CLI interface via Python.
– These bindings allow integration with Python applications and provide an extension module interface to the core.
• Rust Drivers (rust)
– A separate portion that features several Rust-based command drivers for building and searching indices. They are organized with their own Cargo.toml and source directories.
• Continuous Integration/Build Infrastructure (.github, workflows, scripts)
– A wide range of GitHub Actions workflows, Dockerfiles, and build scripts facilitate testing, formatting, and performance testing. This reflects a well-integrated CI/CD pipeline.
• Additional Utilities and Tests (tests, scripts, unit testers)
– These directories contain test cases both for performance and functional correctness.
Key Architectural Components
a. Core Index Engine:
– Data abstraction layer (data store interfaces, in-memory vs. disk storage, file readers/writers)
– Index building and updating (index_factory, builder utilities)
– Search Algorithms (distance calculation, graph search, filtering, product quantization modules)
• A “logger” module and “utility” functions (math_utils, timer, partition, etc.).
b. Application Layer:
c. Rust Drivers:
• A separate but connected “Rust Drivers” component that mirrors many capabilities of the C++ drivers. They’re specialized for building and querying indices and might interface directly with the core data structures.
d. Build, Configuration, and CI/CD:
• Build layer (CMake, Dockerfiles, GitHub workflows) as an external infrastructure component that manages compilation, testing, and deployment.
• This layer ties together the various language modules (C++, Python, Rust) and ensures that dependency management, formatting, and performance tests are automated.
e. External Dependencies:
Relationships and Interactions
– Inputs from command-line interfaces and REST clients are processed by the core engine.
– The Python bindings act as a wrapper around the core engine, bridging calls from high-level Python code into the C++ algorithms.
– The Rust drivers similarly interact with lower-level libraries for index construction and search.
Architectural Patterns and Design Principles
• Separation of Concerns: Different layers have clear responsibilities (presentation layer via REST/CLI; computational engine for index building/search; integrations for Python and Rust).
• Modularity: The design naturally breaks down into multiple modules (data access, search algorithms, API wrappers, utility functions) which can be separately developed, tested, and maintained.
• Extensibility: The project structure supports adding new drivers or interfaces since similar patterns exist across C++, Python, and Rust directories.