# kioxiaamerica/aisaq-diskann architecture

> Architecture diagram of the GitHub repository kioxiaamerica/aisaq-diskann, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kioxiaamerica/aisaq-diskann
- Repository: https://github.com/kioxiaamerica/aisaq-diskann

## Overview

##### Project Type and Purpose
- This is a high-performance approximate nearest neighbor search system that uses disk-based and in‐memory indices to perform efficient searches without requiring DRAM. It is a full-stack system that includes performance-critical C++ (and Rust) code, Python bindings for developer convenience, and REST/CLI interfaces for interaction.
- The project is built around a core engine (adapted from DiskANN) that manages index construction, search algorithms (including product quantization), and supports both disk-based and memory-based indices.

##### File Structure – High-Level Components
- Core C++/CMake Code (src, include)
– Contains the central index algorithms, data structures and utility components.
– Files in “include” (e.g., abstract_index, index_config, pq, restapi) and “src” (e.g., index, logger, distance) are essential parts of the core engine.
- Application Drivers (apps)
– Multiple standalone applications are provided for different tasks: building disk or memory indices, stitching indices, and supporting a REST API.
– Subdirectories under “apps” specifically tailor to different use cases (command line tools, REST API server).
- Python Bindings and Modules (python, apps/python)
– Python directory holds CMake build scripts, the module source code (in python/src), and also a dedicated REST and CLI interface via Python.
– These bindings allow integration with Python applications and provide an extension module interface to the core.
- Rust Drivers (rust)
– A separate portion that features several Rust-based command drivers for building and searching indices. They are organized with their own Cargo.toml and source directories.
- Continuous Integration/Build Infrastructure (.github, workflows, scripts)
– A wide range of GitHub Actions workflows, Dockerfiles, and build scripts facilitate testing, formatting, and performance testing. This reflects a well-integrated CI/CD pipeline.
- Additional Utilities and Tests (tests, scripts, unit testers)
– These directories contain test cases both for performance and functional correctness.

##### Key Architectural Components
a. Core Index Engine:
– Data abstraction layer (data store interfaces, in-memory vs. disk storage, file readers/writers)
– Index building and updating (index_factory, builder utilities)
– Search Algorithms (distance calculation, graph search, filtering, product quantization modules)
- A “logger” module and “utility” functions (math_utils, timer, partition, etc.).

b. Application Layer:

c. Rust Drivers:
- A separate but connected “Rust Drivers” component that mirrors many capabilities of the C++ drivers. They’re specialized for building and querying indices and might interface directly with the core data structures.

d. Build, Configuration, and CI/CD:
- Build layer (CMake, Dockerfiles, GitHub workflows) as an external infrastructure component that manages compilation, testing, and deployment.
- This layer ties together the various language modules (C++, Python, Rust) and ensures that dependency management, formatting, and performance tests are automated.

e. External Dependencies:

##### Relationships and Interactions
– Inputs from command-line interfaces and REST clients are processed by the core engine.
– The Python bindings act as a wrapper around the core engine, bridging calls from high-level Python code into the C++ algorithms.
– The Rust drivers similarly interact with lower-level libraries for index construction and search.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Different layers have clear responsibilities (presentation layer via REST/CLI; computational engine for index building/search; integrations for Python and Rust).
- Modularity: The design naturally breaks down into multiple modules (data access, search algorithms, API wrappers, utility functions) which can be separately developed, tested, and maintained.
- Extensibility: The project structure supports adding new drivers or interfaces since similar patterns exist across C++, Python, and Rust directories.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Backend Layer
    subgraph "Backend Layer"
        CS["Core Source (C++)"]:::backend
        CH["Core Headers (C++)"]:::backend
        RD["Rust Drivers"]:::backend
    end

    %% Frontend Layer
    subgraph "Frontend Layer"
        CLI["CLI Tools (C++)"]:::frontend
        REST["REST API Server (C++)"]:::frontend
        PB["Python Bindings"]:::frontend
        PI["Python Interfaces"]:::frontend
    end

    %% Build & Test Layer
    subgraph "Build & Test Layer"
        CMR["CMake Config (Root)"]:::build
        CICD["CI/CD Workflows"]:::build
        PP["Python Project"]:::build
        UT["Utilities Tools"]:::build
        TEST["Utilities & Tests"]:::build
    end

    %% External Dependencies
    EX["External Libraries<br/>(Intel MKL, Boost, libaio, clang-format)"]:::external

    %% Connections between Frontend and Backend
    CLI -->|"invokes"| CS
    REST -->|"invokes"| CS
    PB -->|"wraps"| CS
    PI -->|"calls"| PB
    RD -->|"integrates with"| CS

    %% Build & Test dependencies into Backend and Frontend
    CMR -->|"builds"| CS
    CICD -->|"tests/builds"| CS
    CICD -->|"tests/builds"| CLI
    CICD -->|"tests/builds"| RD
    PP -->|"configures"| PB
    UT -->|"aids"| CLI
    TEST -->|"verifies"| CS

    %% External Dependencies connection
    EX ---|"provides libs"| CS
    EX ---|"supports"| RD

    %% Class Definitions
    classDef backend fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef frontend fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef build fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;

    %% Click Events
    click CS "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/src/"
    click CH "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/include/"
    click CLI "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/apps/"
    click REST "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/apps/restapi/"
    click PB "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/python/src/"
    click PI "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/apps/python/"
    click RD "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/rust/cmd_drivers/"
    click CMR "https://github.com/kioxiaamerica/aisaq-diskann/blob/aisaq_release/CMakeLists.txt"
    click CICD "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/.github/"
    click PP "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/python/"
    click UT "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/apps/utils/"
    click TEST "https://github.com/kioxiaamerica/aisaq-diskann/tree/aisaq_release/tests/"
```
