# gaurav16gupta/bliss architecture

> Architecture diagram of the GitHub repository gaurav16gupta/bliss, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gaurav16gupta/bliss
- Repository: https://github.com/gaurav16gupta/bliss

## Overview

##### Project type and purpose
- This is an open-source, large-scale vector indexing and retrieval tool (BLISS) combining Python orchestration with a high-performance C++/Cython extension. Its pipelines cover data preparation, iterative partition training (“construct”), index building, and query time retrieval, targeting billion-scale approximate nearest neighbors and extreme classification.

##### File structure
- Top-level:
- requirements.txt, script.sh, README.md, LICENSE.md
- indices/ (output index files)
- logs/ (runtime logs)
- src/ directory:
- InvertedIndex/ (C++ source, Cython wrappers, setup.py, Makefile, build artifacts)
- config.py (dataset parameters)
- dataPrepare.py (ingest and format raw HDF5 datasets)
- construct.py (train iterative partitioning model)
- train.py (internal subroutine for model learning)
- index.py (build inverted index using trained model)
- query.py (query-time retrieval script)
- net.py (network/distributed communication abstractions)
- utils.py (shared utilities)
- Observations:
- The project uses a layered architecture: Python CLI → Python modules → C++/Cython core library → disk (indices, logs)
- Separate pipelines (data prep, model construct, index build, query)
- Configuration driven via config.py and command-line flags
- Build scripts (Makefile) and setup.py for compiling C++ extensions

##### README for additional insights
- Architecture is not diagrammed, but the text describes:
- Python orchestration for training, indexing, querying
- A C++ “InvertedIndex” component exposing Gather and scoreAgg routines via Cython
- Data storage in HDF5 files under a user-provided “data” directory
- Support for distributed, model- and data-parallel deployments

##### Details
- Build step (Makefile / setup.py) that produces the C++ extension under src/InvertedIndex/build
- Optionally, annotate performance characteristics (e.g., “100× less RAM than HNSW”) for context

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        Construct["construct.py"]:::pythonComp
        Index["index.py"]:::pythonComp
        Query["query.py"]:::pythonComp
    end

    %% Orchestration & Control
    subgraph "Orchestration & Control"
        direction TB
        Config["config.py"]:::pythonComp
        DataPrep["dataPrepare.py"]:::pythonComp
        Train["train.py"]:::pythonComp
        Utils["utils.py"]:::pythonComp
    end

    %% Core Indexing Library
    subgraph "Core Indexing Library"
        direction TB
        GatherCPP["Gather.cpp / Gather.h"]:::cppComp
        ScoreAggCPP["scoreAgg.cpp"]:::cppComp
        ScoreAggPyx["scoreAgg.pyx"]:::cppComp
        Makefile["Makefile"]:::cppComp
        Setup["setup.py"]:::cppComp
        Build["build/ (artifacts)"]:::cppComp
    end

    %% Network / Distributed Layer
    subgraph "Distributed Layer" 
        direction TB
        Net["net.py"]:::netComp
    end

    %% Data Stores
    subgraph "Disk-Based Data Stores"
        direction TB
        RawData["HDF5 raw data"]:::dataStore
        ModelFiles["Model files (indices/)"]:::dataStore
        IndexFiles["Inverted index files (indices/)"]:::dataStore
        Logs["Logs (logs/)"]:::dataStore
    end

    %% Connections: CLI to Orchestration
    Construct -->|"load config & call modules"| Config
    Index -->|"load config & call modules"| Config
    Query -->|"load config & call modules"| Config

    Config --> DataPrep
    Config --> Train
    Config --> Utils

    %% Orchestration to Core Library
    DataPrep -->|"vectors → Gather"| GatherCPP
    Train -->|"partitions & scores → scoreAgg"| ScoreAggCPP
    Train --> ScoreAggPyx
    ScoreAggPyx --> ScoreAggCPP

    %% Build step
    Makefile --> Setup
    Setup --> Build

    %% Core Library to Data Stores
    Train -->|"write model files"| ModelFiles
    Index -->|"feed data & write posting lists"| IndexFiles
    Query -->|"load index & retrieve results"| IndexFiles

    %% CLI and Orchestration to Logs
    Construct -->|"log runtime"| Logs
    Index -->|"log runtime"| Logs
    Query -->|"log runtime"| Logs

    %% Data sources
    RawData --> DataPrep

    %% Distributed layer connections
    Net --- Construct
    Net --- Index
    Net --- Query
    Net --> Train

    %% Click Events
    click Construct "https://github.com/gaurav16gupta/bliss/blob/main/src/construct.py"
    click Index "https://github.com/gaurav16gupta/bliss/blob/main/src/index.py"
    click Query "https://github.com/gaurav16gupta/bliss/blob/main/src/query.py"
    click Config "https://github.com/gaurav16gupta/bliss/blob/main/src/config.py"
    click DataPrep "https://github.com/gaurav16gupta/bliss/blob/main/src/dataPrepare.py"
    click Train "https://github.com/gaurav16gupta/bliss/blob/main/src/train.py"
    click Utils "https://github.com/gaurav16gupta/bliss/blob/main/src/utils.py"
    click GatherCPP "https://github.com/gaurav16gupta/bliss/blob/main/src/InvertedIndex/Gather.cpp"
    click ScoreAggCPP "https://github.com/gaurav16gupta/bliss/blob/main/src/InvertedIndex/scoreAgg.cpp"
    click ScoreAggPyx "https://github.com/gaurav16gupta/bliss/blob/main/src/InvertedIndex/scoreAgg.pyx"
    click Makefile "https://github.com/gaurav16gupta/bliss/tree/main/src/InvertedIndex/Makefile"
    click Setup "https://github.com/gaurav16gupta/bliss/blob/main/src/InvertedIndex/setup.py"
    click Build "https://github.com/gaurav16gupta/bliss/tree/main/src/InvertedIndex/build/"
    click Net "https://github.com/gaurav16gupta/bliss/blob/main/src/net.py"

    %% Styles
    classDef pythonComp fill:#AED6F1,stroke:#21618C,color:#1B2631;
    classDef cppComp fill:#A9DFBF,stroke:#196F3D,color:#1B2631;
    classDef dataStore fill:#D5D8DC,stroke:#566573,color:#1B2631,stroke-dasharray: 5 5;
    classDef netComp fill:#F9E79F,stroke:#B7950B,color:#1B2631,stroke-dasharray: 2 2;
```
