How gaurav16gupta/bliss works
Project type and purpose
• This is an open-source, large-scale vector indexing and retrieval tool (BLISS) combining Python orchestration with a high-performance C++/Cython extension. Its pipelines cover data preparation, iterative partition training (“construct”), index building, and query time retrieval, targeting billion-scale approximate nearest neighbors and extreme classification.
File structure
• Top-level:
• requirements.txt, script.sh, README.md, LICENSE.md
• indices/ (output index files)
• logs/ (runtime logs)
• src/ directory:
• InvertedIndex/ (C++ source, Cython wrappers, setup.py, Makefile, build artifacts)
• config.py (dataset parameters)
• dataPrepare.py (ingest and format raw HDF5 datasets)
• construct.py (train iterative partitioning model)
• train.py (internal subroutine for model learning)
• index.py (build inverted index using trained model)
• query.py (query-time retrieval script)
• net.py (network/distributed communication abstractions)
• utils.py (shared utilities)
• Observations:
• The project uses a layered architecture: Python CLI → Python modules → C++/Cython core library → disk (indices, logs)
• Separate pipelines (data prep, model construct, index build, query)
• Configuration driven via config.py and command-line flags
• Build scripts (Makefile) and setup.py for compiling C++ extensions
README for additional insights
• Architecture is not diagrammed, but the text describes:
• Python orchestration for training, indexing, querying
• A C++ “InvertedIndex” component exposing Gather and scoreAgg routines via Cython
• Data storage in HDF5 files under a user-provided “data” directory
• Support for distributed, model- and data-parallel deployments
Details
• Build step (Makefile / setup.py) that produces the C++ extension under src/InvertedIndex/build
• Optionally, annotate performance characteristics (e.g., “100× less RAM than HNSW”) for context