How castorini/rank_llm works
Project Type and Purpose
• This is a Python-based ranking toolkit (“rank_llm”) that provides:
• Retrieval of candidate documents (via Pyserini or external services)
• Reranking using large language models (pairwise, pointwise, listwise)
• Evaluation (TREC-style metrics)
• Cost and response analysis
• A lightweight API server for programmatic access
• Demo scripts and training utilities
Top-Level Components
• Core Library (src/rank_llm/…):
• retrieve/ – abstractions for retrieving candidate docs (Pyserini, service APIs)
• rerank/ – implementations of reranking strategies (identity, pointwise, pairwise, listwise)
• evaluation/ – TREC evaluation wrappers
• analysis/ – cost estimation & response analysis
• api/server.py – RESTful HTTP server exposing retrieval+rering endpoints
• data.py – data-model definitions (topics, indices)
• scripts/ – CLI entrypoints for common workflows
• Demos (src/rank_llm/demo/…): jupyter-style scripts showcasing various LLM rerankers
• Training (training/…):
• configs/ – Deepspeed & ZeRO3 configs
• scripts/ – shell scripts to launch training on different LLMs
• utils/ – data, model, training helper functions
• Tests (test/…): unit tests for each major submodule
• External Dependencies:
• Pyserini index
• OpenAI, Gemini, Qwen, etc. LLM APIs
• TREC datasets