adithya3907/redrob-candidate-ranker

Purpose and entry points

Redrob Candidate Ranker is an offline-capable, CPU-only pipeline that selects and explains the top 100 candidate profiles for a job description. The production ranking command is rank.py; offline artifact construction starts at scripts/build_index.py. A small-file interactive demonstration is served by sandbox/app.py.

Two-phase execution boundary

Phase A ingests candidates, creates dense embeddings, and persists a LanceDB search index. It is expensive and may require model download/GPU when rebuilt. Phase B is the constrained runtime path: it reads prepared artifacts, ranks candidates offline, and writes WhiteNoise.csv.

• scripts/build_index.py

• rank.py

• artifacts/lancedb/candidates.lance

Ingestion, validation, and feature preparation

Candidate records enter through ingestion and validation modules, then are transformed into structured features used by filters, behavioral scoring, retrieval, and explanations. The raw evaluation-scale source is expected externally as candidates.jsonl; the repository also includes candidates.zip.

• src/ranker/ingest.py

• src/ranker/validate.py

• src/ranker/features.py

Job interpretation and eligibility gates

The pipeline extracts JD-oriented queries and applies deterministic business rules before costly model scoring. These gates reject incompatible backgrounds such as CV-only, academic-only without production evidence, or unvalidated consulting profiles. Structural honeypot detection is a separate integrity boundary: inconsistent cross-field claims are zeroed before ranking.

• src/ranker/jd_queries.py

• src/ranker/gates.py

• src/ranker/filters.py

Retrieval and model-scoring funnel

Prepared candidate embeddings support semantic recall from LanceDB, avoiding keyword-only retrieval. The remaining candidates are reranked, with embedding, index, recall, and rerank concerns split by module. The stated model stack is BAAI/bge-small embeddings precomputed in Phase A and an ms-marco-MiniLM cross-encoder during Phase B.

• src/ranker/embed.py

• src/ranker/index.py

• src/ranker/recall.py

• src/ranker/rerank.py

Score composition and explanations

The orchestrating pipeline combines semantic/reranking signals with behavioral availability penalties, then composes final score/rank output. Grounded reasoning is generated for selected results from candidate evidence rather than free-form profile invention.

• src/ranker/pipeline.py

• src/ranker/behavioral.py

• src/ranker/compose.py

• src/ranker/reasoning.py

Runtime, persistence, and deployment

The application is Python 3.12-based (runtime.txt) and containerized for no-network, CPU-only execution via Dockerfile. LanceDB artifacts, including vector and full-text indexes, are baked/persisted locally rather than accessed as a remote service. Streamlit is only the sandbox UI runtime, configured under .streamlit/config.toml. Weight constants can be calibrated offline with Optuna against the checked-in gold set, without affecting the production entry path.

• Dockerfile

• artifacts/lancedb/candidates.lance

• research/tune_weights.py

• gold/gold_set_optuna_v3.jsonl