# adithya3907/redrob-candidate-ranker architecture

> Architecture diagram of the GitHub repository adithya3907/redrob-candidate-ranker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adithya3907/redrob-candidate-ranker
- Repository: https://github.com/adithya3907/redrob-candidate-ranker

## Overview

#### Purpose and entry points
Redrob Candidate Ranker is an offline-capable, CPU-only pipeline that selects and explains the top 100 candidate profiles for a job description. The production ranking command is `rank.py`; offline artifact construction starts at `scripts/build_index.py`. A small-file interactive demonstration is served by `sandbox/app.py`.

#### Two-phase execution boundary
Phase A ingests candidates, creates dense embeddings, and persists a LanceDB search index. It is expensive and may require model download/GPU when rebuilt. Phase B is the constrained runtime path: it reads prepared artifacts, ranks candidates offline, and writes `WhiteNoise.csv`.
- `scripts/build_index.py`
- `rank.py`
- `artifacts/lancedb/candidates.lance`

#### Ingestion, validation, and feature preparation
Candidate records enter through ingestion and validation modules, then are transformed into structured features used by filters, behavioral scoring, retrieval, and explanations. The raw evaluation-scale source is expected externally as `candidates.jsonl`; the repository also includes `candidates.zip`.
- `src/ranker/ingest.py`
- `src/ranker/validate.py`
- `src/ranker/features.py`

#### Job interpretation and eligibility gates
The pipeline extracts JD-oriented queries and applies deterministic business rules before costly model scoring. These gates reject incompatible backgrounds such as CV-only, academic-only without production evidence, or unvalidated consulting profiles. Structural honeypot detection is a separate integrity boundary: inconsistent cross-field claims are zeroed before ranking.
- `src/ranker/jd_queries.py`
- `src/ranker/gates.py`
- `src/ranker/filters.py`

#### Retrieval and model-scoring funnel
Prepared candidate embeddings support semantic recall from LanceDB, avoiding keyword-only retrieval. The remaining candidates are reranked, with embedding, index, recall, and rerank concerns split by module. The stated model stack is BAAI/bge-small embeddings precomputed in Phase A and an `ms-marco-MiniLM` cross-encoder during Phase B.
- `src/ranker/embed.py`
- `src/ranker/index.py`
- `src/ranker/recall.py`
- `src/ranker/rerank.py`

#### Score composition and explanations
The orchestrating pipeline combines semantic/reranking signals with behavioral availability penalties, then composes final score/rank output. Grounded reasoning is generated for selected results from candidate evidence rather than free-form profile invention.
- `src/ranker/pipeline.py`
- `src/ranker/behavioral.py`
- `src/ranker/compose.py`
- `src/ranker/reasoning.py`

#### Runtime, persistence, and deployment
The application is Python 3.12-based (`runtime.txt`) and containerized for no-network, CPU-only execution via `Dockerfile`. LanceDB artifacts, including vector and full-text indexes, are baked/persisted locally rather than accessed as a remote service. Streamlit is only the sandbox UI runtime, configured under `.streamlit/config.toml`. Weight constants can be calibrated offline with Optuna against the checked-in gold set, without affecting the production entry path.
- `Dockerfile`
- `artifacts/lancedb/candidates.lance`
- `research/tune_weights.py`
- `gold/gold_set_optuna_v3.jsonl`

Main entry points: [`src/ranker/compose.py`](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/compose.py) (Compose final ranking), [`src/ranker/filters.py`](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/filters.py) (Honeypot integrity filter), [`src/ranker/recall.py`](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/recall.py) (Semantic recall), [`src/ranker/pipeline.py`](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/pipeline.py) (Ranking pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_build["Phase A · Artifact Build"]
  node_build_index["Build index<br/>build command<br/>[build_index.py]"]
  node_ingest["Ingest candidates<br/>ingestion<br/>[ingest.py]"]
  node_validate["Validate records<br/>validation<br/>[validate.py]"]
  node_features["Prepare features<br/>feature extraction<br/>[features.py]"]
  node_embed["Create embeddings<br/>[embed.py]"]
  node_index["Persist search index<br/>index writer<br/>[index.py]"]
end

subgraph group_runtime["Phase B · Offline Ranking"]
  node_rank_command["Rank command<br/>production entry point<br/>[rank.py]"]
  node_jd_queries["Interpret job description<br/>JD queries<br/>[jd_queries.py]"]
  node_gates["Eligibility gates<br/>business rules<br/>[gates.py]"]
  node_filters["Honeypot integrity filter<br/>claim validation<br/>[filters.py]"]
  node_recall["Semantic recall<br/>vector retrieval<br/>[recall.py]"]
  node_rerank["Cross-encoder rerank<br/>model scoring<br/>[rerank.py]"]
  node_behavioral["Behavioral scoring<br/>availability penalties<br/>[behavioral.py]"]
  node_compose["Compose final ranking<br/>score composition<br/>[compose.py]"]
  node_reasoning["Grounded explanations<br/>evidence reasoning<br/>[reasoning.py]"]
  node_pipeline["Ranking pipeline<br/>orchestrator<br/>[pipeline.py]"]
end

subgraph group_artifacts["Local Data &amp; Artifacts"]
  node_raw_candidates["Candidate source<br/>JSONL input"]
  node_lance_index[("Candidate vector index<br/>LanceDB artifact<br/>[candidates.lance]")]
end

subgraph group_research["Calibration"]
  node_tune_weights["Tune score weights<br/>Optuna calibration<br/>[tune_weights.py]"]
  node_gold_set["Gold set<br/>labeled data"]
end

node_raw_candidates -->|"supplies records"| node_build_index
node_build_index -->|"orchestrates"| node_ingest
node_ingest -->|"records"| node_validate
node_validate -->|"valid records"| node_features
node_features -->|"candidate text"| node_embed
node_embed -->|"vectors"| node_index
node_index -->|"persists"| node_lance_index
node_rank_command -->|"invokes"| node_pipeline
node_pipeline -->|"interprets JD"| node_jd_queries
node_jd_queries -->|"requirements"| node_gates
node_gates -->|"eligible candidates"| node_filters
node_filters -->|"integrity-cleared candidates"| node_recall
node_lance_index -->|"local search"| node_recall
node_recall -->|"semantic matches"| node_rerank
node_rerank -->|"rerank scores"| node_compose
node_filters -->|"candidate features"| node_behavioral
node_behavioral -->|"penalties"| node_compose
node_compose -->|"selected results"| node_reasoning
node_reasoning -->|"ranked explanations"| node_pipeline
node_gold_set -->|"evaluates"| node_tune_weights
node_tune_weights -.->|"calibrates constants"| node_compose

click node_lance_index "https://github.com/adithya3907/redrob-candidate-ranker/tree/main/artifacts/lancedb/candidates.lance"
click node_build_index "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/scripts/build_index.py"
click node_ingest "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/ingest.py"
click node_validate "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/validate.py"
click node_features "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/features.py"
click node_embed "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/embed.py"
click node_index "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/index.py"
click node_rank_command "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/rank.py"
click node_jd_queries "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/jd_queries.py"
click node_gates "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/gates.py"
click node_filters "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/filters.py"
click node_recall "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/recall.py"
click node_rerank "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/rerank.py"
click node_behavioral "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/behavioral.py"
click node_compose "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/compose.py"
click node_reasoning "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/reasoning.py"
click node_pipeline "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/pipeline.py"
click node_tune_weights "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/research/tune_weights.py"
click node_gold_set "https://github.com/adithya3907/redrob-candidate-ranker/blob/main/gold/gold_set_optuna_v3.jsonl"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_build_index,node_ingest,node_validate,node_features,node_embed,node_index toneBlue
class node_rank_command,node_jd_queries,node_gates,node_filters,node_recall,node_rerank,node_behavioral,node_compose,node_reasoning,node_pipeline toneAmber
class node_raw_candidates,node_lance_index toneMint
class node_tune_weights,node_gold_set toneRose
```

## Components

### Phase A · Artifact Build

Expensive offline indexing workflow.

- [Build index](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/scripts/build_index.py): build command · Offline artifact construction entry point. · `scripts/build_index.py`
- [Ingest candidates](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/ingest.py): ingestion · `src/ranker/ingest.py`
- [Validate records](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/validate.py): validation · `src/ranker/validate.py`
- [Prepare features](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/features.py): feature extraction · Produces structured signals for downstream ranking. · `src/ranker/features.py`
- [Create embeddings](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/embed.py): embedding · Precomputes BGE-small candidate vectors. · `src/ranker/embed.py`
- [Persist search index](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/index.py): index writer · `src/ranker/index.py`

### Phase B · Offline Ranking

CPU-only production ranking path.

- [Rank command](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/rank.py): production entry point · Runs ranking against prepared local artifacts. · `rank.py`
- [Interpret job description](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/jd_queries.py): JD queries · `src/ranker/jd_queries.py`
- [Eligibility gates](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/gates.py): business rules · Rejects incompatible or insufficient backgrounds early. · `src/ranker/gates.py`
- [Honeypot integrity filter](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/filters.py): claim validation · Zeroes inconsistent cross-field claims before ranking. · `src/ranker/filters.py`
- [Semantic recall](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/recall.py): vector retrieval · Retrieves candidates from the local LanceDB index. · `src/ranker/recall.py`
- [Cross-encoder rerank](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/rerank.py): model scoring · Uses ms-marco-MiniLM on recalled candidates. · `src/ranker/rerank.py`
- [Behavioral scoring](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/behavioral.py): availability penalties · `src/ranker/behavioral.py`
- [Compose final ranking](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/compose.py): score composition · Combines model and behavioral signals into final rank. · `src/ranker/compose.py`
- [Grounded explanations](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/reasoning.py): evidence reasoning · Explains selected results from candidate evidence. · `src/ranker/reasoning.py`
- [Ranking pipeline](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/src/ranker/pipeline.py): orchestrator · Writes the top-100 ranked output. · `src/ranker/pipeline.py`

### Local Data & Artifacts

Prepared inputs and persisted search index.

- **Candidate source**: JSONL input · External evaluation-scale candidate records.
- [Candidate vector index](https://github.com/adithya3907/redrob-candidate-ranker/tree/main/artifacts/lancedb/candidates.lance): LanceDB artifact · Persisted vector and full-text search indexes. · `artifacts/lancedb/candidates.lance`

### Calibration

Offline score-weight tuning outside production.

- [Tune score weights](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/research/tune_weights.py): Optuna calibration · Offline calibration that does not affect the production path. · `research/tune_weights.py`
- [Gold set](https://github.com/adithya3907/redrob-candidate-ranker/blob/main/gold/gold_set_optuna_v3.jsonl): labeled data · `gold/gold_set_optuna_v3.jsonl`

## Connections

- Candidate source → Build index: supplies records
- Build index → Ingest candidates: orchestrates
- Ingest candidates → Validate records: records
- Validate records → Prepare features: valid records
- Prepare features → Create embeddings: candidate text
- Create embeddings → Persist search index: vectors
- Persist search index → Candidate vector index: persists
- Rank command → Ranking pipeline: invokes
- Ranking pipeline → Interpret job description: interprets JD
- Interpret job description → Eligibility gates: requirements
- Eligibility gates → Honeypot integrity filter: eligible candidates
- Honeypot integrity filter → Semantic recall: integrity-cleared candidates
- Candidate vector index → Semantic recall: local search
- Semantic recall → Cross-encoder rerank: semantic matches
- Cross-encoder rerank → Compose final ranking: rerank scores
- Honeypot integrity filter → Behavioral scoring: candidate features
- Behavioral scoring → Compose final ranking: penalties
- Compose final ranking → Grounded explanations: selected results
- Grounded explanations → Ranking pipeline: ranked explanations
- Gold set → Tune score weights: evaluates
- Tune score weights → Compose final ranking: calibrates constants · Offline-only weight calibration.