# aditya-singh2005/redrob-hackathon-ai-recruitment-engine architecture

> Architecture diagram of the GitHub repository aditya-singh2005/redrob-hackathon-ai-recruitment-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine
- Repository: https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine

## Overview

1. **Overall shape**
   - This repo is a two-stage candidate-ranking pipeline, not an app server or model-training project.
   - The architecture is split between **offline precomputation** and **fast final ranking**, with disk artifacts connecting the stages.

2. **Input boundary**
   - The primary input is a **candidate pool in JSONL format** under `data/` and a **job description requirements artifact**.
   - The repo also includes `data/sample_candidates_100.jsonl` for small-scale testing and `data/validate_submission.py` for checking output format.

3. **Stage 1: JD parsing**
   - `scripts/01_parse_jd.py` converts the raw job description into a structured requirements file.
   - Its output is `artifacts/jd_requirements.json`, which becomes the canonical JD contract for downstream scoring.
   - This stage is an offline normalization step: it turns free text into weighted requirements and constraints.

4. **Stage 2: feature generation and embedding**
   - `scripts/02_feature_gen.py` is the heavy offline preprocessing stage.
   - It reads the full candidate pool, computes candidate embeddings with `sentence-transformers` (`all-MiniLM-L6-v2`), and derives precomputed features for each profile.
   - Outputs are persisted as `artifacts/features.parquet`, `artifacts/embeddings.npy`, and `artifacts/jd_embedding.npy`.

5. **Precomputed artifact layer**
   - The `artifacts/` directory is the main bridge between expensive analysis and runtime ranking.
   - This boundary is important: everything costly, large-scale, or GPU-friendly happens once here; the final ranking script only reads these artifacts.
   - The design is optimized for a constrained compute budget in the final stage.

6. **Stage 3: ranking and submission generation**
   - `scripts/03_rank.py` is the execution-time ranking engine.
   - It loads the precomputed embeddings/features, computes semantic similarity by matrix operation, combines all scoring dimensions, and writes the ranked output CSV.
   - This is the part intended to run under the contest’s CPU/time limits.

7. **Scoring architecture**
   - The score is a weighted composite of semantic similarity, career quality, skill match, location fit, and education, then modified by behavioral and disqualifier factors.
   - Architecturally, this means there are two kinds of signals:
     - **Additive fit signals** that measure relevance
     - **Multiplicative gate signals** that globally boost or suppress candidates
   - This separation is central to the system’s ranking behavior.

8. **Data flow through the pipeline**
   - Raw JD text → structured JD JSON
   - Candidate JSONL → embeddings + feature table
   - JD JSON + candidate artifacts → final ranked CSV
   - The flow is one-directional and file-based, which makes the system reproducible and easy to split across compute environments.

9. **Runtime and compute split**
   - The repo explicitly targets **GPU-friendly offline processing** and **CPU-only fast inference/ranking**.
   - Sentence-transformer embedding generation is the expensive part; ranking is designed to be a lightweight read-and-score pass.
   - This is an important architectural constraint, since the final ranking step must fit a short wall-clock budget.

10. **Supporting tooling and project boundaries**
   - `requirements.txt` defines a Python-based ML/data stack centered on `numpy`, `pandas`, `pyarrow`, `scikit-learn`, `tqdm`, and `sentence-transformers`.
   - `submission_metadata.yaml` likely captures contest submission metadata rather than runtime logic.
   - `sandbox_demo.ipynb` serves as an interactive demo path for small-sample validation, separate from the production ranking pipeline.

11. **Validation and reproducibility**
   - `data/validate_submission.py` indicates there is a schema/format check boundary for the output CSV.
   - The repo is structured to support reproducibility: the same precomputed artifacts should yield the same ranked output, assuming the inputs and model version are fixed.

12. **What an architecture diagram should emphasize**
   - Show three major layers: **JD parsing**, **offline feature/embedding generation**, and **online ranking/output**.
   - Draw the artifact store (`artifacts/`) as the interface between offline and online stages.
   - Include the candidate dataset and the submission validator as external boundaries around the core ranking pipeline.

Main entry points: [`scripts/02_feature_gen.py`](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/02_feature_gen.py) (Feature gen), [`artifacts`](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/tree/main/artifacts) (Artifacts), [`scripts/03_rank.py`](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/03_rank.py) (Ranker), [`README.md`](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/README.md) (README)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Inputs & validation"]
  node_demo["Demo<br/>notebook<br/>[sandbox_demo.ipynb]"]
  node_sample_candidates["Sample candidates<br/>candidate jsonl"]
  node_validator["Validator<br/>output check"]
end

subgraph group_offline["Offline preprocessing"]
  node_jd_parser["JD parser<br/>normalizer<br/>[01_parse_jd.py]"]
  node_jd_requirements["JD requirements<br/>jd contract"]
  node_feature_gen["Feature gen<br/>offline featurizer<br/>[02_feature_gen.py]"]
  node_artifact_store[("Artifacts<br/>precomputed store")]
  node_features["Features<br/>feature table"]
  node_embeddings["Embeddings<br/>candidate vectors"]
  node_jd_embedding["JD vector<br/>query vector"]
end

subgraph group_ranking["Ranking & output"]
  node_ranker["Ranker<br/>scoring engine<br/>[03_rank.py]"]
  node_output["Ranked CSV<br/>submission output"]
end

node_readme["README<br/>guide<br/>[README.md]"]
node_requirements["Deps<br/>python stack<br/>[requirements.txt]"]
node_metadata["Metadata<br/>submission meta"]

node_readme -->|"documents"| node_jd_parser
node_readme -->|"documents"| node_feature_gen
node_readme -->|"documents"| node_ranker
node_sample_candidates -->|"input"| node_feature_gen
node_validator -.->|"checks"| node_output
node_jd_parser -->|"writes"| node_jd_requirements
node_jd_requirements -->|"stored in"| node_artifact_store
node_feature_gen -->|"writes"| node_features
node_feature_gen -->|"writes"| node_embeddings
node_feature_gen -->|"writes"| node_jd_embedding
node_features -->|"stored in"| node_artifact_store
node_embeddings -->|"stored in"| node_artifact_store
node_jd_embedding -->|"stored in"| node_artifact_store
node_artifact_store -->|"loads"| node_ranker
node_jd_requirements -->|"conditions"| node_ranker
node_ranker -->|"writes"| node_output
node_requirements -.->|"supports"| node_jd_parser
node_requirements -.->|"supports"| node_feature_gen
node_requirements -.->|"supports"| node_ranker
node_demo -.->|"uses"| node_sample_candidates

click node_readme "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/README.md"
click node_requirements "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/requirements.txt"
click node_metadata "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/submission_metadata.yaml"
click node_demo "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/sandbox_demo.ipynb"
click node_sample_candidates "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/data/sample_candidates_100.jsonl"
click node_validator "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/data/validate_submission.py"
click node_jd_parser "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/01_parse_jd.py"
click node_jd_requirements "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/artifacts/jd_requirements.json"
click node_feature_gen "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/02_feature_gen.py"
click node_artifact_store "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/tree/main/artifacts"
click node_ranker "https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/03_rank.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo,node_sample_candidates,node_validator toneBlue
class node_jd_parser,node_jd_requirements,node_feature_gen,node_artifact_store,node_features,node_embeddings,node_jd_embedding toneAmber
class node_ranker,node_output toneMint
class node_readme,node_requirements,node_metadata toneNeutral
```

## Components

### Inputs & validation

Candidate data, submission checks, and demo boundary.

- [Demo](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/sandbox_demo.ipynb): notebook · Interactive sandbox for small-scale experimentation. · `sandbox_demo.ipynb`
- [Sample candidates](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/data/sample_candidates_100.jsonl): candidate jsonl · Small candidate pool used for testing and demo workflows. · `data/sample_candidates_100.jsonl`
- [Validator](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/data/validate_submission.py): output check · Submission format and schema validation boundary. · `data/validate_submission.py`

### Offline preprocessing

Job description parsing plus heavy feature and embedding generation.

- [JD parser](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/01_parse_jd.py): normalizer · Converts raw job description text into structured requirements. · `scripts/01_parse_jd.py`
- [JD requirements](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/artifacts/jd_requirements.json): jd contract · Canonical weighted requirements and constraints for downstream scoring. · `artifacts/jd_requirements.json`
- [Feature gen](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/02_feature_gen.py): offline featurizer · Builds candidate features and embeddings from the full pool. · `scripts/02_feature_gen.py`
- [Artifacts](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/tree/main/artifacts): precomputed store · Disk boundary separating expensive offline work from fast ranking. · `artifacts`
- **Features**: feature table · Precomputed candidate feature set used by the ranking stage.
- **Embeddings**: candidate vectors · Precomputed candidate embedding matrix for semantic similarity.
- **JD vector**: query vector · Embedded representation of the parsed job requirements.

### Ranking & output

Fast final scoring, ranking, and submission generation.

- [Ranker](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/scripts/03_rank.py): scoring engine · Combines similarity, fit signals, and gates to produce final scores. · `scripts/03_rank.py`
- **Ranked CSV**: submission output · Final ranked candidate list written for contest submission.

### Other components

- [README](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/README.md): guide · Top-level repository overview and usage guidance. · `README.md`
- [Deps](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/requirements.txt): python stack · Core ML and data dependencies used across the pipeline. · `requirements.txt`
- [Metadata](https://github.com/aditya-singh2005/redrob-hackathon-ai-recruitment-engine/blob/main/submission_metadata.yaml): submission meta · Contest or packaging metadata separate from runtime logic. · `submission_metadata.yaml`

## Connections

- README → JD parser: documents · Explains the JD parsing stage and pipeline intent.
- README → Feature gen: documents · Explains offline preprocessing and artifact generation.
- README → Ranker: documents · Explains final ranking workflow and compute split.
- Sample candidates → Feature gen: input · Small candidate pool path for testing the preprocessing stage.
- Validator → Ranked CSV: checks · Validates the generated submission format.
- JD parser → JD requirements: writes · Normalizes raw job description into structured requirements.
- JD requirements → Artifacts: stored in · Persists canonical JD contract for downstream use.
- Feature gen → Features: writes · Produces candidate feature table.
- Feature gen → Embeddings: writes · Produces candidate embedding matrix.
- Feature gen → JD vector: writes · Produces JD embedding used by final ranking.
- Features → Artifacts: stored in · Precomputed feature table lives in the artifact boundary.
- Embeddings → Artifacts: stored in · Candidate vectors live in the artifact boundary.
- JD vector → Artifacts: stored in · Query embedding lives in the artifact boundary.
- Artifacts → Ranker: loads · Final ranking stage reads precomputed artifacts only.
- JD requirements → Ranker: conditions · Structured JD constraints guide scoring and filtering.
- Ranker → Ranked CSV: writes · Produces ranked submission CSV.
- Deps → JD parser: supports · Shared Python data and ML dependencies.
- Deps → Feature gen: supports · Shared Python data and ML dependencies.
- Deps → Ranker: supports · Shared Python data and ML dependencies.
- Demo → Sample candidates: uses · Notebook demonstrates small-sample workflows.
