How aditya-singh2005/redrob-hackathon-ai-recruitment-engine works
1. Overall shape
- This repo is a two-stage candidate-ranking pipeline, not an app server or model-training project.
- The architecture is split between offline precomputation and fast final ranking, with disk artifacts connecting the stages.
2. Input boundary
- The primary input is a candidate pool in JSONL format under data/ and a job description requirements artifact.
- The repo also includes data/sample_candidates_100.jsonl for small-scale testing and data/validate_submission.py for checking output format.
3. Stage 1: JD parsing
- scripts/01_parse_jd.py converts the raw job description into a structured requirements file.
- Its output is artifacts/jd_requirements.json, which becomes the canonical JD contract for downstream scoring.
- This stage is an offline normalization step: it turns free text into weighted requirements and constraints.
4. Stage 2: feature generation and embedding
- scripts/02_feature_gen.py is the heavy offline preprocessing stage.
- It reads the full candidate pool, computes candidate embeddings with sentence-transformers (all-MiniLM-L6-v2), and derives precomputed features for each profile.
- Outputs are persisted as artifacts/features.parquet, artifacts/embeddings.npy, and artifacts/jd_embedding.npy.
5. Precomputed artifact layer
- The artifacts/ directory is the main bridge between expensive analysis and runtime ranking.
- This boundary is important: everything costly, large-scale, or GPU-friendly happens once here; the final ranking script only reads these artifacts.
- The design is optimized for a constrained compute budget in the final stage.
6. Stage 3: ranking and submission generation
- scripts/03_rank.py is the execution-time ranking engine.
- It loads the precomputed embeddings/features, computes semantic similarity by matrix operation, combines all scoring dimensions, and writes the ranked output CSV.
- This is the part intended to run under the contest’s CPU/time limits.
7. Scoring architecture
- The score is a weighted composite of semantic similarity, career quality, skill match, location fit, and education, then modified by behavioral and disqualifier factors.
- Architecturally, this means there are two kinds of signals:
- Additive fit signals that measure relevance
- Multiplicative gate signals that globally boost or suppress candidates
- This separation is central to the system’s ranking behavior.
8. Data flow through the pipeline
- Raw JD text → structured JD JSON
- Candidate JSONL → embeddings + feature table
- JD JSON + candidate artifacts → final ranked CSV
- The flow is one-directional and file-based, which makes the system reproducible and easy to split across compute environments.
9. Runtime and compute split
- The repo explicitly targets GPU-friendly offline processing and CPU-only fast inference/ranking.
- Sentence-transformer embedding generation is the expensive part; ranking is designed to be a lightweight read-and-score pass.
- This is an important architectural constraint, since the final ranking step must fit a short wall-clock budget.
10. Supporting tooling and project boundaries
- requirements.txt defines a Python-based ML/data stack centered on numpy, pandas, pyarrow, scikit-learn, tqdm, and sentence-transformers.
- submission_metadata.yaml likely captures contest submission metadata rather than runtime logic.
- sandbox_demo.ipynb serves as an interactive demo path for small-sample validation, separate from the production ranking pipeline.
11. Validation and reproducibility
- data/validate_submission.py indicates there is a schema/format check boundary for the output CSV.
- The repo is structured to support reproducibility: the same precomputed artifacts should yield the same ranked output, assuming the inputs and model version are fixed.
12. What an architecture diagram should emphasize
- Show three major layers: JD parsing, offline feature/embedding generation, and online ranking/output.
- Draw the artifact store (artifacts/) as the interface between offline and online stages.
- Include the candidate dataset and the submission validator as external boundaries around the core ranking pipeline.
Main entry points: 02_feature_gen.py Feature gen · artifacts Artifacts · 03_rank.py Ranker · README.md README
Components 15
Inputs & validation
Candidate data, submission checks, and demo boundary.
- Demo · Interactive sandbox for small-scale experimentation.
sandbox_demo.ipynb - Sample candidates · Small candidate pool used for testing and demo workflows.
data/sample_candidates_100.jsonl - Validator · Submission format and schema validation boundary.
data/validate_submission.py
Offline preprocessing
Job description parsing plus heavy feature and embedding generation.
- JD parser · Converts raw job description text into structured requirements.
scripts/01_parse_jd.py - JD requirements · Canonical weighted requirements and constraints for downstream scoring.
artifacts/jd_requirements.json - Feature gen · Builds candidate features and embeddings from the full pool.
scripts/02_feature_gen.py - Artifacts · Disk boundary separating expensive offline work from fast ranking.
artifacts - Features · Precomputed candidate feature set used by the ranking stage.
- Embeddings · Precomputed candidate embedding matrix for semantic similarity.
- JD vector · Embedded representation of the parsed job requirements.
Ranking & output
Fast final scoring, ranking, and submission generation.
- Ranker · Combines similarity, fit signals, and gates to produce final scores.
scripts/03_rank.py - Ranked CSV · Final ranked candidate list written for contest submission.
Connections 20
- README → JD parser: documents · Explains the JD parsing stage and pipeline intent.
- README → Feature gen: documents · Explains offline preprocessing and artifact generation.
- README → Ranker: documents · Explains final ranking workflow and compute split.
- Sample candidates → Feature gen: input · Small candidate pool path for testing the preprocessing stage.
- Validator → Ranked CSV: checks · Validates the generated submission format.
- JD parser → JD requirements: writes · Normalizes raw job description into structured requirements.
- JD requirements → Artifacts: stored in · Persists canonical JD contract for downstream use.
- Feature gen → Features: writes · Produces candidate feature table.
- Feature gen → Embeddings: writes · Produces candidate embedding matrix.
- Feature gen → JD vector: writes · Produces JD embedding used by final ranking.
- Features → Artifacts: stored in · Precomputed feature table lives in the artifact boundary.
- Embeddings → Artifacts: stored in · Candidate vectors live in the artifact boundary.
- JD vector → Artifacts: stored in · Query embedding lives in the artifact boundary.
- Artifacts → Ranker: loads · Final ranking stage reads precomputed artifacts only.
- JD requirements → Ranker: conditions · Structured JD constraints guide scoring and filtering.
- Ranker → Ranked CSV: writes · Produces ranked submission CSV.
- Deps → JD parser: supports · Shared Python data and ML dependencies.
- Deps → Feature gen: supports · Shared Python data and ML dependencies.
- Deps → Ranker: supports · Shared Python data and ML dependencies.
- Demo → Sample candidates: uses · Notebook demonstrates small-sample workflows.