How duty00/job-scam-dectection works
This repository is a reproducible NLP project for screening English job postings for scam signals. Its main workflow downloads and audits EMSCAD, removes duplicates and groups related postings before creating leakage-resistant splits, then compares traditional text models and GloVe features, selects a model and threshold on validation, and evaluates the locked test set. An optional TextCNN benchmark and an explanatory prediction demo extend that workflow. Source excerpts show the main module dependencies; some stage transitions are documented only in the README.
Main entry points: experiment.py Experiment workflow · data.py Data audit and splits · run_deep_learning.py TextCNN CLI · deep_learning.py TextCNN training
Components 18
Data preparation
- Run configuration
modules/config.py - Data audit and splits
modules/data.py - Exploratory analysis
modules/eda.py
Model experiments
- Experiment CLI
run_experiments.py - Experiment workflow
modules/experiment.py - BoW and TF-IDF
modules/experiment.py - GloVe pooling
modules/embeddings.py - Traditional candidates
modules/experiment.py - Thresholds and metrics
modules/evaluation.py - Experiment artifacts
results
Deep learning
- TextCNN CLI
run_deep_learning.py - TextCNN training
modules/deep_learning.py - TextCNN artifacts
results
Prediction demo
- Explanation demo
modules/inference.py - Fitted demo model
Other components
- Project user
- EMSCAD postings
- Pretrained GloVe
Connections 22
- Project user → Experiment CLI: runs experiments
README.md - Experiment CLI → Experiment workflow: starts workflow
run_experiments.py - Experiment workflow → Run configuration: uses settings
modules/experiment.py - Experiment workflow → Data audit and splits: loads and prepares
modules/experiment.py - Data audit and splits → EMSCAD postings: downloads data
modules/data.py - Experiment workflow → Exploratory analysis: creates analysis
modules/experiment.py - Data audit and splits → BoW and TF-IDF: provides split text
README.md - Experiment workflow → BoW and TF-IDF: extracts features
README.md - Experiment workflow → GloVe pooling: builds representations
README.md - GloVe pooling → Pretrained GloVe: loads vectors
modules/embeddings.py - BoW and TF-IDF → Traditional candidates: feeds candidates
README.md - Traditional candidates → Thresholds and metrics: selects on validation
README.md - Thresholds and metrics → Experiment artifacts: records scores
- Experiment workflow → Experiment artifacts: saves selection and models
modules/experiment.py - Project user → TextCNN CLI: runs benchmark
README.md - TextCNN CLI → Experiment workflow: prepares shared data
run_deep_learning.py - TextCNN CLI → TextCNN training: runs TextCNN
run_deep_learning.py - TextCNN training → Thresholds and metrics: selects and scores
modules/deep_learning.py - TextCNN training → TextCNN artifacts: writes metrics
modules/deep_learning.py - Project user → Explanation demo: submits posting
README.md - Explanation demo → Data audit and splits: cleans text
modules/inference.py - Explanation demo → Fitted demo model: loads fitted bundle
modules/inference.py