How dbabbitt/job-hunting works
Project type and purpose
• This is a full-stack data-processing and ML pipeline with a web-based inference and visualization layer.
• Purpose: scrape job postings (Indeed, Glassdoor, etc.), extract and tokenize minimum-requirements sections, train classification models (CRF, LR, BERT), evaluate them, and expose inference endpoints and dashboards via Flask + Dash.
File structure
• Top-level folders:
– scrapers, builders, creators, trainers, evaluators, optimizers, maintenance, labelers, visualizations: batch-oriented Jupyter notebook pipelines for ETL, feature building, model training/tuning, evaluation, maintenance.
– flaskr: Flask application with server (REST endpoints and a Dash sub-app), producer/consumer for async tasks, static and template assets.
– src/functions + serverless.yml: serverless (AWS Lambda) functions (JS/Python) plus tests/templates.
– py/: standalone Python modules and APIs (data_generation_flask_api.py, quals_queue_flask_api.py, unscored_quals_flask_api.py) plus jobpostlib library for shared utilities.
– saves/: local data store (CSV, SQL/Neo4j dumps, pickled artifacts, graphs, presentations).
– ps1, sh: environment and deployment scripts (PowerShell and Bash).
• Configuration & provisioning: environment.yml, requirements.txt, jupyter_notebook_config.py, serverless.yml.
README for additional insights
• Describes the end-to-end job-hunting automation: ingestion from URLs or CSV, CRF-based section detection, tokenization → TF-IDF → predictive models to guide application decisions.
• Notes reliance on local file paths, Neo4j, Conda, and adjustments for OS path separators.
Main components & interactions
a. Components
1. External Services: Indeed, Glassdoor, other job sites
2. Scraper Layer (Jupyter notebooks in scrapers/)
3. Raw Data Storage (local file system: HTML, CSV in saves/)
4. ETL & Graph DB Ingestion (builders/creators notebooks → Neo4j via cypher/)
5. Feature Extraction & Dictionary Builders (builders/, creators/)
6. Model Training & Optimization (trainers/, optimizers/)
7. Model Artifacts Store (pickled models & topic models in saves/pkl/)
8. Evaluation Layer (evaluators/)
9. Inference API Layer
– Flask app (flaskr/app: server.py, producer/consumer, dash_app.py)
– Python micro-APIs (py/*_flask_api.py)
– Serverless functions (src/functions + serverless.yml)
10. Front-end & Visualization
– Flask templates (Jinja2), Dash UIs, standalone Dash apps (py/data_visualization_dash_app.py, py/unscored_quals_visualization_dash_app.py)
11. DevOps & Environment
– Conda env (environment.yml, ps1/, sh/)
– Versioning (git, .gitignore, .gitmodules)
b. Relationships & data flows
• External Services → Scraper Layer → Raw Data Storage
• Raw Data Storage → ETL/Graph Ingestion → Neo4j
• Neo4j & Raw Data → Feature Extraction → Dictionaries/CSV
• Dictionaries/CSV → Model Training → Model Artifacts Store
• Model Artifacts Store → Evaluation Layer (feedback loop back to Training)
• Model Artifacts Store → Inference API Layer
• Inference API Layer ↔ Front-end & Visualization (HTTP requests)
• DevOps & Environment scripts support all layers
c. Architectural patterns & principles
• Batch pipeline for data ingestion, feature building, and model training (notebook-driven).
• Service-oriented inference layer (Flask microservices + serverless functions).
• Clear separation of concerns: scraping, ETL, model lifecycle, API, UI.
• Reusable library module (jobpostlib) for shared utilities.
• Polyglot deployment scripts (PowerShell, Bash, serverless.yml).
d. Key technologies
• Python and Jupyter notebooks
• Neo4j (graph database)
• CRF, logistic regression, LDA, BERT for ML
• Flask & Dash for web server & visualization
• AWS Lambda / Serverless framework
• Conda for environment management