# dbabbitt/job-hunting architecture

> Architecture diagram of the GitHub repository dbabbitt/job-hunting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbabbitt/job-hunting
- Repository: https://github.com/dbabbitt/job-hunting

## Overview

##### Project type and purpose
- This is a full-stack data-processing and ML pipeline with a web-based inference and visualization layer.
- Purpose: scrape job postings (Indeed, Glassdoor, etc.), extract and tokenize minimum-requirements sections, train classification models (CRF, LR, BERT), evaluate them, and expose inference endpoints and dashboards via Flask + Dash.

##### File structure
- Top-level folders:
– scrapers, builders, creators, trainers, evaluators, optimizers, maintenance, labelers, visualizations: batch-oriented Jupyter notebook pipelines for ETL, feature building, model training/tuning, evaluation, maintenance.
– flaskr: Flask application with server (REST endpoints and a Dash sub-app), producer/consumer for async tasks, static and template assets.
– src/functions + serverless.yml: serverless (AWS Lambda) functions (JS/Python) plus tests/templates.
– py/: standalone Python modules and APIs (data_generation_flask_api.py, quals_queue_flask_api.py, unscored_quals_flask_api.py) plus jobpostlib library for shared utilities.
– saves/: local data store (CSV, SQL/Neo4j dumps, pickled artifacts, graphs, presentations).
– ps1, sh: environment and deployment scripts (PowerShell and Bash).
- Configuration & provisioning: environment.yml, requirements.txt, jupyter_notebook_config.py, serverless.yml.

##### README for additional insights
- Describes the end-to-end job-hunting automation: ingestion from URLs or CSV, CRF-based section detection, tokenization → TF-IDF → predictive models to guide application decisions.
- Notes reliance on local file paths, Neo4j, Conda, and adjustments for OS path separators.

##### Main components & interactions
a. Components
1. External Services: Indeed, Glassdoor, other job sites
2. Scraper Layer (Jupyter notebooks in scrapers/)
3. Raw Data Storage (local file system: HTML, CSV in saves/)
4. ETL & Graph DB Ingestion (builders/creators notebooks → Neo4j via cypher/)
5. Feature Extraction & Dictionary Builders (builders/, creators/)
6. Model Training & Optimization (trainers/, optimizers/)
7. Model Artifacts Store (pickled models & topic models in saves/pkl/)
8. Evaluation Layer (evaluators/)
9. Inference API Layer
– Flask app (flaskr/app: server.py, producer/consumer, dash_app.py)
– Python micro-APIs (py/*_flask_api.py)
– Serverless functions (src/functions + serverless.yml)
10. Front-end & Visualization
– Flask templates (Jinja2), Dash UIs, standalone Dash apps (py/data_visualization_dash_app.py, py/unscored_quals_visualization_dash_app.py)
11. DevOps & Environment
– Conda env (environment.yml, ps1/, sh/)
– Versioning (git, .gitignore, .gitmodules)

b. Relationships & data flows
- External Services → Scraper Layer → Raw Data Storage
- Raw Data Storage → ETL/Graph Ingestion → Neo4j
- Neo4j & Raw Data → Feature Extraction → Dictionaries/CSV
- Dictionaries/CSV → Model Training → Model Artifacts Store
- Model Artifacts Store → Evaluation Layer (feedback loop back to Training)
- Model Artifacts Store → Inference API Layer
- Inference API Layer ↔ Front-end & Visualization (HTTP requests)
- DevOps & Environment scripts support all layers

c. Architectural patterns & principles
- Batch pipeline for data ingestion, feature building, and model training (notebook-driven).
- Service-oriented inference layer (Flask microservices + serverless functions).
- Clear separation of concerns: scraping, ETL, model lifecycle, API, UI.
- Reusable library module (jobpostlib) for shared utilities.
- Polyglot deployment scripts (PowerShell, Bash, serverless.yml).

d. Key technologies
- Python and Jupyter notebooks
- Neo4j (graph database)
- CRF, logistic regression, LDA, BERT for ML
- Flask & Dash for web server & visualization
- AWS Lambda / Serverless framework
- Conda for environment management

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Services
    subgraph "External Services"
        direction TB
        Indeed["Indeed"]:::external
        Glassdoor["Glassdoor"]:::external
    end

    %% Batch Processing Pipeline
    subgraph "Batch Pipeline" 
        direction TB
        Scraper["Scraper Layer"]:::compute
        RawStorage["Raw Data Storage"]:::datastore
        ETL["ETL & Graph Ingestion"]:::compute
        FeatureBuilder["Feature & Dictionary Builders"]:::compute
        Trainer["Model Training & Optimization"]:::compute
        Evaluator["Evaluation Layer"]:::compute
        ModelStore["Model Artifacts Store"]:::datastore
    end

    %% Inference Service Layer
    subgraph "Inference API Layer"
        direction TB
        FlaskServer["Flask Server"]:::service
        Producer["Producer"]:::service
        Consumer["Consumer"]:::service
        DashSubApp["Dash Sub-App"]:::service
        MicroAPIs["Python Micro-APIs"]:::service
        Serverless["Serverless Functions"]:::service
    end

    %% Front-end & Visualization
    subgraph "Front-end & Visualization"
        direction TB
        Templates["Flask Templates & Static"]:::frontend
        DashApps["Standalone Dash Apps"]:::frontend
    end

    %% Shared Utilities
    SharedUtils["Shared Utilities (jobpostlib)"]:::compute

    %% DevOps & Environment
    subgraph "DevOps & Environment"
        direction TB
        CondaEnv["environment.yml"]:::devops
        PScripts["PowerShell Scripts"]:::devops
        BashScripts["Bash Scripts"]:::devops
        ServerlessConfig["serverless.yml"]:::devops
    end

    %% Connections
    Indeed -->|scrapes| Scraper
    Glassdoor -->|scrapes| Scraper
    Scraper -->|stores raw HTML/CSV| RawStorage
    RawStorage -->|feeds| ETL
    ETL -->|loads into Neo4j| RawStorage
    ETL -->|provides data| FeatureBuilder
    FeatureBuilder -->|features| Trainer
    Trainer -->|models| ModelStore
    ModelStore -->|artifacts| Evaluator
    Evaluator -->|feedback loop| Trainer
    ModelStore -->|serves models| FlaskServer
    FlaskServer -->|push tasks| Producer
    Producer -->|queue| Consumer
    Consumer -->|loads model| ModelStore
    FlaskServer -->|hosts| DashSubApp
    FlaskServer -->|calls| MicroAPIs
    FlaskServer -->|triggers| Serverless
    Templates -->|UI| FlaskServer
    DashApps -->|UI| MicroAPIs
    SharedUtils -->|used by| Scraper
    SharedUtils -->|used by| ETL
    SharedUtils -->|used by| FeatureBuilder
    SharedUtils -->|used by| Trainer
    SharedUtils -->|used by| Evaluator
    SharedUtils -->|used by| FlaskServer
    CondaEnv -->|configures| Scraper
    CondaEnv -->|configures| Trainer
    CondaEnv -->|configures| FlaskServer
    PScripts -->|deploys| Serverless
    BashScripts -->|deploys| Serverless
    ServerlessConfig -->|defines| Serverless

    %% Click Events
    click Scraper "https://github.com/dbabbitt/job-hunting/tree/master/scrapers/"
    click RawStorage "https://github.com/dbabbitt/job-hunting/tree/master/saves/csv/"
    click RawStorage "https://github.com/dbabbitt/job-hunting/tree/master/saves/dumps/"
    click RawStorage "https://github.com/dbabbitt/job-hunting/tree/master/saves/sql/"
    click ETL "https://github.com/dbabbitt/job-hunting/tree/master/builders/"
    click ETL "https://github.com/dbabbitt/job-hunting/tree/master/creators/"
    click ETL "https://github.com/dbabbitt/job-hunting/tree/master/cypher/"
    click FeatureBuilder "https://github.com/dbabbitt/job-hunting/tree/master/builders/"
    click FeatureBuilder "https://github.com/dbabbitt/job-hunting/tree/master/creators/"
    click Trainer "https://github.com/dbabbitt/job-hunting/tree/master/trainers/"
    click Trainer "https://github.com/dbabbitt/job-hunting/tree/master/optimizers/"
    click ModelStore "https://github.com/dbabbitt/job-hunting/tree/master/saves/pkl/"
    click Evaluator "https://github.com/dbabbitt/job-hunting/tree/master/evaluators/"
    click FlaskServer "https://github.com/dbabbitt/job-hunting/blob/master/flaskr/app/server.py"
    click Producer "https://github.com/dbabbitt/job-hunting/blob/master/flaskr/app/producer.py"
    click Consumer "https://github.com/dbabbitt/job-hunting/blob/master/flaskr/app/consumer.py"
    click DashSubApp "https://github.com/dbabbitt/job-hunting/blob/master/flaskr/app/dash_app.py"
    click MicroAPIs "https://github.com/dbabbitt/job-hunting/blob/master/py/data_generation_flask_api.py"
    click MicroAPIs "https://github.com/dbabbitt/job-hunting/blob/master/py/quals_queue_flask_api.py"
    click MicroAPIs "https://github.com/dbabbitt/job-hunting/blob/master/py/unscored_quals_flask_api.py"
    click Serverless "https://github.com/dbabbitt/job-hunting/tree/master/src/functions/"
    click ServerlessConfig "https://github.com/dbabbitt/job-hunting/blob/master/serverless.yml"
    click Templates "https://github.com/dbabbitt/job-hunting/tree/master/flaskr/templates/"
    click Templates "https://github.com/dbabbitt/job-hunting/tree/master/flaskr/static/"
    click DashApps "https://github.com/dbabbitt/job-hunting/blob/master/py/data_visualization_dash_app.py"
    click DashApps "https://github.com/dbabbitt/job-hunting/blob/master/py/unscored_quals_visualization_dash_app.py"
    click SharedUtils "https://github.com/dbabbitt/job-hunting/tree/master/py/jobpostlib/"
    click CondaEnv "https://github.com/dbabbitt/job-hunting/blob/master/environment.yml"
    click PScripts "https://github.com/dbabbitt/job-hunting/tree/master/ps1/"
    click BashScripts "https://github.com/dbabbitt/job-hunting/tree/master/sh/"

    %% Styles
    classDef datastore fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#98FB98,stroke:#333,stroke-width:1px
    classDef service fill:#FFD700,stroke:#333,stroke-width:1px
    classDef frontend fill:#FFA500,stroke:#333,stroke-width:1px
    classDef devops fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef external fill:#F0E68C,stroke:#333,stroke-width:1px
```
