How abbas99-hub/job-recommendation-system works
Project Type and Purpose
• This is a full-stack, machine-learning-driven job recommendation system.
• Purpose: scrape job data from Glassdoor, clean & engineer features, train a TF-IDF + NearestNeighbors model, expose an API and a Streamlit frontend, and deploy on Azure (Docker, ACR, AKS).
Main Architectural Layers & Components
A. Data Ingestion Layer
• Glassdoor Scraper (uses chromedriver)
• Raw Data Store (CSV / JSON in src/data)
B. Data Processing Layer
• jd_data_cleaner.py: missing-value handling, normalization
• jd_data_extractor.py: parse/unstructure job descriptions
• Skill Extractor Utility (Spacy-based PDF skill parsing in utilities)
C. Feature Engineering + Model Training Layer
• job_recommender.py: TF-IDF vectorizer, NearestNeighbors trainer
• Model Artifacts (serialized .pkl or similar)
• Jupyter Notebooks (src/notebook) for EDA & prototyping
D. Serving/API Layer
• Containerized service (e.g. Flask/FastAPI wrapper around model)
• API Endpoints:
• /recommend (accepts user resume or preferences)
• /skills (extract skills from uploaded resume)
E. Presentation Layer
• Streamlit App (__init__.py entrypoint)
• UI Components: file upload, results table, filters
F. Deployment & Infrastructure Layer
• Dockerfile (package model + API)
• Azure Container Registry (store images)
• Azure Kubernetes Service (or Azure App Service)
• Monitoring & Logging (Azure Monitor / Application Insights)
Relationships & Data Flows
1. Glassdoor → Scraper → Raw CSV/JSON
2. Raw Data → Cleaner/Extractor → Structured Feature Set
3. Structured Data → TF-IDF + NearestNeighbors → Model Artifact
4. Model Artifact + API Code → Docker Container → ACR → AKS
5. Streamlit UI ↔ API Service
6. User uploads resume → Skill Extractor → skills vector → /recommend → recommendations
Architectural Patterns & Design Principles
• Separation of Concerns: clearly separate ingestion, processing, modeling, API, and UI.
• Microservice-style Deployment: model and API in a container, scalable via AKS.
• CI/CD Ready: container registry + Kubernetes for automated builds and rollouts.
Technology Stack
• Python 3.10
• Scraping: Selenium + ChromeDriver
• Data Processing: pandas, numpy
• NLP: scikit-learn (TF-IDF, NearestNeighbors), SpaCy
• Frontend: Streamlit
• Packaging: setup.py, requirements.txt, Docker
• Cloud: Azure Container Registry, Azure Kubernetes Service
• Notebooks: Jupyter
• All six layers (ingestion, processing, modeling, API, UI, deployment)
• External dependencies (Glassdoor, Azure)
• Data stores (CSV/JSON, serialized model)
• Execution flow: from scraping to user recommendation
• Real-time path: resume upload → skill extraction → recommendation API → UI