How nguyenanhducit2/sal_proj_app works
Project type and purpose
• This is a data‐science/ML pipeline project with an end‐user facing REST API.
• The goal is to scrape job postings, clean and explore the data, build a salary‐estimation model, and expose it via a Flask API for external clients.
File structure
• Top‐level scripts and notebooks:
• crawl_data.ipynb / Data Integration.py – web scraping (Selenium + Python)
• data_cleaning.py / data_cleaning‑checkpoint.ipynb – ETL/feature engineering
• data_eda.ipynb – exploratory analysis (pivot tables, plots)
• model_building.py – model training and evaluation (Random Forest, Linear, Ridge)
• salary_data_cleaned.csv / eda_data.csv / glassdoor_jobs.csv – intermediate datasets
• FlaskAPI/ directory (productionization layer):
• app.py – defines Flask routes
• data_input.py – request parsing and input validation
• models/model_file.p – serialized Random Forest model (pickle)
• requests.py – internal client or test calls to the API
• wsgi.py – WSGI entry point for deployment
• requirements.txt – Python dependencies
README for additional insights
• Four phases: Web scraping → Data cleaning → EDA → Model building → Flask API serving
• Key features extracted (Python, AWS, Excel, Spark…)
• Model choice: Random Forest (MAE = 11.22)
• API: accepts a JSON payload of feature values and returns a salary estimate
Detailed capture of architectural elements
• Raw vs. cleaned vs. feature‐engineered datasets as separate storage nodes.
• Explicitly show model artifact storage and loading path into Flask.
• External dependencies (Glassdoor site, client applications).
• Testing workflows (requests.py) to illustrate end‐to‐end validation.