How nguyenanhducit2/api_projdt works
Project Type and Purpose
• This is a data‑science/ML estimator tool that consists of two broad phases:
a. Offline data pipeline (scraping, cleaning, EDA, model training)
b. A production inference service (Flask API that loads a serialized model and returns salary estimates)
• Its primary goal is to let clients send job‑feature payloads and receive a data‑science salary estimate.
File Structure Analysis
• Top‑level scripts and notebooks under the root handle data ingestion (crawl_data.ipynb, crawl_Integrated_data.csv), cleaning (data preprocessing..py, data_cleaned.csv), EDA (data_eda.ipynb, eda_data.csv), and model building (model_building.py, resulting in model_file.p).
• The FlaskAPI directory contains the production service:
• app.py – entry point for the Flask application
• api.py – defines the HTTP endpoint(s) for inference
• data_input.py – request payload parsing & feature‐vector assembly
• models/model_file.p – serialized RandomForest model
• requests.py – example client that calls the API
• requirements.txt – Python dependencies
README Insights
• Data is scraped from Glassdoor using Python & Selenium (1000+ records)
• Feature engineering extracts numeric indicators for skills (Python, R, Excel, AWS, Spark)
• Model training compares RandomForest, LinearRegression, Ridge; RandomForest wins
• Productionization uses Flask to expose a POST endpoint for salary predictions
Main Components
a. Data Ingestion & Scraping
• Selenium‐based scraper script → raw CSV
b. Data Cleaning & Feature Engineering
• Preprocessing scripts/notebooks (Pandas transformations) → cleaned CSV & feature matrix
c. EDA & Model Training
• EDA notebooks for visualization
• model_building.py → trains models → serializes best model (model_file.p)
d. Model Storage
• model_file.p in FlaskAPI/models
e. Production API Service
• Flask app loads model at startup
• /predict endpoint (api.py) uses data_input.py to validate & vectorize incoming JSON
• Returns JSON response with estimated salary
f. External Client
• requests.py demonstrates HTTP POST to the Flask endpoint
Component Relationships & Data Flow
• At runtime: External Client → Flask API → Model Store → Prediction → Flask API → External Client
Architectural Patterns & Design Principles
• Pipeline pattern for offline data processing stages
• Separation of concerns:
• Data pipeline scripts separated from production code
• API uses a lightweight service encapsulating only inference logic
• Model serialization for reproducible deployments
• Configurable requirements.txt for environment setup
Key Technologies & Frameworks to Annotate
• Python 3.x
• Selenium (scraping)
• Pandas / NumPy (cleaning & feature engineering)
• scikit‑learn (model training, serialization)
• Flask (API)
• pickle (model persistence)
• HTTP/JSON (client–server communication)
Detail
• Offline stages (scraping → cleaning → EDA → training).
• Serialized model as a distinct artifact.
• How the FlaskAPI loads and uses it.
• Don’t omit configuration/dependencies (requirements.txt) and example client (requests.py).