# nguyenanhducit2/sal_proj_app architecture

> Architecture diagram of the GitHub repository nguyenanhducit2/sal_proj_app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nguyenanhducit2/sal_proj_app
- Repository: https://github.com/nguyenanhducit2/sal_proj_app

## Overview

##### Project type and purpose
- This is a data‐science/ML pipeline project with an end‐user facing REST API.
- The goal is to scrape job postings, clean and explore the data, build a salary‐estimation model, and expose it via a Flask API for external clients.

##### File structure
- Top‐level scripts and notebooks:
- crawl_data.ipynb / Data Integration.py – web scraping (Selenium + Python)
- data_cleaning.py / data_cleaning‑checkpoint.ipynb – ETL/feature engineering
- data_eda.ipynb – exploratory analysis (pivot tables, plots)
- model_building.py – model training and evaluation (Random Forest, Linear, Ridge)
- salary_data_cleaned.csv / eda_data.csv / glassdoor_jobs.csv – intermediate datasets
- FlaskAPI/ directory (productionization layer):
- app.py – defines Flask routes
- data_input.py – request parsing and input validation
- models/model_file.p – serialized Random Forest model (pickle)
- requests.py – internal client or test calls to the API
- wsgi.py – WSGI entry point for deployment
- requirements.txt – Python dependencies

##### README for additional insights
- Four phases: Web scraping → Data cleaning → EDA → Model building → Flask API serving
- Key features extracted (Python, AWS, Excel, Spark…)
- Model choice: Random Forest (MAE = 11.22)
- API: accepts a JSON payload of feature values and returns a salary estimate

##### Detailed capture of architectural elements
- Raw vs. cleaned vs. feature‐engineered datasets as separate storage nodes.
- Explicitly show model artifact storage and loading path into Flask.
- External dependencies (Glassdoor site, client applications).
- Testing workflows (requests.py) to illustrate end‐to‐end validation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Web Scraper Service
    subgraph "Web Scraper Service"
        direction TB
        ScraperScript["Data Integration.py"]:::batch
        ScraperNotebook["crawl_data.ipynb"]:::batch
    end
    click ScraperScript "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/Data Integration.py"
    click ScraperNotebook "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/crawl_data.ipynb"

    %% Raw Data Store
    RawData[(glassdoor_jobs.csv)]:::data
    click RawData "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/glassdoor_jobs.csv"

    %% Data Processing Pipeline
    subgraph "Data Processing Pipeline"
        direction TB
        CleanerScript["data_cleaning.py"]:::batch
        CleanerNB1["data_cleaning‑checkpoint.ipynb"]:::batch
        CleanerNB2["data_cleaning.ipynb"]:::batch
    end
    click CleanerScript "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/data_cleaning.py"
    click CleanerNB1 "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/data_cleaning‑checkpoint.ipynb"
    click CleanerNB2 "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/data_cleaning.ipynb"

    CleanData[(salary_data_cleaned.csv)]:::data
    click CleanData "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/salary_data_cleaned.csv"

    %% Exploratory Analysis Module
    subgraph "Exploratory Analysis Module"
        direction TB
        EDA["data_eda.ipynb"]:::exploration
        EDAData[(eda_data.csv)]:::data
    end
    click EDA "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/data_eda.ipynb"
    click EDAData "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/eda_data.csv"

    %% Model Training Service
    subgraph "Model Training Service"
        direction TB
        Trainer["model_building.py"]:::training
    end
    click Trainer "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/model_building.py"

    ModelArtifact[(model_file.p)]:::data
    click ModelArtifact "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/models/model_file.p"

    %% Flask API Service
    subgraph "Flask API Service"
        direction TB
        APIServer["app.py"]:::api
        Input["data_input.py"]:::api
        Requests["requests.py"]:::api
        WSGI["wsgi.py"]:::api
        ReqFile["requirements.txt"]:::api
    end
    click APIServer "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/app.py"
    click Input "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/data_input.py"
    click Requests "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/requests.py"
    click WSGI "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/wsgi.py"
    click ReqFile "https://github.com/nguyenanhducit2/sal_proj_app/blob/main/FlaskAPI/requirements.txt"

    %% Client
    Client["Client Application"]:::external

    %% Data Flows
    ScraperScript -->|"outputs raw data"| RawData
    ScraperNotebook --> RawData
    RawData --> CleanerScript
    RawData --> CleanerNB1
    RawData --> CleanerNB2
    CleanerScript --> CleanData
    CleanerNB1 --> CleanData
    CleanerNB2 --> CleanData
    CleanData --> EDA
    CleanData --> Trainer
    EDA --> EDAData
    Trainer --> ModelArtifact
    ModelArtifact --> APIServer
    Client -->|"POST JSON"| APIServer
    APIServer -->|"returns estimate"| Client

    %% Styles
    classDef batch fill:#d0e1f9,stroke:#0366d6,color:#0366d6;
    classDef data fill:#e8f1f2,stroke:#31708f,color:#31708f;
    classDef exploration fill:#f3e6ff,stroke:#8c37ff,color:#8c37ff;
    classDef training fill:#dbf0dc,stroke:#228b22,color:#228b22;
    classDef api fill:#fff2cc,stroke:#d69f00,color:#d69f00;
    classDef external fill:#f5f5f5,stroke:#999999,color:#666666;
```
