# ahmedabdulghany7/tech-market-analysis architecture

> Architecture diagram of the GitHub repository ahmedabdulghany7/tech-market-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmedabdulghany7/tech-market-analysis
- Repository: https://github.com/ahmedabdulghany7/tech-market-analysis

## Overview

##### Project type and purpose
- This is a full-stack Python/ML data analysis application with an interactive dashboard.
- Purpose: analyze software job market data (Egyptian market & Stack Overflow Survey 2024), build ML models (salary prediction, demand forecasting), and expose insights via a Streamlit app.

##### File structure
- Top level:
- README.md – project overview and instructions
- Tech-Market-Analysis/ – contains serialized artifacts (.pkl files) and a ZIP (likely raw data, notebooks or code)
- Within `Tech-Market-Analysis/`:
- `prepare_*` pickles: data preprocessing pipelines
- `nn_model.pkl`: trained neural net for salary prediction
- `AI_ML_Demand_Predictor.pkl`: forecasting model
- `label_encoders.pkl`, `scaler.pkl`: feature-engineering artifacts
- `Tech-Market.zip`: presumably raw data, source code, notebooks, or static assets

##### Key insights from the README
- Two main ML deliverables:
a. Salary Prediction (NN based)
b. Demand Forecasting (Stack Overflow trends)
- Visualization layer: Streamlit dashboard (`app.py` invoked at runtime).
- Tech stack: Python, pandas, NumPy, scikit-learn, TensorFlow/Keras, Streamlit.

##### Main components & interactions
a. Data Sources
- External CSV/DB:
- Stack Overflow Developer Survey data
- Local Egyptian job market datasets
b. Data Processing & Feature Engineering
- Preprocessing modules (prepare_learning_data, prepare_remote_infra_data)
- Output: cleaned feature tables
c. Model Training
- Neural Network salary predictor (nn_model.pkl)
- Forecasting model (AI_ML_Demand_Predictor.pkl)
d. Model Artifacts Storage
- Serialized objects (.pkl) stored in `Tech-Market-Analysis/` directory
e. Streamlit Dashboard (app.py + UI Code)
- Loads pickles at startup
- Accepts user inputs (job type, experience, city, hours)
- Calls ML artifacts to generate predictions
- Renders plots and interactive visuals
f. Users
- Companies & employees interact via web browser

##### Architectural patterns & principles
- Clear separation of concerns:
- ETL/Preprocessing → Model Training → Model Serving (Streamlit)
- Reusable serialized artifacts to decouple training from serving
- Interactive visualization layer on top of ML backend
- Use of standard ML library pipelines (scikit-learn, Keras)

##### Detailed capture of essential elements
- Preprocessing pipelines (`prepare_learning_data`, `prepare_remote_infra_data`)
- Encoders and scaler as part of feature engineering block
- Tools: Python, pandas, scikit-learn, TensorFlow/Keras, Streamlit
- How the Streamlit app loads and uses the pickled objects at runtime

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    RawDataArchive["Raw Data Archive\n(Tech-Market.zip)"]:::dataStore
    click RawDataArchive "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market.zip"
    README["Project Overview & Instructions\n(README.md)"]:::external
    click README "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/README.md"

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        PreprocessLearning["prepare_learning_data\n(scikit-learn Pipeline)"]:::compute
        click PreprocessLearning "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/prepare_learning_data_Demand_Predictor.pkl"
        PreprocessInfra["prepare_remote_infra_data\n(scikit-learn Pipeline)"]:::compute
        click PreprocessInfra "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/prepare_remote_infra_data_Demand_Predictor.pkl"
        LabelEncoders["Label Encoders\n(label_encoders.pkl)"]:::compute
        click LabelEncoders "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/label_encoders.pkl"
        Scaler["Scaler\n(scaler.pkl)"]:::compute
        click Scaler "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/scaler.pkl"
    end

    RawDataArchive -->|"raw datasets"| PreprocessLearning
    RawDataArchive -->|"raw datasets"| PreprocessInfra
    PreprocessLearning -->|"cleaned features"| LabelEncoders
    PreprocessInfra -->|"cleaned features"| Scaler

    %% Model Training
    subgraph "Model Training"
        direction TB
        SalaryModel["Salary Prediction Model\n(nn_model.pkl)\n(TensorFlow/Keras)"]:::compute
        click SalaryModel "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/nn_model.pkl"
        DemandModel["Demand Forecasting Model\n(AI_ML_Demand_Predictor.pkl)"]:::compute
        click DemandModel "https://github.com/ahmedabdulghany7/tech-market-analysis/blob/main/Tech-Market-Analysis/AI_ML_Demand_Predictor.pkl"
    end

    LabelEncoders -->|"encoded features"| SalaryModel
    Scaler -->|"scaled features"| DemandModel

    %% Artifacts Storage
    ArtifactStoreDir["Model Artifacts Storage\n(Tech-Market-Analysis/)"]:::dataStore
    click ArtifactStoreDir "https://github.com/ahmedabdulghany7/tech-market-analysis/tree/main/Tech-Market-Analysis/"

    SalaryModel -->|"nn_model.pkl"| ArtifactStoreDir
    DemandModel -->|"AI_ML_Demand_Predictor.pkl"| ArtifactStoreDir

    %% Serving & Visualization
    subgraph "Serving & Visualization"
        direction TB
        StreamlitApp["Streamlit Dashboard\n(app.py)"]:::ui
    end

    ArtifactStoreDir -->|"loads .pkl artifacts"| StreamlitApp

    %% Users
    User["Users\n(Companies & Employees)"]:::external
    StreamlitApp -->|"interactive UI"| User

    %% Style Definitions
    classDef dataStore fill:#a2d5f2,stroke:#333,stroke-width:1px;
    classDef compute fill:#b4f8c8,stroke:#333,stroke-width:1px;
    classDef ui fill:#ffd6a5,stroke:#333,stroke-width:1px;
    classDef external fill:#e2e2e2,stroke:#333,stroke-width:1px;
```
