How ahmedabdulghany7/tech-market-analysis works
Project type and purpose
• This is a full-stack Python/ML data analysis application with an interactive dashboard.
• Purpose: analyze software job market data (Egyptian market & Stack Overflow Survey 2024), build ML models (salary prediction, demand forecasting), and expose insights via a Streamlit app.
File structure
• Top level:
• README.md – project overview and instructions
• Tech-Market-Analysis/ – contains serialized artifacts (.pkl files) and a ZIP (likely raw data, notebooks or code)
• Within Tech-Market-Analysis/:
• prepare_* pickles: data preprocessing pipelines
• nn_model.pkl: trained neural net for salary prediction
• AI_ML_Demand_Predictor.pkl: forecasting model
• label_encoders.pkl, scaler.pkl: feature-engineering artifacts
• Tech-Market.zip: presumably raw data, source code, notebooks, or static assets
Key insights from the README
• Two main ML deliverables:
a. Salary Prediction (NN based)
b. Demand Forecasting (Stack Overflow trends)
• Visualization layer: Streamlit dashboard (app.py invoked at runtime).
• Tech stack: Python, pandas, NumPy, scikit-learn, TensorFlow/Keras, Streamlit.
Main components & interactions
a. Data Sources
• External CSV/DB:
• Stack Overflow Developer Survey data
• Local Egyptian job market datasets
b. Data Processing & Feature Engineering
• Preprocessing modules (prepare_learning_data, prepare_remote_infra_data)
• Output: cleaned feature tables
c. Model Training
• Neural Network salary predictor (nn_model.pkl)
• Forecasting model (AI_ML_Demand_Predictor.pkl)
d. Model Artifacts Storage
• Serialized objects (.pkl) stored in Tech-Market-Analysis/ directory
e. Streamlit Dashboard (app.py + UI Code)
• Loads pickles at startup
• Accepts user inputs (job type, experience, city, hours)
• Calls ML artifacts to generate predictions
• Renders plots and interactive visuals
f. Users
• Companies & employees interact via web browser
Architectural patterns & principles
• Clear separation of concerns:
• ETL/Preprocessing → Model Training → Model Serving (Streamlit)
• Reusable serialized artifacts to decouple training from serving
• Interactive visualization layer on top of ML backend
• Use of standard ML library pipelines (scikit-learn, Keras)
Detailed capture of essential elements
• Preprocessing pipelines (prepare_learning_data, prepare_remote_infra_data)
• Encoders and scaler as part of feature engineering block
• Tools: Python, pandas, scikit-learn, TensorFlow/Keras, Streamlit
• How the Streamlit app loads and uses the pickled objects at runtime