How khaung-jpg/ecs171-sofs works
This repository supports analysis of Stack Overflow developer surveys through data acquisition, year-specific preparation, shared cleaning and schema harmonization, and exploratory/modeling notebooks. The clearest implemented workflow is cleaning a survey frame and passing its result to harmonize; the harmonizer documents that usage. The downloader fetches survey archive files from GitHub. The tree also shows prepared CSV collections and analysis notebooks, but their runtime relationships are not established by the available excerpts, so those links are omitted.
Main entry points: download_data.py Archive Downloader · clean_survey.py Survey Cleaner · harmonize.py Schema Harmonizer
Components 16
Data Acquisition
- Archive Downloader
download_data.py
Survey Preparation
- Year Cleaning
cleaning_notebooks - Survey Cleaner
clean_survey.py - Schema Harmonizer
harmonize.py
Survey Datasets
- Canonical Datasets
clean_dataset - Premerge Datasets
premerge_cleaned_dataset - Cleaned Datasets
cleaned - Cleaned V2 Datasets
cleaned_v2 - Harmonized Data
data/harmonized
Analysis
- Survey Exploration
Data Analysis.ipynb - Fixed Data EDA
AllYrsFixed_EDA.ipynb - Modeling
CURRENTmodeling.ipynb - Merged Data Analysis
MASTERcleanfixMergedData.ipynb - Merged Data Analysis V2
MASTERcleanfixMergedData_v2.ipynb
Other components
- Analyst
- Survey Archive
Connections 2
- Archive Downloader → Survey Archive: downloads from
download_data.py - Survey Cleaner → Schema Harmonizer: provides cleaned frame
harmonize.py