# khaung-jpg/ecs171-sofs architecture

> Architecture diagram of the GitHub repository khaung-jpg/ecs171-sofs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/khaung-jpg/ecs171-sofs
- Repository: https://github.com/khaung-jpg/ecs171-sofs

## Overview

This repository supports analysis of Stack Overflow developer surveys through data acquisition, year-specific preparation, shared cleaning and schema harmonization, and exploratory/modeling notebooks. The clearest implemented workflow is cleaning a survey frame and passing its result to harmonize; the harmonizer documents that usage. The downloader fetches survey archive files from GitHub. The tree also shows prepared CSV collections and analysis notebooks, but their runtime relationships are not established by the available excerpts, so those links are omitted.

Main entry points: [`download_data.py`](https://github.com/khaung-jpg/ecs171-sofs/blob/main/download_data.py) (Archive Downloader), [`clean_survey.py`](https://github.com/khaung-jpg/ecs171-sofs/blob/main/clean_survey.py) (Survey Cleaner), [`harmonize.py`](https://github.com/khaung-jpg/ecs171-sofs/blob/main/harmonize.py) (Schema Harmonizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Data Acquisition"]
  node_downloader["Archive Downloader<br/>[download_data.py]"]
end

subgraph group_preparation["Survey Preparation"]
  node_year_cleaning["Year Cleaning"]
  node_cleaner["Survey Cleaner<br/>[clean_survey.py]"]
  node_harmonizer["Schema Harmonizer<br/>[harmonize.py]"]
end

subgraph group_datasets["Survey Datasets"]
  node_canonical_data[("Canonical Datasets")]
  node_premerge_data[("Premerge Datasets")]
  node_cleaned_data[("Cleaned Datasets")]
  node_cleaned_v2_data[("Cleaned V2 Datasets")]
  node_harmonized_data[("Harmonized Data")]
end

subgraph group_analysis["Analysis"]
  node_eda["Survey Exploration"]
  node_fixed_eda["Fixed Data EDA"]
  node_modeling["Modeling"]
  node_merged_analysis["Merged Data Analysis"]
  node_merged_analysis_v2["Merged Data Analysis V2"]
end

node_analyst(("Analyst"))
node_archive["Survey Archive"]

node_downloader -->|"downloads from"| node_archive
node_cleaner -->|"provides cleaned frame"| node_harmonizer

click node_downloader "https://github.com/khaung-jpg/ecs171-sofs/blob/main/download_data.py"
click node_year_cleaning "https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaning_notebooks"
click node_cleaner "https://github.com/khaung-jpg/ecs171-sofs/blob/main/clean_survey.py"
click node_harmonizer "https://github.com/khaung-jpg/ecs171-sofs/blob/main/harmonize.py"
click node_canonical_data "https://github.com/khaung-jpg/ecs171-sofs/tree/main/clean_dataset"
click node_premerge_data "https://github.com/khaung-jpg/ecs171-sofs/tree/main/premerge_cleaned_dataset"
click node_cleaned_data "https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaned"
click node_cleaned_v2_data "https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaned_v2"
click node_harmonized_data "https://github.com/khaung-jpg/ecs171-sofs/tree/main/data/harmonized"
click node_eda "https://github.com/khaung-jpg/ecs171-sofs/blob/main/Data%20Analysis.ipynb"
click node_fixed_eda "https://github.com/khaung-jpg/ecs171-sofs/blob/main/AllYrsFixed_EDA.ipynb"
click node_modeling "https://github.com/khaung-jpg/ecs171-sofs/blob/main/CURRENTmodeling.ipynb"
click node_merged_analysis "https://github.com/khaung-jpg/ecs171-sofs/blob/main/MASTERcleanfixMergedData.ipynb"
click node_merged_analysis_v2 "https://github.com/khaung-jpg/ecs171-sofs/blob/main/MASTERcleanfixMergedData_v2.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_downloader toneBlue
class node_year_cleaning,node_cleaner,node_harmonizer toneAmber
class node_canonical_data,node_premerge_data,node_cleaned_data,node_cleaned_v2_data,node_harmonized_data toneMint
class node_eda,node_fixed_eda,node_modeling,node_merged_analysis,node_merged_analysis_v2 toneRose
class node_analyst,node_archive toneIndigo
```

## Components

### Data Acquisition

- [Archive Downloader](https://github.com/khaung-jpg/ecs171-sofs/blob/main/download_data.py): `download_data.py`

### Survey Preparation

- [Year Cleaning](https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaning_notebooks): `cleaning_notebooks`
- [Survey Cleaner](https://github.com/khaung-jpg/ecs171-sofs/blob/main/clean_survey.py): `clean_survey.py`
- [Schema Harmonizer](https://github.com/khaung-jpg/ecs171-sofs/blob/main/harmonize.py): `harmonize.py`

### Survey Datasets

- [Canonical Datasets](https://github.com/khaung-jpg/ecs171-sofs/tree/main/clean_dataset): `clean_dataset`
- [Premerge Datasets](https://github.com/khaung-jpg/ecs171-sofs/tree/main/premerge_cleaned_dataset): `premerge_cleaned_dataset`
- [Cleaned Datasets](https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaned): `cleaned`
- [Cleaned V2 Datasets](https://github.com/khaung-jpg/ecs171-sofs/tree/main/cleaned_v2): `cleaned_v2`
- [Harmonized Data](https://github.com/khaung-jpg/ecs171-sofs/tree/main/data/harmonized): `data/harmonized`

### Analysis

- [Survey Exploration](https://github.com/khaung-jpg/ecs171-sofs/blob/main/Data%20Analysis.ipynb): `Data Analysis.ipynb`
- [Fixed Data EDA](https://github.com/khaung-jpg/ecs171-sofs/blob/main/AllYrsFixed_EDA.ipynb): `AllYrsFixed_EDA.ipynb`
- [Modeling](https://github.com/khaung-jpg/ecs171-sofs/blob/main/CURRENTmodeling.ipynb): `CURRENTmodeling.ipynb`
- [Merged Data Analysis](https://github.com/khaung-jpg/ecs171-sofs/blob/main/MASTERcleanfixMergedData.ipynb): `MASTERcleanfixMergedData.ipynb`
- [Merged Data Analysis V2](https://github.com/khaung-jpg/ecs171-sofs/blob/main/MASTERcleanfixMergedData_v2.ipynb): `MASTERcleanfixMergedData_v2.ipynb`

### Other components

- **Analyst**
- **Survey Archive**

## Connections

- Archive Downloader → Survey Archive: downloads from (evidence: `download_data.py`)
- Survey Cleaner → Schema Harmonizer: provides cleaned frame (evidence: `harmonize.py`)
