# csc4693-fall-2026/csc4693 architecture

> Architecture diagram of the GitHub repository csc4693-fall-2026/csc4693, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/csc4693-fall-2026/csc4693
- Repository: https://github.com/csc4693-fall-2026/csc4693

## Overview

This repository is a collection of university-data research workflows, not one integrated application. Its main activities are collecting Cal Poly faculty listings, transforming faculty datasets, joining Cal Poly faculty with CSU salary records for major-level summaries, and preparing California college earnings data. The graph preserves these distinct workflows and their file-backed outputs. Source excerpts show several transformations and file references; the README documents no broader product workflow, and unsampled modules are represented only where their role is evident from the tree or index.

Main entry points: [`Faculty/data_scrapers/calpoly_faculty_collector.py`](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_scrapers/calpoly_faculty_collector.py) (Faculty collector), [`Steel Thread/CalPoly/join_calpoly.py`](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/join_calpoly.py) (Join faculty salaries), [`Steel Thread/graduate_data/clean_graduate.py`](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/graduate_data/clean_graduate.py) (Filter graduate records), [`Steel Thread/for_analysis/calpoly_major_medians.py`](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_major_medians.py) (Major pay medians)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Faculty collection"]
  node_collector["Faculty collector"]
  node_collected_faculty[("Collected faculty CSV")]
end

subgraph group_faculty["Faculty processing"]
  node_gcc_processor["GCC processor<br/>[gcc.py]"]
  node_tc_processor["Transparent California processor"]
  node_tc_data[("Faculty salary parquet")]
  node_uc_processor["UC CPRA processor<br/>[uc_cpra.py]"]
  node_uc_data[("UC faculty parquet")]
end

subgraph group_salary["Salary analysis"]
  node_faculty_html["Cal Poly faculty HTML"]
  node_parse_calpoly["Parse faculty table<br/>[parse_calpoly.py]"]
  node_faculty_csv[("Parsed faculty CSV")]
  node_csu_salary["CSU salary records"]
  node_join_calpoly["Join faculty salaries<br/>[join_calpoly.py]"]
  node_salary_data[("Cal Poly salary data")]
  node_median_calc["Major pay medians"]
  node_major_medians[("Major median results")]
end

subgraph group_earnings["Earnings analysis"]
  node_scorecard_source["Scorecard field data"]
  node_school_list["California school list"]
  node_clean_graduate["Filter graduate records<br/>[clean_graduate.py]"]
  node_ca_scorecard[("California scorecard data")]
  node_filter_calpoly["Map Cal Poly majors<br/>[filter_calpoly.py]"]
  node_calpoly_scorecard[("Cal Poly scorecard")]
  node_student_processor["Student earnings processor<br/>[process_data.py]"]
  node_student_data[("Student earnings parquet")]
end

node_catalog(("Cal Poly catalog"))
node_directory_sites(("College directories"))
node_student_sources(("Scorecard and crosswalk files"))

node_collector -->|"fetches catalog"| node_catalog
node_collector -->|"fetches listings"| node_directory_sites
node_collector -->|"writes records"| node_collected_faculty
node_tc_processor -->|"writes parquet"| node_tc_data
node_uc_processor -->|"writes parquet"| node_uc_data
node_faculty_html -->|"parsed from"| node_parse_calpoly
node_parse_calpoly -->|"writes rows"| node_faculty_csv
node_join_calpoly -->|"reads faculty"| node_faculty_csv
node_join_calpoly -->|"reads salaries"| node_csu_salary
node_join_calpoly -->|"writes filtered data"| node_salary_data
node_median_calc -->|"reads salary data"| node_salary_data
node_median_calc -->|"writes medians"| node_major_medians
node_clean_graduate -->|"filters records"| node_scorecard_source
node_clean_graduate -->|"filters by schools"| node_school_list
node_clean_graduate -->|"writes filtered data"| node_ca_scorecard
node_filter_calpoly -->|"maps majors"| node_ca_scorecard
node_filter_calpoly -->|"writes mapped data"| node_calpoly_scorecard
node_student_processor -->|"reads source files"| node_student_sources
node_student_processor -->|"writes parquet"| node_student_data

click node_collector "https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_scrapers/calpoly_faculty_collector.py"
click node_collected_faculty "https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/Data/cal_poly_faculty.parquet"
click node_gcc_processor "https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/gcc.py"
click node_tc_processor "https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/transparent_california.py"
click node_uc_processor "https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/uc_cpra.py"
click node_faculty_html "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/calpoly_faculty_html.html"
click node_parse_calpoly "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/parse_calpoly.py"
click node_faculty_csv "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/calpoly_faculty.csv"
click node_csu_salary "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CSU_Data/california-state-university-salary-2024.csv"
click node_join_calpoly "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/join_calpoly.py"
click node_salary_data "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_salary_data.csv"
click node_median_calc "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_major_medians.py"
click node_major_medians "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_major_medians.csv"
click node_school_list "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/graduate_data/ca_universities"
click node_clean_graduate "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/graduate_data/clean_graduate.py"
click node_ca_scorecard "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/ca_collegescore.csv"
click node_filter_calpoly "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/filter_calpoly.py"
click node_calpoly_scorecard "https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_scorecard.csv"
click node_student_processor "https://github.com/csc4693-fall-2026/csc4693/blob/main/Students/process_data.py"
click node_student_data "https://github.com/csc4693-fall-2026/csc4693/blob/main/Students/data/student-earnings.parquet"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collector,node_collected_faculty toneBlue
class node_gcc_processor,node_tc_processor,node_tc_data,node_uc_processor,node_uc_data toneAmber
class node_faculty_html,node_parse_calpoly,node_faculty_csv,node_csu_salary,node_join_calpoly,node_salary_data,node_median_calc,node_major_medians toneMint
class node_scorecard_source,node_school_list,node_clean_graduate,node_ca_scorecard,node_filter_calpoly,node_calpoly_scorecard,node_student_processor,node_student_data toneRose
class node_catalog,node_directory_sites,node_student_sources toneIndigo
```

## Components

### Faculty collection

- [Faculty collector](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_scrapers/calpoly_faculty_collector.py): `Faculty/data_scrapers/calpoly_faculty_collector.py`
- [Collected faculty CSV](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/Data/cal_poly_faculty.parquet): `Faculty/Data/cal_poly_faculty.parquet`

### Faculty processing

- [GCC processor](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/gcc.py): `Faculty/data_processors/gcc.py`
- [Transparent California processor](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/transparent_california.py): `Faculty/data_processors/transparent_california.py`
- **Faculty salary parquet**
- [UC CPRA processor](https://github.com/csc4693-fall-2026/csc4693/blob/main/Faculty/data_processors/uc_cpra.py): `Faculty/data_processors/uc_cpra.py`
- **UC faculty parquet**

### Salary analysis

- [Cal Poly faculty HTML](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/calpoly_faculty_html.html): `Steel Thread/CalPoly/calpoly_faculty_html.html`
- [Parse faculty table](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/parse_calpoly.py): `Steel Thread/CalPoly/parse_calpoly.py`
- [Parsed faculty CSV](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/calpoly_faculty.csv): `Steel Thread/CalPoly/calpoly_faculty.csv`
- [CSU salary records](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CSU_Data/california-state-university-salary-2024.csv): `Steel Thread/CSU_Data/california-state-university-salary-2024.csv`
- [Join faculty salaries](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/CalPoly/join_calpoly.py): `Steel Thread/CalPoly/join_calpoly.py`
- [Cal Poly salary data](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_salary_data.csv): `Steel Thread/for_analysis/calpoly_salary_data.csv`
- [Major pay medians](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_major_medians.py): `Steel Thread/for_analysis/calpoly_major_medians.py`
- [Major median results](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_major_medians.csv): `Steel Thread/for_analysis/calpoly_major_medians.csv`

### Earnings analysis

- **Scorecard field data**
- [California school list](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/graduate_data/ca_universities): `Steel Thread/graduate_data/ca_universities`
- [Filter graduate records](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/graduate_data/clean_graduate.py): `Steel Thread/graduate_data/clean_graduate.py`
- [California scorecard data](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/ca_collegescore.csv): `Steel Thread/for_analysis/ca_collegescore.csv`
- [Map Cal Poly majors](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/filter_calpoly.py): `Steel Thread/for_analysis/filter_calpoly.py`
- [Cal Poly scorecard](https://github.com/csc4693-fall-2026/csc4693/blob/main/Steel%20Thread/for_analysis/calpoly_scorecard.csv): `Steel Thread/for_analysis/calpoly_scorecard.csv`
- [Student earnings processor](https://github.com/csc4693-fall-2026/csc4693/blob/main/Students/process_data.py): `Students/process_data.py`
- [Student earnings parquet](https://github.com/csc4693-fall-2026/csc4693/blob/main/Students/data/student-earnings.parquet): `Students/data/student-earnings.parquet`

### Other components

- **Cal Poly catalog**
- **College directories**
- **Scorecard and crosswalk files**

## Connections

- Faculty collector → Cal Poly catalog: fetches catalog (evidence: `Faculty/data_scrapers/calpoly_faculty_collector.py`)
- Faculty collector → College directories: fetches listings (evidence: `Faculty/data_scrapers/calpoly_faculty_collector.py`)
- Faculty collector → Collected faculty CSV: writes records (evidence: `Faculty/data_scrapers/calpoly_faculty_collector.py`)
- Transparent California processor → Faculty salary parquet: writes parquet (evidence: `Faculty/data_processors/transparent_california.py`)
- UC CPRA processor → UC faculty parquet: writes parquet (evidence: `Faculty/data_processors/uc_cpra.py`)
- Cal Poly faculty HTML → Parse faculty table: parsed from (evidence: `Steel Thread/CalPoly/parse_calpoly.py`)
- Parse faculty table → Parsed faculty CSV: writes rows (evidence: `Steel Thread/CalPoly/parse_calpoly.py`)
- Join faculty salaries → Parsed faculty CSV: reads faculty (evidence: `Steel Thread/CalPoly/join_calpoly.py`)
- Join faculty salaries → CSU salary records: reads salaries (evidence: `Steel Thread/CalPoly/join_calpoly.py`)
- Join faculty salaries → Cal Poly salary data: writes filtered data (evidence: `Steel Thread/CalPoly/join_calpoly.py`)
- Major pay medians → Cal Poly salary data: reads salary data (evidence: `Steel Thread/for_analysis/calpoly_major_medians.py`)
- Major pay medians → Major median results: writes medians (evidence: `Steel Thread/for_analysis/calpoly_major_medians.py`)
- Filter graduate records → Scorecard field data: filters records (evidence: `Steel Thread/graduate_data/clean_graduate.py`)
- Filter graduate records → California school list: filters by schools (evidence: `Steel Thread/graduate_data/clean_graduate.py`)
- Filter graduate records → California scorecard data: writes filtered data (evidence: `Steel Thread/graduate_data/clean_graduate.py`)
- Map Cal Poly majors → California scorecard data: maps majors (evidence: `Steel Thread/for_analysis/filter_calpoly.py`)
- Map Cal Poly majors → Cal Poly scorecard: writes mapped data (evidence: `Steel Thread/for_analysis/filter_calpoly.py`)
- Student earnings processor → Scorecard and crosswalk files: reads source files (evidence: `Students/process_data.py`)
- Student earnings processor → Student earnings parquet: writes parquet (evidence: `Students/process_data.py`)
