How csc4693-fall-2026/csc4693 works
This repository is a collection of university-data research workflows, not one integrated application. Its main activities are collecting Cal Poly faculty listings, transforming faculty datasets, joining Cal Poly faculty with CSU salary records for major-level summaries, and preparing California college earnings data. The graph preserves these distinct workflows and their file-backed outputs. Source excerpts show several transformations and file references; the README documents no broader product workflow, and unsampled modules are represented only where their role is evident from the tree or index.
Main entry points: calpoly_faculty_collector.py Faculty collector · join_calpoly.py Join faculty salaries · clean_graduate.py Filter graduate records · calpoly_major_medians.py Major pay medians
Components 26
Faculty collection
- Faculty collector
Faculty/data_scrapers/calpoly_faculty_collector.py - Collected faculty CSV
Faculty/Data/cal_poly_faculty.parquet
Faculty processing
- GCC processor
Faculty/data_processors/gcc.py - Transparent California processor
Faculty/data_processors/transparent_california.py - Faculty salary parquet
- UC CPRA processor
Faculty/data_processors/uc_cpra.py - UC faculty parquet
Salary analysis
- Cal Poly faculty HTML
Steel Thread/CalPoly/calpoly_faculty_html.html - Parse faculty table
Steel Thread/CalPoly/parse_calpoly.py - Parsed faculty CSV
Steel Thread/CalPoly/calpoly_faculty.csv - CSU salary records
Steel Thread/CSU_Data/california-state-university-salary-2024.csv - Join faculty salaries
Steel Thread/CalPoly/join_calpoly.py - Cal Poly salary data
Steel Thread/for_analysis/calpoly_salary_data.csv - Major pay medians
Steel Thread/for_analysis/calpoly_major_medians.py - Major median results
Steel Thread/for_analysis/calpoly_major_medians.csv
Earnings analysis
- Scorecard field data
- California school list
Steel Thread/graduate_data/ca_universities - Filter graduate records
Steel Thread/graduate_data/clean_graduate.py - California scorecard data
Steel Thread/for_analysis/ca_collegescore.csv - Map Cal Poly majors
Steel Thread/for_analysis/filter_calpoly.py - Cal Poly scorecard
Steel Thread/for_analysis/calpoly_scorecard.csv - Student earnings processor
Students/process_data.py - Student earnings parquet
Students/data/student-earnings.parquet
Other components
- Cal Poly catalog
- College directories
- Scorecard and crosswalk files
Connections 19
- Faculty collector → Cal Poly catalog: fetches catalog
Faculty/data_scrapers/calpoly_faculty_collector.py - Faculty collector → College directories: fetches listings
Faculty/data_scrapers/calpoly_faculty_collector.py - Faculty collector → Collected faculty CSV: writes records
Faculty/data_scrapers/calpoly_faculty_collector.py - Transparent California processor → Faculty salary parquet: writes parquet
Faculty/data_processors/transparent_california.py - UC CPRA processor → UC faculty parquet: writes parquet
Faculty/data_processors/uc_cpra.py - Cal Poly faculty HTML → Parse faculty table: parsed from
Steel Thread/CalPoly/parse_calpoly.py - Parse faculty table → Parsed faculty CSV: writes rows
Steel Thread/CalPoly/parse_calpoly.py - Join faculty salaries → Parsed faculty CSV: reads faculty
Steel Thread/CalPoly/join_calpoly.py - Join faculty salaries → CSU salary records: reads salaries
Steel Thread/CalPoly/join_calpoly.py - Join faculty salaries → Cal Poly salary data: writes filtered data
Steel Thread/CalPoly/join_calpoly.py - Major pay medians → Cal Poly salary data: reads salary data
Steel Thread/for_analysis/calpoly_major_medians.py - Major pay medians → Major median results: writes medians
Steel Thread/for_analysis/calpoly_major_medians.py - Filter graduate records → Scorecard field data: filters records
Steel Thread/graduate_data/clean_graduate.py - Filter graduate records → California school list: filters by schools
Steel Thread/graduate_data/clean_graduate.py - Filter graduate records → California scorecard data: writes filtered data
Steel Thread/graduate_data/clean_graduate.py - Map Cal Poly majors → California scorecard data: maps majors
Steel Thread/for_analysis/filter_calpoly.py - Map Cal Poly majors → Cal Poly scorecard: writes mapped data
Steel Thread/for_analysis/filter_calpoly.py - Student earnings processor → Scorecard and crosswalk files: reads source files
Students/process_data.py - Student earnings processor → Student earnings parquet: writes parquet
Students/process_data.py