How reodite/ubc-unified-data works
Unified UBC Data is a public-data collection and publishing repository: refresh jobs gather official institutional information into queryable JSON/CSV/GeoJSON datasets. The documented workflow is collection, normalization and publication, then optional validation; the README describes broad student-facing collections beyond the sampled implementation. The excerpts show selected parser, prose, host-crawl, and validation relationships, but not the full update orchestration or most collection modules, so those links are omitted or identified as documented.
Main entry points: base.ts Output and HTTP · data Published datasets · source-documents.ts Source documents · htmldoc.ts HTML reader
Components 20
Collection
- Refresh all
src/update.ts - Host crawler
src/collect-host.ts - Crawl policy
src/host-crawl - Prose collector
src/collect-prose.ts - Prose collection
src/prose
Parsing
- Calendar pages
src/calendarpages.ts - Course text parser
src/coursetext.ts - HTML reader
src/htmldoc.ts - HTML entities
src/entities.ts - Source documents
src/source-documents.ts
Data publishing
- Output and HTTP
src/base.ts - Public facts
src/public-facts.ts - Published datasets
data - Admissions data
data/admissions - Campus services
data/campus-services - Academic calendar
data/academic-calendar
Validation
- Prose validation
src/validate-prose.ts - Undergrad validation
src/validate-undergrad.ts
Other components
- Data maintainer
- Student or researcher
Connections 22
- Data maintainer → Refresh all: starts refresh
README.md - Refresh all → Published datasets: refreshes all
README.md - Host crawler → Output and HTTP: uses HTTP
src/collect-host.ts - Host crawler → Crawl policy: uses policy
src/collect-host.ts - Prose collector → Prose collection: collects sources
src/collect-prose.ts - Prose collector → Output and HTTP: uses utilities
src/collect-prose.ts - Calendar pages → Course text parser: parses text
src/calendarpages.ts - Calendar pages → HTML reader: reads HTML
src/calendarpages.ts - Course text parser → HTML reader: decodes HTML
src/coursetext.ts - HTML reader → HTML entities: decodes entities
src/htmldoc.ts - Source documents → Output and HTTP: uses HTTP types
src/source-documents.ts - Source documents → HTML reader: cleans HTML
src/source-documents.ts - Public facts → Output and HTTP: uses output types
src/public-facts.ts - Prose validation → Output and HTTP: reads data helpers
src/validate-prose.ts - Prose validation → Source documents: checks documents
src/validate-prose.ts - Undergrad validation → Output and HTTP: reads data helpers
src/validate-undergrad.ts - Undergrad validation → Public facts: checks facts
src/validate-undergrad.ts - Undergrad validation → Source documents: uses row types
src/validate-undergrad.ts - Published datasets → Student or researcher: supports querying
README.md - Published datasets → Admissions data: includes collection
README.md - Published datasets → Campus services: includes collection
README.md - Published datasets → Academic calendar: includes collection
README.md