# reodite/ubc-unified-data architecture

> Architecture diagram of the GitHub repository reodite/ubc-unified-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reodite/ubc-unified-data
- Repository: https://github.com/reodite/ubc-unified-data

## Overview

Unified UBC Data is a public-data collection and publishing repository: refresh jobs gather official institutional information into queryable JSON/CSV/GeoJSON datasets. The documented workflow is collection, normalization and publication, then optional validation; the README describes broad student-facing collections beyond the sampled implementation. The excerpts show selected parser, prose, host-crawl, and validation relationships, but not the full update orchestration or most collection modules, so those links are omitted or identified as documented.

Main entry points: [`src/base.ts`](https://github.com/reodite/ubc-unified-data/blob/main/src/base.ts) (Output and HTTP), [`data`](https://github.com/reodite/ubc-unified-data/tree/main/data) (Published datasets), [`src/source-documents.ts`](https://github.com/reodite/ubc-unified-data/blob/main/src/source-documents.ts) (Source documents), [`src/htmldoc.ts`](https://github.com/reodite/ubc-unified-data/blob/main/src/htmldoc.ts) (HTML reader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collect["Collection"]
  node_update["Refresh all<br/>[update.ts]"]
  node_host_collect["Host crawler<br/>[collect-host.ts]"]
  node_host_crawl["Crawl policy"]
  node_prose_collect["Prose collector<br/>[collect-prose.ts]"]
  node_prose_runtime["Prose collection"]
end

subgraph group_transform["Parsing"]
  node_calendar["Calendar pages<br/>[calendarpages.ts]"]
  node_course_text["Course text parser<br/>[coursetext.ts]"]
  node_html_doc["HTML reader<br/>[htmldoc.ts]"]
  node_entities["HTML entities<br/>[entities.ts]"]
  node_source_docs["Source documents"]
end

subgraph group_publish["Data publishing"]
  node_base["Output and HTTP<br/>[base.ts]"]
  node_public_facts["Public facts<br/>[public-facts.ts]"]
  node_dataset[("Published datasets")]
  node_admissions_data[("Admissions data")]
  node_services_data[("Campus services")]
  node_calendar_data[("Academic calendar")]
end

subgraph group_validate["Validation"]
  node_validate_prose["Prose validation<br/>[validate-prose.ts]"]
  node_validate_undergrad["Undergrad validation"]
end

node_user(("Data maintainer"))
node_user_result(("Student or researcher"))

node_user -.->|"starts refresh"| node_update
node_update -->|"refreshes all"| node_dataset
node_host_collect -->|"uses HTTP"| node_base
node_host_collect -->|"uses policy"| node_host_crawl
node_prose_collect -->|"collects sources"| node_prose_runtime
node_prose_collect -->|"uses utilities"| node_base
node_calendar -->|"parses text"| node_course_text
node_calendar -->|"reads HTML"| node_html_doc
node_course_text -->|"decodes HTML"| node_html_doc
node_html_doc -->|"decodes entities"| node_entities
node_source_docs -->|"uses HTTP types"| node_base
node_source_docs -->|"cleans HTML"| node_html_doc
node_public_facts -->|"uses output types"| node_base
node_validate_prose -->|"reads data helpers"| node_base
node_validate_prose -->|"checks documents"| node_source_docs
node_validate_undergrad -->|"reads data helpers"| node_base
node_validate_undergrad -->|"checks facts"| node_public_facts
node_validate_undergrad -->|"uses row types"| node_source_docs
node_dataset -->|"supports querying"| node_user_result
node_dataset -->|"includes collection"| node_admissions_data
node_dataset -->|"includes collection"| node_services_data
node_dataset -->|"includes collection"| node_calendar_data

click node_update "https://github.com/reodite/ubc-unified-data/blob/main/src/update.ts"
click node_host_collect "https://github.com/reodite/ubc-unified-data/blob/main/src/collect-host.ts"
click node_host_crawl "https://github.com/reodite/ubc-unified-data/tree/main/src/host-crawl"
click node_prose_collect "https://github.com/reodite/ubc-unified-data/blob/main/src/collect-prose.ts"
click node_prose_runtime "https://github.com/reodite/ubc-unified-data/tree/main/src/prose"
click node_calendar "https://github.com/reodite/ubc-unified-data/blob/main/src/calendarpages.ts"
click node_course_text "https://github.com/reodite/ubc-unified-data/blob/main/src/coursetext.ts"
click node_html_doc "https://github.com/reodite/ubc-unified-data/blob/main/src/htmldoc.ts"
click node_entities "https://github.com/reodite/ubc-unified-data/blob/main/src/entities.ts"
click node_source_docs "https://github.com/reodite/ubc-unified-data/blob/main/src/source-documents.ts"
click node_base "https://github.com/reodite/ubc-unified-data/blob/main/src/base.ts"
click node_public_facts "https://github.com/reodite/ubc-unified-data/blob/main/src/public-facts.ts"
click node_dataset "https://github.com/reodite/ubc-unified-data/tree/main/data"
click node_admissions_data "https://github.com/reodite/ubc-unified-data/tree/main/data/admissions"
click node_services_data "https://github.com/reodite/ubc-unified-data/tree/main/data/campus-services"
click node_calendar_data "https://github.com/reodite/ubc-unified-data/tree/main/data/academic-calendar"
click node_validate_prose "https://github.com/reodite/ubc-unified-data/blob/main/src/validate-prose.ts"
click node_validate_undergrad "https://github.com/reodite/ubc-unified-data/blob/main/src/validate-undergrad.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_update,node_host_collect,node_host_crawl,node_prose_collect,node_prose_runtime toneBlue
class node_calendar,node_course_text,node_html_doc,node_entities,node_source_docs toneAmber
class node_base,node_public_facts,node_dataset,node_admissions_data,node_services_data,node_calendar_data toneMint
class node_validate_prose,node_validate_undergrad toneRose
class node_user,node_user_result toneIndigo
```

## Components

### Collection

- [Refresh all](https://github.com/reodite/ubc-unified-data/blob/main/src/update.ts): `src/update.ts`
- [Host crawler](https://github.com/reodite/ubc-unified-data/blob/main/src/collect-host.ts): `src/collect-host.ts`
- [Crawl policy](https://github.com/reodite/ubc-unified-data/tree/main/src/host-crawl): `src/host-crawl`
- [Prose collector](https://github.com/reodite/ubc-unified-data/blob/main/src/collect-prose.ts): `src/collect-prose.ts`
- [Prose collection](https://github.com/reodite/ubc-unified-data/tree/main/src/prose): `src/prose`

### Parsing

- [Calendar pages](https://github.com/reodite/ubc-unified-data/blob/main/src/calendarpages.ts): `src/calendarpages.ts`
- [Course text parser](https://github.com/reodite/ubc-unified-data/blob/main/src/coursetext.ts): `src/coursetext.ts`
- [HTML reader](https://github.com/reodite/ubc-unified-data/blob/main/src/htmldoc.ts): `src/htmldoc.ts`
- [HTML entities](https://github.com/reodite/ubc-unified-data/blob/main/src/entities.ts): `src/entities.ts`
- [Source documents](https://github.com/reodite/ubc-unified-data/blob/main/src/source-documents.ts): `src/source-documents.ts`

### Data publishing

- [Output and HTTP](https://github.com/reodite/ubc-unified-data/blob/main/src/base.ts): `src/base.ts`
- [Public facts](https://github.com/reodite/ubc-unified-data/blob/main/src/public-facts.ts): `src/public-facts.ts`
- [Published datasets](https://github.com/reodite/ubc-unified-data/tree/main/data): `data`
- [Admissions data](https://github.com/reodite/ubc-unified-data/tree/main/data/admissions): `data/admissions`
- [Campus services](https://github.com/reodite/ubc-unified-data/tree/main/data/campus-services): `data/campus-services`
- [Academic calendar](https://github.com/reodite/ubc-unified-data/tree/main/data/academic-calendar): `data/academic-calendar`

### Validation

- [Prose validation](https://github.com/reodite/ubc-unified-data/blob/main/src/validate-prose.ts): `src/validate-prose.ts`
- [Undergrad validation](https://github.com/reodite/ubc-unified-data/blob/main/src/validate-undergrad.ts): `src/validate-undergrad.ts`

### Other components

- **Data maintainer**
- **Student or researcher**

## Connections

- Data maintainer → Refresh all: starts refresh (evidence: `README.md`)
- Refresh all → Published datasets: refreshes all (evidence: `README.md`)
- Host crawler → Output and HTTP: uses HTTP (evidence: `src/collect-host.ts`)
- Host crawler → Crawl policy: uses policy (evidence: `src/collect-host.ts`)
- Prose collector → Prose collection: collects sources (evidence: `src/collect-prose.ts`)
- Prose collector → Output and HTTP: uses utilities (evidence: `src/collect-prose.ts`)
- Calendar pages → Course text parser: parses text (evidence: `src/calendarpages.ts`)
- Calendar pages → HTML reader: reads HTML (evidence: `src/calendarpages.ts`)
- Course text parser → HTML reader: decodes HTML (evidence: `src/coursetext.ts`)
- HTML reader → HTML entities: decodes entities (evidence: `src/htmldoc.ts`)
- Source documents → Output and HTTP: uses HTTP types (evidence: `src/source-documents.ts`)
- Source documents → HTML reader: cleans HTML (evidence: `src/source-documents.ts`)
- Public facts → Output and HTTP: uses output types (evidence: `src/public-facts.ts`)
- Prose validation → Output and HTTP: reads data helpers (evidence: `src/validate-prose.ts`)
- Prose validation → Source documents: checks documents (evidence: `src/validate-prose.ts`)
- Undergrad validation → Output and HTTP: reads data helpers (evidence: `src/validate-undergrad.ts`)
- Undergrad validation → Public facts: checks facts (evidence: `src/validate-undergrad.ts`)
- Undergrad validation → Source documents: uses row types (evidence: `src/validate-undergrad.ts`)
- Published datasets → Student or researcher: supports querying (evidence: `README.md`)
- Published datasets → Admissions data: includes collection (evidence: `README.md`)
- Published datasets → Campus services: includes collection (evidence: `README.md`)
- Published datasets → Academic calendar: includes collection (evidence: `README.md`)
