How hipo/university-domains-list works
1. This repository is primarily a data distribution repo, not a full application. Its core asset is a single canonical dataset: world_universities_and_domains.json, which contains university names, countries, ISO country codes, domains, and web pages.
2. The dataset is the main architectural boundary. Everything else in the repo exists to filter, validate, format, and continuously check that JSON file rather than to serve business logic or a UI.
3. filter.py is the only visible code artifact in the root that performs data transformation. Based on the README, it acts as a country-based filter utility: given one or more country names, it emits a reduced JSON file (filtered_world_universities_and_domains.json) for smaller downstream payloads.
4. The data flow is straightforward: canonical JSON in → optional filter step → smaller JSON out. There is no database, message queue, or backend service in this repo; the dataset itself is the source of truth.
5. The tests/ directory suggests the repository treats the data file as a product that must be machine-checked. The test files indicate coverage for:
- duplicate detection,
- filter behavior,
- formatting rules,
- schema/structure validation.
This implies the repo’s main risk is data integrity, not runtime logic.
6. tests/base_test.py and tests/requirements.txt indicate a lightweight Python test harness around the dataset and filter script. The tests are likely designed to run in CI and verify both the JSON payload and the utility script behavior.
7. The .github/workflows/ folder shows the CI/CD boundary. lint-test.yml is the primary quality gate, deploy.yml suggests publication or release automation, and ai-pr-review.py / ai-review.yml add automated review assistance. These workflows are around the repository, not inside the runtime path.
8. Local development tooling is centered on code/data hygiene: .pre-commit-config.yaml, .prettierrc, and the docs in docs/local_linting_guide.md and docs/local_testing_guide.md. This reinforces that maintainability and consistency are first-class concerns for the dataset.
9. The README’s “hosted API” and “built-in API” references describe external consumption paths, not an in-repo server implementation. The hosted API is served by a separate project/package (university-domains-list-api), while this repo provides the underlying data source.
10. In diagram terms, the architecture is best represented as:
- Source data: world_universities_and_domains.json
- Transform utility: filter.py
- Validation layer: tests/
- Automation layer: GitHub Actions + pre-commit/tooling
- Consumers: direct JSON readers, the hosted API, or downstream users filtering the dataset locally
11. The overall design is simple but important: a single authoritative dataset with a small supporting toolchain that ensures the file stays valid and can be selectively reduced for downstream use cases such as email-domain validation or university lookup.
Main entry points: tests Test suite · world_universities_and_domains.json Universities JSON · filter.py Filter script · lint-test.yml Lint & test
Components 16
Data
Canonical dataset and derived output
- Universities JSON · Single source of truth for university names, countries, domains, and web pages
world_universities_and_domains.json - Filtered JSON · Country-reduced output produced by the filter utility
Tooling
Local scripts and developer guides
- Filter script · Creates a smaller JSON payload from selected countries
filter.py - Dev guides · Local linting and testing guidance for contributors
- Pre-commit · Local hygiene checks before commits
.pre-commit-config.yaml
Validation
Checks that protect data integrity
- Test suite · Machine checks for data integrity and filter behavior
tests - Duplicates · Detects repeated or conflicting entries
tests/test_duplicates.py - Filter tests · Verifies country-based filtering output
tests/test_filter.py - Format tests · Enforces formatting expectations for the JSON payload
tests/test_format.py - Schema tests · Validates JSON structure and required fields
tests/test_schema.py
Automation
CI, review, and release workflows
- Lint & test · Primary CI gate for validation and hygiene
.github/workflows/lint-test.yml - Deploy · Publishes or packages the dataset for downstream use
.github/workflows/deploy.yml - AI review · Automated review assistance for pull requests
.github/workflows/ai-review.yml - AI review script · Script used by the AI review workflow
.github/workflows/ai-pr-review.py
Consumers
Ways the dataset is used outside the repo
- Hosted API · Separate consumer service built from this dataset
- Downstream users · Local readers and tools that consume the JSON directly
Connections 18
- Universities JSON → Filter script: source · The canonical dataset is the input to filtering
- Filter script → Filtered JSON: produces · Creates a reduced dataset for selected countries
- Universities JSON → Test suite: validated by · Core tests guard the canonical data file
- Filter script → Test suite: validated by · Filter behavior is covered by the test suite
- Test suite → Duplicates: includes · Duplicate detection is one validation slice
- Test suite → Filter tests: includes · Filter behavior is one validation slice
- Test suite → Format tests: includes · Formatting rules are one validation slice
- Test suite → Schema tests: includes · Schema checks are one validation slice
- Lint & test → Test suite: runs · CI executes the repository checks
- Lint & test → Pre-commit: aligns with · CI mirrors local hygiene rules
- Deploy → Universities JSON: publishes · Release automation centers on the canonical dataset
- AI review → AI review script: invokes · Workflow runs the review helper script
- AI review → Test suite: reviews against · Automated review is oriented around repository quality checks
- Dev guides → Filter script: documents · Guides explain local filtering and linting
- Dev guides → Test suite: documents · Guides explain how to run validation locally
- Hosted API → Universities JSON: consumes · Separate API project uses this repo as its data source
- Downstream users → Universities JSON: reads · Users can consume the canonical JSON directly
- Downstream users → Filtered JSON: reads · Users can also consume the filtered output