How twangodev/uw-coursemap works
Project Type and Purpose
• This is a full-stack, data-driven web application (“uw-coursemap”) for browsing UW–Madison courses, visualizing prerequisite graphs, and searching professor ratings.
• It consists of three logical subsystems:
– A Python-based data-generation pipeline that scrapes and aggregates raw data into static JSON.
– A Python “search” microservice (with FastAPI or Flask) backed by Elasticsearch for full-text and filtered course searches.
– A SvelteKit/Node.js frontend that renders pages, fetches static JSON, and calls the search API.
File Structure
• Top-level orchestration:
– docker-compose.yml + Dockerfile: Compose ties together the SvelteKit app, the search service, and Elasticsearch.
– .github/workflows: CI/CD pipelines build and deploy Docker images, regenerate docs, run tests.
• generation/ directory:
– Python modules (aggregate.py, webscrape.py, etc.) that load source data (UW API, OSM geojson), process it, and emit static JSON into data/.
– main.py coordinates the pipeline.
• search/ directory:
– A Python-based REST API (app.py) that indexes JSON into Elasticsearch (via es_util.py) and exposes search endpoints.
– Dockerfile and docker-compose.yml allow this service to run in isolation (with an ES container).
• src/ directory:
– SvelteKit frontend:
• /routes: page definitions and load functions (+page.ts) that read from data/ or call the search API.
• /lib: reusable UI components (many .svelte files), API wrappers, types, and utilities.
• data/: generated JSON files consumed by the frontend and indexed by the search service.
README for Additional Insights
• Confirms core features (graph visualization, course details, search).
• Highlights use of OpenAPI for public endpoints (static-openapi.yaml).
Main Components
a. Data Generation Pipeline (Python)
– Scraper & loaders → aggregation → JSON output (data/ directory)
b. Search Microservice (Python + Elasticsearch)
– Reads JSON → indexes into ES → exposes RESTful search endpoints
c. Frontend Web App (SvelteKit / Node.js)
– Static-page generation & SSR → serves UI components → reads static JSON and calls search API
d. Infrastructure & Deployment
– Docker Compose orchestrator
– GitHub Actions CI/CD
Relationships and Interactions
• Data Pipeline → produces JSON → placed into shared volume or artifact store
• Search Service → ingests JSON (at container startup or via manual trigger) → stores in Elasticsearch → API endpoints available
• Frontend →
– On build/SSR, reads static JSON files (prereqs, courses, instructors)
– At runtime, issues HTTP calls to Search Service’s REST API
• Docker Compose → wires together the three containers + Elasticsearch
• GitHub Actions → automates testing, building, versioning, and pushing images
Architectural Patterns and Design Principles
• Microservices: separate data-gen, search, and frontend concerns
• Component-based UI: Svelte components in /src/lib/components
• Static-site & SSR hybrid: pre-generated JSON for fast initial load + dynamic search API
• Containerization: Docker for consistent environments