How openedx/course-discovery works
Project Type & Purpose
• Full-stack backend service (Django + Django REST Framework) providing catalog APIs and search for consolidated course & program metadata.
• Integrates with external sources (Salesforce, Contentful, Google APIs, Analytics API, Algolia) via data loaders and management commands.
• Employs Elasticsearch for search indexing, Celery for async tasks, Memcached for caching, and a relational database for persistence.
Key Components
A. Clients & Entry Points
• Browser or API consumer (Swagger UI, third-party integrations)
• Command line via manage.py or make discovery-shell for data loading/debug
B. Edge / Proxy
• Docker-deployed Load Balancer / API Gateway (in devstack or Kubernetes)
• Gunicorn WSGI server hosting Django application
C. Django Application
• course_discovery/ (settings, wsgi, URLs)
• apps/ (Django apps: api, course_metadata, publisher, tagging, taxonomy_support, learner_pathway, edx_* extensions, etc.)
• REST endpoints under /api/v1 and /api/v2 (DRF routers, views, serializers)
• Admin site (course_metadata models, tagging UI)
• Celery configuration (celery.py) for background jobs
D. Data Stores
• Relational DB (MySQL/Postgres) for primary metadata (Django models, migrations)
• Elasticsearch cluster for document indices (django-elasticsearch-dsl)
• Memcached for HTTP response caching (CompressedCacheResponseMixin)
• Message broker (Redis or RabbitMQ) for Celery tasks
E. Asynchronous Workers
• Celery Workers executing scheduled management commands and tasks (update_index, import_course_metadata, send_course_deadline_emails, etc.)
• Periodic tasks triggered via Celery Beat or external scheduler
F. External Services
• Salesforce (course_metadata.salesforce integration)
• Contentful (course_metadata.contentful_utils)
• Google APIs & GSpread (data loaders, client libs)
• Analytics API (data_loaders.analytics_api)
• Algolia (course_metadata.algolia_models & search_indexes)
• Any other LMS or partner APIs via core.api_client.lms
Relationships & Data Flows
• Client → Load Balancer/API Gateway → Gunicorn/Django
• Django request handler →
• Cache lookup / update in Memcached
• DB queries via Django ORM
• Elasticsearch queries via DRF filters & search_indexes
• Admin actions or management commands → Database changes → Signals → Elasticsearch indexing
• Celery Workers ← Message Broker ← Django (task.apply_async)
• Celery Workers → External Services (fetch, push) → DB → Elasticsearch → Up-to-date search indices
• Swagger UI served by Django’s static files
Architectural Patterns & Principles
• Monolithic Django codebase segmented into modular Django apps
• MVC (Models in apps/*/models.py, Views in apps/*/views.py/DRF, Templates for admin & docs)
• “Fat model” for business logic; DRF serializers for data shaping
• Separate sync (HTTP request) vs. async (Celery) processing
• CQRS-like read side (Elasticsearch) vs. write side (relational DB)
• Config as code: Dockerfile, docker-compose, Kubernetes manifests, GitHub workflows
Components to Include & Label
• API Gateway / Load Balancer
• Gunicorn WSGI → Django application
• Apps folder (highlight course_metadata, api, publisher, tagging, taxonomy_support…)
• Relational Database (MySQL/PostgreSQL)
• Elasticsearch cluster
• Memcached cache
• Message Broker (Redis/RabbitMQ)
• Celery Workers & Scheduler
• External integrations: Salesforce, Contentful, Google APIs, Analytics API, Algolia
Detail Level
• Django apps only at a high level (group under “Django Apps”)
• Frameworks (Django ORM, DRF, django-elasticsearch-dsl)
• CI/CD (GitHub workflows, Docker, Kubernetes) optionally as a side note