# openedx/course-discovery architecture

> Architecture diagram of the GitHub repository openedx/course-discovery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openedx/course-discovery
- Repository: https://github.com/openedx/course-discovery

## Overview

##### Project Type & Purpose
- Full-stack backend service (Django + Django REST Framework) providing catalog APIs and search for consolidated course & program metadata.
- Integrates with external sources (Salesforce, Contentful, Google APIs, Analytics API, Algolia) via data loaders and management commands.
- Employs Elasticsearch for search indexing, Celery for async tasks, Memcached for caching, and a relational database for persistence.

##### Key Components
A. Clients & Entry Points
- Browser or API consumer (Swagger UI, third-party integrations)
- Command line via `manage.py` or `make discovery-shell` for data loading/debug
B. Edge / Proxy
- Docker-deployed Load Balancer / API Gateway (in devstack or Kubernetes)
- Gunicorn WSGI server hosting Django application
C. Django Application
- course_discovery/ (settings, wsgi, URLs)
- apps/ (Django apps: api, course_metadata, publisher, tagging, taxonomy_support, learner_pathway, edx_* extensions, etc.)
- REST endpoints under `/api/v1` and `/api/v2` (DRF routers, views, serializers)
- Admin site (course_metadata models, tagging UI)
- Celery configuration (celery.py) for background jobs
D. Data Stores
- Relational DB (MySQL/Postgres) for primary metadata (Django models, migrations)
- Elasticsearch cluster for document indices (django-elasticsearch-dsl)
- Memcached for HTTP response caching (CompressedCacheResponseMixin)
- Message broker (Redis or RabbitMQ) for Celery tasks
E. Asynchronous Workers
- Celery Workers executing scheduled management commands and tasks (update_index, import_course_metadata, send_course_deadline_emails, etc.)
- Periodic tasks triggered via Celery Beat or external scheduler
F. External Services
- Salesforce (course_metadata.salesforce integration)
- Contentful (course_metadata.contentful_utils)
- Google APIs & GSpread (data loaders, client libs)
- Analytics API (data_loaders.analytics_api)
- Algolia (course_metadata.algolia_models & search_indexes)
- Any other LMS or partner APIs via core.api_client.lms

##### Relationships & Data Flows
- Client → Load Balancer/API Gateway → Gunicorn/Django
- Django request handler →
- Cache lookup / update in Memcached
- DB queries via Django ORM
- Elasticsearch queries via DRF filters & search_indexes
- Admin actions or management commands → Database changes → Signals → Elasticsearch indexing
- Celery Workers ← Message Broker ← Django (task.apply_async)
- Celery Workers → External Services (fetch, push) → DB → Elasticsearch → Up-to-date search indices
- Swagger UI served by Django’s static files

##### Architectural Patterns & Principles
- Monolithic Django codebase segmented into modular Django apps
- MVC (Models in apps/*/models.py, Views in apps/*/views.py/DRF, Templates for admin & docs)
- “Fat model” for business logic; DRF serializers for data shaping
- Separate sync (HTTP request) vs. async (Celery) processing
- CQRS-like read side (Elasticsearch) vs. write side (relational DB)
- Config as code: Dockerfile, docker-compose, Kubernetes manifests, GitHub workflows

##### Components to Include & Label
- API Gateway / Load Balancer
- Gunicorn WSGI → Django application
- Apps folder (highlight course_metadata, api, publisher, tagging, taxonomy_support…)
- Relational Database (MySQL/PostgreSQL)
- Elasticsearch cluster
- Memcached cache
- Message Broker (Redis/RabbitMQ)
- Celery Workers & Scheduler
- External integrations: Salesforce, Contentful, Google APIs, Analytics API, Algolia

##### Detail Level
- Django apps only at a high level (group under “Django Apps”)
- Frameworks (Django ORM, DRF, django-elasticsearch-dsl)
- CI/CD (GitHub workflows, Docker, Kubernetes) optionally as a side note

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Clients & Entry Points"
        Browser["Browser / API Consumer"]:::external
        CLI["CLI (manage.py)"]:::core
        Swagger["Swagger UI"]:::core
    end
    click CLI "https://github.com/openedx/course-discovery/blob/master/manage.py"
    click Swagger "https://github.com/openedx/course-discovery/blob/master/course_discovery/static/css/edx-swagger.css"
    click Swagger "https://github.com/openedx/course-discovery/blob/master/course_discovery/templates/rest_framework_swagger/index.html"
    click Browser "https://github.com/openedx/course-discovery/blob/master/course_discovery/urls.py"

    Browser -->|HTTP| APIGW
    CLI -->|manage.py| DjangoApp

    subgraph "Web Tier"
        APIGW["Load Balancer / API Gateway"]:::core
        Gunicorn["Gunicorn WSGI Server"]:::core
        DjangoApp["Django Application"]:::core
    end
    click APIGW "https://github.com/openedx/course-discovery/tree/master/Dockerfile"
    click APIGW "https://github.com/openedx/course-discovery/blob/master/docker-compose.yml"
    click APIGW "https://github.com/openedx/course-discovery/blob/master/.ci/docker-compose-ci.yml"
    click Gunicorn "https://github.com/openedx/course-discovery/blob/master/course_discovery/wsgi.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/settings/base.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/settings/production.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/settings/devstack.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/settings/kubernetes.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/settings/local.py"
    click DjangoApp "https://github.com/openedx/course-discovery/blob/master/course_discovery/urls.py"

    APIGW -->|HTTP| Gunicorn
    Gunicorn -->|WSGI| DjangoApp

    DjangoApp -->|Cache lookup| Memcached
    DjangoApp -->|ORM queries| RelationalDB
    DjangoApp -->|ES DSL queries| Elasticsearch
    DjangoApp -->|AMQP| Broker

    subgraph "Data Stores"
        Memcached["Memcached Cache"]:::dataStores
        RelationalDB["Relational DB\n(MySQL/Postgres)"]:::dataStores
        Elasticsearch["Elasticsearch Cluster"]:::dataStores
    end
    click Memcached "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/api/cache.py"
    click RelationalDB "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/models.py"
    click Elasticsearch "https://github.com/openedx/course-discovery/tree/master/course_discovery/apps/course_metadata/search_indexes/"

    Broker["Message Broker\n(Redis/RabbitMQ)"]:::async
    click Broker "https://github.com/openedx/course-discovery/blob/master/course_discovery/celery.py"

    subgraph "Worker Tier"
        Celery["Celery Workers"]:::async
        Beat["Celery Beat\n(Scheduler)"]:::async
    end
    click Celery "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/tasks.py"
    click Celery "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/management/commands/import_course_metadata.py"
    click Celery "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/management/commands/send_course_deadline_emails.py"
    click Celery "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/core/management/commands/install_es_indexes.py"

    Broker -->|AMQP tasks| Celery
    Beat -->|Periodic tasks| Celery

    subgraph "External Services"
        Salesforce["Salesforce"]:::external
        Contentful["Contentful"]:::external
        Google["Google APIs & GSpread"]:::external
        Analytics["Analytics API"]:::external
        Algolia["Algolia"]:::external
        LMS["LMS Client"]:::external
    end
    click Salesforce "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/salesforce.py"
    click Contentful "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/contentful_utils.py"
    click Google "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/googleapi_client.py"
    click Google "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/gspread_client.py"
    click Analytics "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/data_loaders/analytics_api.py"
    click Algolia "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/algolia_models.py"
    click Algolia "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/course_metadata/algolia_forms.py"
    click LMS "https://github.com/openedx/course-discovery/blob/master/course_discovery/apps/core/api_client/lms.py"

    Celery -->|Fetch/Push| Salesforce
    Celery -->|Fetch/Push| Contentful
    Celery -->|Fetch/Push| Google
    Celery -->|Fetch/Push| Analytics
    Celery -->|Fetch/Push| Algolia
    Celery -->|Fetch/Push| LMS

    Celery -->|Write| RelationalDB
    Celery -->|Index| Elasticsearch

    classDef core fill:#ADD8E6,stroke:#000
    classDef dataStores fill:#90EE90,stroke:#000
    classDef async fill:#FFA500,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
```
