# dandi/dandi-archive architecture

> Architecture diagram of the GitHub repository dandi/dandi-archive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dandi/dandi-archive
- Repository: https://github.com/dandi/dandi-archive

## Overview

##### Main Components
- Backend (Django API):
▪ Contains multiple Django apps such as “analytics”, “api”, “search”, and “zarr”.
▪ Hosts REST API endpoints, models, views, and business logic.
▪ Uses management commands, migrations, and scheduled tasks (Celery workers as indicated by the “celery.py” file in “dandiapi”).
▪ Includes administrative and operational support (authentication, asset management, DOI generation, audit and embargo services).
- Frontend (Vue Application):
▪ Located in the “web” folder.
▪ Provides the user interface with a component-based structure (e.g., AppBar, DLP components, file browser, search, dashboards).
▪ Communicates with the backend via REST API calls.
- External and Supporting Components:
▪ Asynchronous Task Processing: The Celery integration (dandiapi/celery.py and related tasks) managing background jobs like data ingestion and garbage collection.
▪ Databases: Underlying relational database (likely PostgreSQL) that is interfaced by Django via its ORM.
▪ Storage/Blob Services: Management commands and routines for asset ingestion, embargo handling, and asset paths hint at integration with object storage (e.g., S3 or equivalent).
▪ CI/CD and Deployment: GitHub workflows, Docker Compose files, and environment files (in “dev” and “docker-compose.yml”) indicate containerized deployment, staging, and production environments.
▪ E2E Testing: End-to-end test suites in “e2e/playwright” and “e2e/puppeteer” verifying both API and UI interactions.
▪ External integrations: DOI registration and domain-specific services used for neurophysiology data.

##### Relationships and Interactions
- The Vue-based frontend makes API calls to the Django backend for data retrieval, authentication, file uploads/downloads, and search.
- Within the Django backend, the “api” app is the central hub:
▪ User requests are routed through Django’s URL configuration, which then dispatches them to the proper view (e.g., for dashboards, asset management, and file operations).
▪ Some endpoints trigger background tasks processed asynchronously by Celery workers.
- The Django apps (“analytics”, “search”, “zarr”) provide modular services. For example, “analytics” might log usage statistics, “search” handles data indexing and querying, and “zarr” manages data formats or bulk ingest processes.
- The database acts as a persistent store for metadata, user data, and audit logs.
- Deployment configuration (Dockerfiles, docker-compose) and GitHub workflows ensure consistent build, test, and deploy routines.

##### Architectural Patterns and Design Principles
- Modular Design: The separation of concerns into distinct Django apps mirrors the micro-module pattern. Each app (analytics, API, search, zarr) encapsulates related functionalities.
- RESTful API: The system uses Django REST practices, exposing endpoints that the frontend consumes.
- Asynchronous Processing: With Celery tasks handling long-running activities (e.g., asset ingestion and garbage collection), the architecture supports high throughput without blocking the main request-response cycle.
- CI/CD Integration: Automated testing, deployment workflows and Docker containers ensure robust delivery pipelines.

##### Final Advice
- Communication flow (REST API calls, asynchronous task handling, data persistence).

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Tier
    subgraph "Client Tier"
        FVue["Vue Frontend"]:::frontend
        FComp["UI Components"]:::frontend
        FViews["UI Views"]:::frontend
    end

    %% Server Tier
    subgraph "Server Tier"
        DAPI["DANDI API (Django Backend)"]:::backend
        AApp["Analytics App"]:::backend
        APIApp["API App"]:::backend
        SApp["Search App"]:::backend
        ZApp["Zarr App"]:::backend
    end

    %% Asynchronous Processing
    subgraph "Asynchronous Processing"
        Celery["Celery (Async Processing)"]:::async
    end

    %% Data Tier
    subgraph "Data Tier"
        DB["Relational Database (PostgreSQL)"]:::database
        Blob["Blob Storage (S3/Equivalent)"]:::database
    end

    %% CI/CD & Testing
    subgraph "CI/CD & Testing"
        CICD["GitHub Workflows"]:::deploy
        Docker["Docker & Env"]:::deploy
        E2EPlay["E2E Testing (Playwright)"]:::test
        E2EPup["E2E Testing (Puppeteer)"]:::test
    end

    %% Connections between tiers
    FVue -->|"RESTAPICalls"| DAPI
    FComp -->|"RESTAPICalls"| DAPI
    FViews -->|"RESTAPICalls"| DAPI

    %% Internal routing within Django backend
    DAPI -->|"routes_to"| AApp
    DAPI -->|"routes_to"| APIApp
    DAPI -->|"routes_to"| SApp
    DAPI -->|"routes_to"| ZApp

    %% Asynchronous tasks
    APIApp -->|"triggers_task"| Celery
    AApp -->|"scheduled_task"| Celery

    %% Backend interactions with data tier
    DAPI -->|"db_access"| DB
    DAPI -->|"asset_mgmt"| Blob

    %% Deployment and testing links
    CICD -->|"deploys_to"| Docker
    Docker -->|"container_deploy"| DAPI

    E2EPlay -.->|"tests_frontend"| FVue
    E2EPlay -.->|"tests_backend"| DAPI
    E2EPup -.->|"tests_frontend"| FVue
    E2EPup -.->|"tests_backend"| DAPI

    %% Click Events
    click DAPI "https://github.com/dandi/dandi-archive/tree/master/dandiapi"
    click AApp "https://github.com/dandi/dandi-archive/tree/master/dandiapi/analytics"
    click APIApp "https://github.com/dandi/dandi-archive/tree/master/dandiapi/api"
    click SApp "https://github.com/dandi/dandi-archive/tree/master/dandiapi/search"
    click ZApp "https://github.com/dandi/dandi-archive/tree/master/dandiapi/zarr"
    click Celery "https://github.com/dandi/dandi-archive/blob/master/dandiapi/celery.py"
    click FVue "https://github.com/dandi/dandi-archive/tree/master/web"
    click FComp "https://github.com/dandi/dandi-archive/tree/master/web/src/components"
    click FViews "https://github.com/dandi/dandi-archive/tree/master/web/src/views"
    click CICD "https://github.com/dandi/dandi-archive/tree/master/.github/workflows"
    click Docker "https://github.com/dandi/dandi-archive/tree/master/dev"
    click Docker "https://github.com/dandi/dandi-archive/blob/master/docker-compose.yml"
    click Docker "https://github.com/dandi/dandi-archive/blob/master/docker-compose.override.yml"
    click E2EPlay "https://github.com/dandi/dandi-archive/tree/master/e2e/playwright"
    click E2EPup "https://github.com/dandi/dandi-archive/tree/master/e2e/puppeteer"

    %% Styling classes
    classDef frontend fill:#a2d2ff,stroke:#000,stroke-width:1px;
    classDef backend fill:#ffafcc,stroke:#000,stroke-width:1px;
    classDef async fill:#cdb4db,stroke:#000,stroke-width:1px;
    classDef database fill:#bde0fe,stroke:#000,stroke-width:1px;
    classDef deploy fill:#ffc8dd,stroke:#000,stroke-width:1px;
    classDef test fill:#dcedc1,stroke:#000,stroke-width:1px;
```
