How dandi/dandi-archive works
Main Components
• Backend (Django API):
▪ Contains multiple Django apps such as “analytics”, “api”, “search”, and “zarr”.
▪ Hosts REST API endpoints, models, views, and business logic.
▪ Uses management commands, migrations, and scheduled tasks (Celery workers as indicated by the “celery.py” file in “dandiapi”).
▪ Includes administrative and operational support (authentication, asset management, DOI generation, audit and embargo services).
• Frontend (Vue Application):
▪ Located in the “web” folder.
▪ Provides the user interface with a component-based structure (e.g., AppBar, DLP components, file browser, search, dashboards).
▪ Communicates with the backend via REST API calls.
• External and Supporting Components:
▪ Asynchronous Task Processing: The Celery integration (dandiapi/celery.py and related tasks) managing background jobs like data ingestion and garbage collection.
▪ Databases: Underlying relational database (likely PostgreSQL) that is interfaced by Django via its ORM.
▪ Storage/Blob Services: Management commands and routines for asset ingestion, embargo handling, and asset paths hint at integration with object storage (e.g., S3 or equivalent).
▪ CI/CD and Deployment: GitHub workflows, Docker Compose files, and environment files (in “dev” and “docker-compose.yml”) indicate containerized deployment, staging, and production environments.
▪ E2E Testing: End-to-end test suites in “e2e/playwright” and “e2e/puppeteer” verifying both API and UI interactions.
▪ External integrations: DOI registration and domain-specific services used for neurophysiology data.
Relationships and Interactions
• The Vue-based frontend makes API calls to the Django backend for data retrieval, authentication, file uploads/downloads, and search.
• Within the Django backend, the “api” app is the central hub:
▪ User requests are routed through Django’s URL configuration, which then dispatches them to the proper view (e.g., for dashboards, asset management, and file operations).
▪ Some endpoints trigger background tasks processed asynchronously by Celery workers.
• The Django apps (“analytics”, “search”, “zarr”) provide modular services. For example, “analytics” might log usage statistics, “search” handles data indexing and querying, and “zarr” manages data formats or bulk ingest processes.
• The database acts as a persistent store for metadata, user data, and audit logs.
• Deployment configuration (Dockerfiles, docker-compose) and GitHub workflows ensure consistent build, test, and deploy routines.
Architectural Patterns and Design Principles
• Modular Design: The separation of concerns into distinct Django apps mirrors the micro-module pattern. Each app (analytics, API, search, zarr) encapsulates related functionalities.
• RESTful API: The system uses Django REST practices, exposing endpoints that the frontend consumes.
• Asynchronous Processing: With Celery tasks handling long-running activities (e.g., asset ingestion and garbage collection), the architecture supports high throughput without blocking the main request-response cycle.
• CI/CD Integration: Automated testing, deployment workflows and Docker containers ensure robust delivery pipelines.
Final Advice
• Communication flow (REST API calls, asynchronous task handling, data persistence).
Happy diagramming!