How artefactual/archivematica-storage-service works
Project Type and Purpose
• This is a full-stack, backend-heavy web application written in Django.
• Purpose: manage storage “locations” (filesystems, S3, GPG‐encrypted, etc.), move files into Archivematica for processing, and replicate AIPs into long-term storage.
File Structure
• Top-level “storage_service” Django project:
– settings (base, production, test, auth, CSP, S3 components)
– urls.py, wsgi.py, views.py
• Django apps under src/archivematica/storage_service:
– administration: user/API-key management, admin forms, templates
– common: shared utilities, management commands, GPG helpers, middleware
– locations: REST API (v1/v2), SWORD protocol endpoints, models for each backend, migrations, admin/views/forms/templates
• Static assets: JS/CSS for the front-end UI (Backbone, jQuery).
• install/: gunicorn config, deployment scripts.
• Dockerfile, Makefile, CI workflows, requirements files.
README
• Confirms the service is an Archivematica component, exposes a web UI and API for managing storage.
• Mentions long-term preservation workflows, integration with the larger Archivematica ecosystem.
a. External Clients & Services
– Archivematica Dashboard/MCPServer invoking Storage Service APIs
– CLI tools or scripts (e.g., management commands, make_key.py)
– SWORD clients depositing into the service
– OIDC/Shibboleth/Keycloak for authentication
b. Reverse Proxy & WSGI Server
– Nginx (or any front proxy)
– Gunicorn (install/storage-service.gunicorn-config.py)
c. Django Application
– URL router → “administration” app
– URL router → “locations” app (REST v1/v2 & SWORD)
– URL router → static file serving
– Shared “common” library
d. Database
– PostgreSQL/MySQL storing users, API keys, location configs, fixity logs, pipelines, replicas
e. Storage Backends
– Local filesystem
– AWS S3
– Swift, Duracloud, LOCKSS, Arkivum, Dataverse, DSpace, rclone, etc.
– GPG-encrypted staging areas
f. Long-term Storage
– External buckets or archives managed via the above backends
g. Logging & Monitoring
– JSON logging (install/storageService.logging.json)
– Metrics within locations app
h. CI/CD & Testing
– GitHub Actions workflows
– Docker Compose for integration tests
Relationships & Data Flow
• Within Django:
– administration app ↔ database (manage API keys, users)
– locations app ↔ database (persist location & pipeline metadata)
– locations app → storage backends (PUT/GET file transfers, fixity checks)
– common utilities invoked by both apps (e.g., GPG encryption, PREMIS logging).
Architectural Patterns & Principles
• MVC (Django’s MVI): Models (DB), Views (templates, JSON APIs), Templates (HTML, XML for SWORD).
• Modular apps: separation of concerns (administration vs. locations vs. common).
• Pluggable storage backend pattern: each backend implements a common interface.
• Management commands for batch jobs (import AIP, checksum population).
• Configuration by environment via settings components.
Technologies, Frameworks & Libraries
• Python 3, Django
• Gunicorn, Nginx (or other reverse proxy)
• Backbone.js, jQuery for front end
• DRF-style APIs (resources.py, v1/v2 modules)
• SWORD protocol for deposit
• GPG for encryption utilities
• Various storage SDKs (boto3 for S3, python-swiftclient, dataverse client, etc.)