# artefactual/archivematica-storage-service architecture

> Architecture diagram of the GitHub repository artefactual/archivematica-storage-service, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/artefactual/archivematica-storage-service
- Repository: https://github.com/artefactual/archivematica-storage-service

## Overview

##### Project Type and Purpose
- This is a full-stack, backend-heavy web application written in Django.
- Purpose: manage storage “locations” (filesystems, S3, GPG‐encrypted, etc.), move files into Archivematica for processing, and replicate AIPs into long-term storage.

##### File Structure
- Top-level “storage_service” Django project:
– settings (base, production, test, auth, CSP, S3 components)
– urls.py, wsgi.py, views.py
- Django apps under src/archivematica/storage_service:
– administration: user/API-key management, admin forms, templates
– common: shared utilities, management commands, GPG helpers, middleware
– locations: REST API (v1/v2), SWORD protocol endpoints, models for each backend, migrations, admin/views/forms/templates
- Static assets: JS/CSS for the front-end UI (Backbone, jQuery).
- install/: gunicorn config, deployment scripts.
- Dockerfile, Makefile, CI workflows, requirements files.

##### README
- Confirms the service is an Archivematica component, exposes a web UI and API for managing storage.
- Mentions long-term preservation workflows, integration with the larger Archivematica ecosystem.

a. External Clients & Services
– Archivematica Dashboard/MCPServer invoking Storage Service APIs
– CLI tools or scripts (e.g., management commands, make_key.py)
– SWORD clients depositing into the service
– OIDC/Shibboleth/Keycloak for authentication
b. Reverse Proxy & WSGI Server
– Nginx (or any front proxy)
– Gunicorn (install/storage-service.gunicorn-config.py)
c. Django Application
– URL router → “administration” app
– URL router → “locations” app (REST v1/v2 & SWORD)
– URL router → static file serving
– Shared “common” library
d. Database
– PostgreSQL/MySQL storing users, API keys, location configs, fixity logs, pipelines, replicas
e. Storage Backends
– Local filesystem
– AWS S3
– Swift, Duracloud, LOCKSS, Arkivum, Dataverse, DSpace, rclone, etc.
– GPG-encrypted staging areas
f. Long-term Storage
– External buckets or archives managed via the above backends
g. Logging & Monitoring
– JSON logging (install/storageService.logging.json)
– Metrics within locations app
h. CI/CD & Testing
– GitHub Actions workflows
– Docker Compose for integration tests

##### Relationships & Data Flow
- Within Django:
– administration app ↔ database (manage API keys, users)
– locations app ↔ database (persist location & pipeline metadata)
– locations app → storage backends (PUT/GET file transfers, fixity checks)
– common utilities invoked by both apps (e.g., GPG encryption, PREMIS logging).

##### Architectural Patterns & Principles
- MVC (Django’s MVI): Models (DB), Views (templates, JSON APIs), Templates (HTML, XML for SWORD).
- Modular apps: separation of concerns (administration vs. locations vs. common).
- Pluggable storage backend pattern: each backend implements a common interface.
- Management commands for batch jobs (import AIP, checksum population).
- Configuration by environment via settings components.

##### Technologies, Frameworks & Libraries
- Python 3, Django
- Gunicorn, Nginx (or other reverse proxy)
- Backbone.js, jQuery for front end
- DRF-style APIs (resources.py, v1/v2 modules)
- SWORD protocol for deposit
- GPG for encryption utilities
- Various storage SDKs (boto3 for S3, python-swiftclient, dataverse client, etc.)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Clients & Services
    subgraph "External Clients & Services"
        AMUI["Archivematica Dashboard UI"]:::client
        MCP["MCPServer"]:::client
        CLI["CLI Tools"]:::client
        SWORDCL["SWORD Clients"]:::client
        AuthSV["Keycloak (OIDC)"]:::auth
    end

    %% Reverse Proxy & WSGI
    subgraph "Reverse Proxy & WSGI"
        NGINX["Nginx"]:::compute
        GUNICORN["Gunicorn"]:::compute
    end
    AMUI -->|HTTP/JSON| NGINX
    MCP -->|HTTP/JSON| NGINX
    SWORDCL -->|SWORD XML| NGINX
    CLI -->|manage.py| GUNICORN
    AuthSV -->|OIDC| NGINX
    NGINX -->|pass HTTP| GUNICORN
    click GUNICORN "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/install/storage-service.gunicorn-config.py"

    %% Django Application
    subgraph "Django Application"
        subgraph "Project Core"
            SETTINGS_BASE["Settings (base)"]:::config
            SETTINGS_LOCAL["Settings (local)"]:::config
            SETTINGS_PROD["Settings (production)"]:::config
            SETTINGS_TEST["Settings (test)"]:::config
            URLS["URL Dispatcher"]:::backend
            WSGI["WSGI Entry Point"]:::backend
            MANAGE["manage.py"]:::backend
        end
        click SETTINGS_BASE "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/settings/base.py"
        click SETTINGS_LOCAL "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/settings/local.py"
        click SETTINGS_PROD "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/settings/production.py"
        click SETTINGS_TEST "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/settings/test.py"
        click URLS "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/urls.py"
        click WSGI "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/storage_service/wsgi.py"
        click MANAGE "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/manage.py"

        subgraph "Apps"
            ADMIN_MODELS["Administration Models"]:::backend
            ADMIN_VIEWS["Administration Views/Forms"]:::backend
            ADMIN_URLS["Administration URLs"]:::backend
            COMMON_UT["Common Utilities"]:::backend
            LOC_API1["Locations API v1/v2"]:::backend
            LOC_SWORD["Locations SWORD Endpoints"]:::backend
            LOC_MODELS["Storage Backend Models"]:::backend
            METRICS["Metrics"]:::backend
        end
        click ADMIN_MODELS "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/administration/models.py"
        click ADMIN_VIEWS "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/administration/views.py"
        click ADMIN_URLS "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/administration/urls.py"
        click COMMON_UT "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/common/"
        click LOC_API1 "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/locations/api/v1.py"
        click LOC_SWORD "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/locations/api/sword/views.py"
        click LOC_MODELS "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/locations/models/"
        click METRICS "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/src/archivematica/storage_service/locations/metrics.py"

        STATIC["Static Assets"]:::frontend
        click STATIC "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/static/"
        subgraph "Templates"
            T_ADMIN["Admin HTML Templates"]:::frontend
            T_SWORD["SWORD XML Templates"]:::frontend
            T_LOC["Locations UI Templates"]:::frontend
        end
        click T_ADMIN "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/templates/administration/"
        click T_SWORD "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/templates/locations/api/sword/"
        click T_LOC "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/src/archivematica/storage_service/templates/locations/"

    end

    %% Internal Flows
    GUNICORN -->|WSGI| WSGI
    WSGI --> URLS
    URLS --> ADMIN_URLS
    URLS --> LOC_API1
    URLS --> LOC_SWORD
    URLS --> STATIC

    ADMIN_URLS --> ADMIN_VIEWS
    LOC_SWORD --> T_SWORD
    LOC_API1 --> LOC_MODELS
    COMMON_UT -->|used by| ADMIN_MODELS
    COMMON_UT -->|used by| LOC_MODELS

    %% Database
    subgraph "Database"
        DB["PostgreSQL/MySQL"]:::db
    end
    ADMIN_MODELS -->|ORM| DB
    LOC_MODELS -->|ORM| DB

    %% Storage Backends
    subgraph "Storage Backends"
        FS["Local Filesystem"]:::storage
        S3["AWS S3"]:::storage
        SWIFT["Swift"]:::storage
        GPG["GPG Encrypted"]:::storage
        OTHERS["Other Backends"]:::storage
    end
    LOC_MODELS -->|PUT/GET| FS
    LOC_MODELS -->|PUT/GET| S3
    LOC_MODELS -->|PUT/GET| SWIFT
    LOC_MODELS -->|PUT/GET| GPG
    LOC_MODELS -->|PUT/GET| OTHERS

    %% Logging & CI/CD
    subgraph "Logging & CI/CD"
        LOGCFG["Logging Config (JSON)"]:::config
        CI["CI/CD Workflows"]:::ci
        DOCKER["Docker & Make"]:::ci
    end
    click LOGCFG "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/install/storageService.logging.json"
    click CI "https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/.github/workflows/test.yml"
    click DOCKER "https://github.com/artefactual/archivematica-storage-service/tree/qa/0.x/Dockerfile"

    DB -->|logs metrics| LOGCFG
    LOC_MODELS -->|emit metrics| LOGCFG
    CI --> DOCKER

    %% Styles
    classDef client fill:#f9c74f,stroke:#000
    classDef compute fill:#90be6d,stroke:#000
    classDef backend fill:#43aa8b,stroke:#000
    classDef frontend fill:#577590,stroke:#000,color:#fff
    classDef db fill:#52b69a,stroke:#000
    classDef storage fill:#4cc9f0,stroke:#000
    classDef config fill:#f94144,stroke:#000,color:#fff
    classDef auth fill:#e76f51,stroke:#000,color:#fff
    classDef ci fill:#adb5bd,stroke:#000
```
