# usnistgov/oar-dist-service architecture

> Architecture diagram of the GitHub repository usnistgov/oar-dist-service, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/usnistgov/oar-dist-service
- Repository: https://github.com/usnistgov/oar-dist-service

## Overview

##### Project type and purpose
- This is a backend-only, full-stack REST Web service implemented in Java using Spring Boot.
- Its purpose is to provide stable, long-lived URLs for data products stored in local disk or AWS S3, to cache them for fast access, and to expose three main operations over HTTP: single-file download, bundle-plan generation, and bundle download.

##### High-level architecture inferred from file structure and README
- Layered architecture following a Spring MVC pattern:
- API/Controller Layer (src/main/java/gov/nist/oar/distrib/web)
- Service Layer (src/main/java/gov/nist/oar/distrib/service)
- Cache Management Layer (src/main/java/gov/nist/oar/distrib/cachemgr)
- Packaging & Distribution Layer (src/main/java/gov/nist/oar/distrib/datapackage & distrib)
- Storage Layer (src/main/java/gov/nist/oar/distrib/storage)
- External/Client Integrations (src/main/java/gov/nist/oar/clients and service/rpa)
- Persistence for inventory/checks (SQLite or JDBC in cachemgr/inventory)
- Infrastructure & testing (Docker, s3mock, Maven, CI workflows)

##### Main components
a. API/Controller Layer
- HTTP endpoints (VersionController, DatasetAccessController, BundleDownloadPlanController, RPARequestHandlerController, etc.)
- Security filters (BearerTokenAuthenticationFilter, JwtTokenValidator)
- Exception handlers and request sanitizers

b. Service Layer
- PreservationBagService: generate or validate preservation bags
- DataPackagingService: plan and assemble bundle plans
- FileDownloadService variants: direct, Nerdm-driven, cache-enabled
- RPADatasetCacher & RPARequestHandler: integrate with external RPA system (Salesforce API, Recaptcha)

c. Cache Management Layer
- Core: CacheManager interface and implementations (BasicCacheManager, ConfigurableCache)
- CacheObjects, ExpiryCheck, IntegrityMonitor
- StorageInventoryDB (JDBC or SQLite) to track cached volumes
- VolumeConfig, DeletionPlanner, SelectionStrategy

d. Storage Layer
- LongTermStorage interface and implementations:
- FilesystemLongTermStorage
- AWSS3LongTermStorage (using AWSS3ClientProvider)
- WebLongTermStorage (proxy or redirect)
- Cache volumes: AWSS3CacheVolume, FilesystemCacheVolume

e. External Components
- AWS S3 bucket(s) for data and cache volumes
- NIST Public Data Repository (PDR) for original data
- RMM service for resource resolution
- Recaptcha service and Salesforce APIs for RPA flows

f. Infrastructure & DevOps
- Docker images for build and test (docker/build-test, scripts/testall.docker)
- s3mock for local S3 simulation in tests
- Maven as the build system (pom.xml)
- CI pipelines (.github/workflows, Travis)

##### Relationships and data flows
- Client HTTP request → Controller
→ Security filter & request sanitizer
→ Service layer method
→ If downloading a file: check cache through CacheManager
→ Cache hit: serve directly from cache volume
→ Cache miss: fetch from LongTermStorage (S3 or FS), store in cache, then serve
→ If bundle plan: compute file sizes, return plan JSON
→ If RPA request: call external RPARequestHandler → remote systems → cache/download
→ Controller assembles HTTP response and logs request metrics

##### Architectural patterns & design principles
- Layered separation (Controller / Service / Persistence)
- Strategy pattern for cache deletion, volume selection, storage provider
- Dependency injection of implementations via Spring Boot (@Configuration classes)
- Fail-safe and mockable components (s3mock, NullCacheVolume) for testing
- Use of lightweight embedded DB (SQLiteStorageInventoryDB) or pluggable JDBC

##### Technologies and frameworks to annotate
- Spring Boot (REST controllers, filters, dependency injection)
- Maven for build management
- Docker & s3mock for containerized build/test
- AWS SDK for Java (S3 client)
- SQLite or any JDBC-compatible inventory DB
- Logback for logging (logback.xml)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% API/Controller Layer
    subgraph "API / Controller Layer"
        direction TB
        Controllers["HTTP Controllers\n(src/main/java/gov/nist/oar/distrib/web)"]:::api
        SecurityFilters["Security Filters\n(BearerTokenAuthenticationFilter,\nJwtTokenValidator)"]:::security
        ExceptionHandlers["Exception Handlers\n(Request Sanitizers)"]:::api
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        CoreServices["Core Services\n(src/main/java/gov/nist/oar/distrib/service)"]:::service
        RPAServices["RPA Services\n(src/main/java/gov/nist/oar/distrib/service/rpa)"]:::service
    end

    %% Packaging & Distribution
    subgraph "Packaging & Distribution"
        direction TB
        DataPackage["Data Packaging\n(src/main/java/gov/nist/oar/distrib/datapackage)"]:::service
        DistribCore["Distrib Core Types\n(src/main/java/gov/nist/oar/distrib)"]:::service
    end

    %% Cache Management Layer
    subgraph "Cache Management Layer"
        direction TB
        CacheCore["CacheCore Interfaces\n(src/main/java/gov/nist/oar/distrib/cachemgr)"]:::cache
        Inventory["Inventory Persistence\n(src/main/java/gov/nist/oar/distrib/cachemgr/inventory)"]:::cache
        PDRCache["PDR Cache\n(src/main/java/gov/nist/oar/distrib/cachemgr/pdr)"]:::cache
        RestoreStrat["Restore Strategies\n(src/main/java/gov/nist/oar/distrib/cachemgr/restore)"]:::cache
        SimpleCache["Simple Cache Impl\n(src/main/java/gov/nist/oar/distrib/cachemgr/simple)"]:::cache
        CacheVolumes["Cache Volumes\n(src/main/java/gov/nist/oar/distrib/cachemgr/storage)"]:::cache
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        LongTermStorage["LongTermStorage Interface\n(src/main/java/gov/nist/oar/distrib/storage)"]:::storage
        FSStorage["FilesystemLongTermStorage"]:::storage
        S3Storage["AWSS3LongTermStorage"]:::storage
        WebStorage["WebLongTermStorage"]:::storage
    end

    %% Client Integrations
    subgraph "Client Integrations"
        direction TB
        GenericClient["Generic Client Resolver\n(src/main/java/gov/nist/oar/clients)"]:::external
        RMMClient["RMM Resolver\n(src/main/java/gov/nist/oar/clients/rmm)"]:::external
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        AWS_S3["AWS S3 Bucket"]:::external
        NIST_PDR["NIST Public Data Repo"]:::external
        RMM["RMM Service"]:::external
        Recaptcha["Recaptcha Service"]:::external
        Salesforce["Salesforce API"]:::external
    end

    %% Infrastructure & DevOps
    subgraph "Infrastructure & DevOps"
        direction TB
        Docker["Docker Build/Test\n(docker/build-test)"]:::infra
        S3Mock["Local S3 Mock\n(s3mock)"]:::infra
        CI["CI/CD Workflows\n(.github/workflows)"]:::infra
        Maven["Maven POM\n(pom.xml)"]:::infra
        Logback["Logging Config\n(src/main/resources/logback.xml)"]:::infra
    end

    %% Relationships
    Client["Client"]:::external -->|"HTTP Request"| Controllers
    Controllers -->|authn/authz| SecurityFilters
    Controllers -->|sanitize| ExceptionHandlers
    Controllers -->|invoke| CoreServices
    Controllers -->|invoke| RPAServices

    CoreServices -->|plan/download| DataPackage
    CoreServices -->|use types| DistribCore

    CoreServices -->|cache lookup| CacheCore
    CacheCore -->|track| Inventory
    CacheCore -->|pdr logic| PDRCache
    CacheCore -->|restore| RestoreStrat
    CacheCore -->|simple impl| SimpleCache
    CacheCore -->|volumes| CacheVolumes

    CacheVolumes -->|store/retrieve| FSStorage
    CacheVolumes -->|store/retrieve| S3Storage

    CoreServices -->|long-term fetch| LongTermStorage
    LongTermStorage -->|FS API| FSStorage
    LongTermStorage -->|S3 API| S3Storage
    LongTermStorage -->|HTTP| WebStorage

    RPAServices -->|resolve| GenericClient
    RPAServices -->|resolve| RMMClient
    RPAServices -->|call| Salesforce
    RPAServices -->|verify| Recaptcha

    GenericClient -->|HTTP| NIST_PDR
    RMMClient -->|HTTP| RMM

    CoreServices -->|upload/cache| AWS_S3
    CacheVolumes -->|cache buckets| AWS_S3

    %% Infrastructure
    CoreServices -->|built/tested in| Docker
    CoreServices -->|tested against| S3Mock
    Maven -->|build| CoreServices
    CI -->|pipeline| Docker

    %% Click Events
    click Controllers "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/web"
    click CoreServices "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/service"
    click RPAServices "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/service/rpa"
    click CacheCore "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr"
    click Inventory "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr/inventory"
    click PDRCache "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr/pdr"
    click RestoreStrat "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr/restore"
    click SimpleCache "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr/simple"
    click CacheVolumes "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/cachemgr/storage"
    click DataPackage "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/datapackage"
    click DistribCore "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib"
    click LongTermStorage "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/distrib/storage"
    click GenericClient "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/clients"
    click RMMClient "https://github.com/usnistgov/oar-dist-service/tree/integration/src/main/java/gov/nist/oar/clients/rmm"
    click Inventory "https://github.com/usnistgov/oar-dist-service/blob/integration/src/main/resources/gov/nist/oar/distrib/cachemgr/inventory/res/sqlite_create.sql"
    click Docker "https://github.com/usnistgov/oar-dist-service/tree/integration/docker/build-test"
    click S3Mock "https://github.com/usnistgov/oar-dist-service/tree/integration/s3mock"
    click CI "https://github.com/usnistgov/oar-dist-service/tree/integration/.github/workflows"
    click Maven "https://github.com/usnistgov/oar-dist-service/blob/integration/pom.xml"
    click Logback "https://github.com/usnistgov/oar-dist-service/blob/integration/src/main/resources/logback.xml"

    %% Styles
    classDef api fill:#E3F2FD,stroke:#42A5F5,color:#0D47A1
    classDef service fill:#E8F5E9,stroke:#66BB6A,color:#1B5E20
    classDef cache fill:#FFF3E0,stroke:#FFB74D,color:#E65100
    classDef storage fill:#F3E5F5,stroke:#AB47BC,color:#4A148C
    classDef external fill:#FFEBEE,stroke:#E57373,color:#B71C1C
    classDef infra fill:#ECEFF1,stroke:#78909C,color:#263238
    classDef security fill:#FFCDD2,stroke:#E53935,color:#B71C1C
```
