How usnistgov/oar-dist-service works
Project type and purpose
• This is a backend-only, full-stack REST Web service implemented in Java using Spring Boot.
• Its purpose is to provide stable, long-lived URLs for data products stored in local disk or AWS S3, to cache them for fast access, and to expose three main operations over HTTP: single-file download, bundle-plan generation, and bundle download.
High-level architecture inferred from file structure and README
• Layered architecture following a Spring MVC pattern:
• API/Controller Layer (src/main/java/gov/nist/oar/distrib/web)
• Service Layer (src/main/java/gov/nist/oar/distrib/service)
• Cache Management Layer (src/main/java/gov/nist/oar/distrib/cachemgr)
• Packaging & Distribution Layer (src/main/java/gov/nist/oar/distrib/datapackage & distrib)
• Storage Layer (src/main/java/gov/nist/oar/distrib/storage)
• External/Client Integrations (src/main/java/gov/nist/oar/clients and service/rpa)
• Persistence for inventory/checks (SQLite or JDBC in cachemgr/inventory)
• Infrastructure & testing (Docker, s3mock, Maven, CI workflows)
Main components
a. API/Controller Layer
• HTTP endpoints (VersionController, DatasetAccessController, BundleDownloadPlanController, RPARequestHandlerController, etc.)
• Security filters (BearerTokenAuthenticationFilter, JwtTokenValidator)
• Exception handlers and request sanitizers
b. Service Layer
• PreservationBagService: generate or validate preservation bags
• DataPackagingService: plan and assemble bundle plans
• FileDownloadService variants: direct, Nerdm-driven, cache-enabled
• RPADatasetCacher & RPARequestHandler: integrate with external RPA system (Salesforce API, Recaptcha)
c. Cache Management Layer
• Core: CacheManager interface and implementations (BasicCacheManager, ConfigurableCache)
• CacheObjects, ExpiryCheck, IntegrityMonitor
• StorageInventoryDB (JDBC or SQLite) to track cached volumes
• VolumeConfig, DeletionPlanner, SelectionStrategy
d. Storage Layer
• LongTermStorage interface and implementations:
• FilesystemLongTermStorage
• AWSS3LongTermStorage (using AWSS3ClientProvider)
• WebLongTermStorage (proxy or redirect)
• Cache volumes: AWSS3CacheVolume, FilesystemCacheVolume
e. External Components
• AWS S3 bucket(s) for data and cache volumes
• NIST Public Data Repository (PDR) for original data
• RMM service for resource resolution
• Recaptcha service and Salesforce APIs for RPA flows
f. Infrastructure & DevOps
• Docker images for build and test (docker/build-test, scripts/testall.docker)
• s3mock for local S3 simulation in tests
• Maven as the build system (pom.xml)
• CI pipelines (.github/workflows, Travis)
Relationships and data flows
• Client HTTP request → Controller
→ Security filter & request sanitizer
→ Service layer method
→ If downloading a file: check cache through CacheManager
→ Cache hit: serve directly from cache volume
→ Cache miss: fetch from LongTermStorage (S3 or FS), store in cache, then serve
→ If bundle plan: compute file sizes, return plan JSON
→ If RPA request: call external RPARequestHandler → remote systems → cache/download
→ Controller assembles HTTP response and logs request metrics
Architectural patterns & design principles
• Layered separation (Controller / Service / Persistence)
• Strategy pattern for cache deletion, volume selection, storage provider
• Dependency injection of implementations via Spring Boot (@Configuration classes)
• Fail-safe and mockable components (s3mock, NullCacheVolume) for testing
• Use of lightweight embedded DB (SQLiteStorageInventoryDB) or pluggable JDBC
Technologies and frameworks to annotate
• Spring Boot (REST controllers, filters, dependency injection)
• Maven for build management
• Docker & s3mock for containerized build/test
• AWS SDK for Java (S3 client)
• SQLite or any JDBC-compatible inventory DB
• Logback for logging (logback.xml)