How nii-dg/nii-dg works
Project Type and Purpose
• The project is primarily a Python library intended for research data governance. It centers on three functionalities:
• Schema Definition: Defining metadata schemas and validation rules.
• Packaging: Packaging research data in the RO-Crate format.
• Validation: Validating packaged RO-Crate data using both type checking and advanced, cross-entity validations.
• Additionally, the project provides a REST API (as described in the README and open-api_spec.yml) that exposes these functionalities.
• It is complemented by Docker/Docker Compose for containerized development/deployment, comprehensive testing suites, and CI/CD workflows defined in the .github directory.
Analyzing the File Structure
• Core Library (nii_dg directory): Contains the main logic such as API, packaging (ro_crate.py), validation, and utility functions.
• Schema Directories:
• nii_dg/schema: Contains both YAML files (e.g., base.yml, amed.yml, etc.) and corresponding Python modules implementing schema specifics.
• schema: Contains additional context files (JSON-LD contexts), documentation, and scripts that generate contexts and docs.
• API & Examples:
• The open-api_spec.yml defines the REST API interface.
• The sapporo_example directory provides a quick start/use case demonstration.
• Testing and CI:
• tests directory holds unit, functional, and lint/style tests.
• .github/workflows contains configuration for automated testing, linting, type-checking, and deployment.
• Docker files and Compose configurations support containerized execution and development.
• Client or External Systems: Consumers of the REST API (if used).
• REST API Server: Representing the endpoints and how the API requests trigger library functionalities.
• Core Library (Business Logic) – including:
• Schema Definition Module: Where metadata schemas and rules are defined (e.g., nii_dg/schema and schema folders).
• Packaging Component: Where the RO-Crate is generated (e.g., nii_dg/ro_crate.py, entity classes).
• Validation Component: Where type checking and cross-entity validations occur.
• Data Store or External Storage (if applicable): While not explicitly a DB layer, think of file-based RO-Crate outputs and external JSON-LD context references.
• Docker & Deployment Containers: Illustrating how Docker (Dockerfile, compose.yml) encapsulates the runtime environment.
b. Relationships and Interactions:
• Flow for packaging data: from incoming research data → creation of entities (with metadata validation/checks) → generation of RO-Crate JSON-LD.
• Test flows where unit and functional tests validate component behavior.
c. Architectural Patterns and Design Principles:
• Separation of concerns with distinct modules for schema, packaging, and validation.
• Use of external referencing (via JSON-LD @context) to decouple schema definitions from their usage.
• That the project design supports extensibility by allowing different schemas (amed, ginfork, etc.) to be integrated and validated separately.
d. Relevant Technologies and Frameworks:
• Python 3.8+ is used as the programming base.
• Docker and Docker Compose for containerization.
• GitHub Actions for CI/CD.
• REST frameworks (Flask or Waitress) for the API server.
• JSON-LD for the structured metadata representation.
• YAML for schema and configuration management.