# nii-dg/nii-dg architecture

> Architecture diagram of the GitHub repository nii-dg/nii-dg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nii-dg/nii-dg
- Repository: https://github.com/nii-dg/nii-dg

## Overview

##### Project Type and Purpose
- The project is primarily a Python library intended for research data governance. It centers on three functionalities:
- Schema Definition: Defining metadata schemas and validation rules.
- Packaging: Packaging research data in the RO-Crate format.
- Validation: Validating packaged RO-Crate data using both type checking and advanced, cross-entity validations.
- Additionally, the project provides a REST API (as described in the README and open-api_spec.yml) that exposes these functionalities.
- It is complemented by Docker/Docker Compose for containerized development/deployment, comprehensive testing suites, and CI/CD workflows defined in the .github directory.

##### Analyzing the File Structure
- Core Library (nii_dg directory): Contains the main logic such as API, packaging (ro_crate.py), validation, and utility functions.
- Schema Directories:
- nii_dg/schema: Contains both YAML files (e.g., base.yml, amed.yml, etc.) and corresponding Python modules implementing schema specifics.
- schema: Contains additional context files (JSON-LD contexts), documentation, and scripts that generate contexts and docs.
- API & Examples:
- The open-api_spec.yml defines the REST API interface.
- The sapporo_example directory provides a quick start/use case demonstration.
- Testing and CI:
- tests directory holds unit, functional, and lint/style tests.
- .github/workflows contains configuration for automated testing, linting, type-checking, and deployment.
- Docker files and Compose configurations support containerized execution and development.

- Client or External Systems: Consumers of the REST API (if used).
- REST API Server: Representing the endpoints and how the API requests trigger library functionalities.
- Core Library (Business Logic) – including:
- Schema Definition Module: Where metadata schemas and rules are defined (e.g., nii_dg/schema and schema folders).
- Packaging Component: Where the RO-Crate is generated (e.g., nii_dg/ro_crate.py, entity classes).
- Validation Component: Where type checking and cross-entity validations occur.
- Data Store or External Storage (if applicable): While not explicitly a DB layer, think of file-based RO-Crate outputs and external JSON-LD context references.
- Docker & Deployment Containers: Illustrating how Docker (Dockerfile, compose.yml) encapsulates the runtime environment.

b. Relationships and Interactions:
- Flow for packaging data: from incoming research data → creation of entities (with metadata validation/checks) → generation of RO-Crate JSON-LD.
- Test flows where unit and functional tests validate component behavior.

c. Architectural Patterns and Design Principles:
- Separation of concerns with distinct modules for schema, packaging, and validation.
- Use of external referencing (via JSON-LD @context) to decouple schema definitions from their usage.
- That the project design supports extensibility by allowing different schemas (amed, ginfork, etc.) to be integrated and validated separately.

d. Relevant Technologies and Frameworks:
- Python 3.8+ is used as the programming base.
- Docker and Docker Compose for containerization.
- GitHub Actions for CI/CD.
- REST frameworks (Flask or Waitress) for the API server.
- JSON-LD for the structured metadata representation.
- YAML for schema and configuration management.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Interface
    subgraph "External Interface"
        Client["Client / Example Use Case"]:::external
    end

    %% API Layer
    subgraph "API Layer"
        API["REST API Server"]:::api
        OpenAPI["OpenAPI Spec"]:::api
    end

    %% Core Library
    subgraph "Core Library"
        %% Schema Definition Module
        subgraph "Schema Definition Module"
            NiiSchema["NII Schema (nii_dg/schema)"]:::core
            ExtSchema["External Schema (schema)"]:::core
        end

        %% Packaging Component
        subgraph "Packaging Component"
            PackageEngine["Packaging Engine"]:::core
            PackageExample["Packaging Example"]:::core
        end

        %% Validation Component
        subgraph "Validation Component"
            ValidateEngine["Validation Engine"]:::core
            FuncTest["Functional Tests"]:::test
            UnitTest["Unit Tests"]:::test
        end
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        DockerFiles["Dockerfile(s)"]:::container
        DockerCompose["Docker Compose"]:::container
        CICD["CI/CD Pipeline"]:::cicd
    end

    %% Connections between External Interface and API Layer
    Client -->|"calls"| API

    %% Connections from API Layer to Core Library
    API -->|"triggers"| NiiSchema
    API -->|"triggers"| PackageEngine
    API -->|"triggers"| ValidateEngine
    API -->|"references"| OpenAPI

    %% Connections within Core Library
    NiiSchema -->|"provides schema"| PackageEngine
    NiiSchema -->|"provides schema"| ValidateEngine
    ExtSchema -->|"provides schema"| PackageEngine
    ExtSchema -->|"provides schema"| ValidateEngine
    PackageExample -->|"demonstrates"| PackageEngine
    ValidateEngine -->|"tested by"| FuncTest
    ValidateEngine -->|"tested by"| UnitTest

    %% Connections from Deployment & CI/CD to API Layer and Core Library
    CICD -->|"deploys"| API
    CICD -->|"deploys"| NiiSchema
    CICD -->|"deploys"| PackageEngine
    CICD -->|"deploys"| ValidateEngine
    DockerFiles -->|"containerizes"| API
    DockerCompose -->|"orchestrates"| API

    %% Click Events
    click Client "https://github.com/nii-dg/nii-dg/tree/main/sapporo_example"
    click API "https://github.com/nii-dg/nii-dg/blob/main/nii_dg/api.py"
    click OpenAPI "https://github.com/nii-dg/nii-dg/blob/main/open-api_spec.yml"
    click NiiSchema "https://github.com/nii-dg/nii-dg/tree/main/nii_dg/schema"
    click ExtSchema "https://github.com/nii-dg/nii-dg/tree/main/schema"
    click PackageEngine "https://github.com/nii-dg/nii-dg/blob/main/nii_dg/ro_crate.py"
    click PackageExample "https://github.com/nii-dg/nii-dg/blob/main/sapporo_example/package_ro_crate.py"
    click ValidateEngine "https://github.com/nii-dg/nii-dg/blob/main/nii_dg/check_functions.py"
    click FuncTest "https://github.com/nii-dg/nii-dg/blob/main/tests/functional_test/test_validate_crate.py"
    click UnitTest "https://github.com/nii-dg/nii-dg/blob/main/tests/unit_test/test_utils.py"
    click DockerFiles "https://github.com/nii-dg/nii-dg/tree/main/Dockerfile"
    click DockerCompose "https://github.com/nii-dg/nii-dg/blob/main/compose.yml"
    click CICD "https://github.com/nii-dg/nii-dg/tree/main/.github/workflows"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef api fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef core fill:#B0C4DE,stroke:#000,stroke-width:2px;
    classDef test fill:#FFFACD,stroke:#000,stroke-width:2px;
    classDef container fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef cicd fill:#DDA0DD,stroke:#000,stroke-width:2px;
```
