# metoffice/cdds architecture

> Architecture diagram of the GitHub repository metoffice/cdds, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/metoffice/cdds
- Repository: https://github.com/metoffice/cdds

## Overview

##### Main Components
- Front-end (command-line interfaces and user interaction): Represented by the numerous “command_line.py” scripts that are spread throughout modules like archive, clean, configure, convert, extract, inventory, prepare, qc, and request_transform.
- Core Processing Modules:
– Ingestion/Preparation: Modules in “prepare” that build directory structures, create request files and process user variable definitions.
– Conversion: The “convert” module (located in both cdds and the separate mip_convert package) handles transformation of raw climate model data into CMOR-standard formats. This stage involves configuration management, processing pipelines, and mapping generation.
– Quality Control (QC): The “qc” directory organizes checks, validators, and dataset verifications (including plugins for different standards such as cf17, cmip6, and cordex).
– Archiving: The “archive” module deals with publishing, storing, and managing historical states of the processed data.
– Data Request & Inventory: Components under “request” and “inventory” handle parsing of metadata/configuration files and manage inventory/DB interactions.
- Plugin Framework: The “common/plugins” and its subdirectories (e.g., cmip6, cmip6_plus, cordex, gcmodeldev) provide the extensibility required for handling various climate model configurations and data streams.
- Workflow Orchestration: The “workflows” directory contains definitions (with files like “flow.cylc” and rose-app.conf) for the overall orchestration in processing stages such as conversion and processing.
- Configuration Management: Directories and files (config/, setup.cfg, and environment*.yml) support consistent configuration and dependency management across the system.
- Testing: Extensive tests (located primarily under cdds/cdds/tests and mip_convert/mip_convert/tests) ensure the integrity and correct behavior of each component.

##### Relationships and Data Flow
- The system starts with user-supplied configuration and request files that are parsed by “configure” modules.
- Data flows from raw input (model outputs) into the preparation module, then moves into conversion where request parameters and mapping configurations are applied.
- After conversion, quality control checks (using the plugins and QC validators) are run.
- Upon successful QC, data is processed for archiving and publishing via the “archive” module.
- Plugins extend behavior at each step to handle model-specific attributes and mapping configurations.
- Workflows coordinate this progression by invoking various command-line scripts in a defined sequence.

##### Architectural Patterns and Design Principles
- Modular architecture: Each processing stage (prepare, convert, extract, qc, archive) is isolated in its own directory, making the system extendable.
- Plugin-based extensibility: A dedicated plugin framework allows adding support for new models or experiments easily.
- CLI Driven: Many modules expose command-line interfaces, indicating that the system is designed for batch processing and automation.
- Workflow orchestration: Several “rose-app.conf” files and “flow.cylc” definitions suggest a workflow management system (possibly Cylc) is used to manage execution order and dependencies.

##### Relevant Technologies and Frameworks
- Python is the primary language, with heavy use of configuration files (JSON, CFG, etc.) and command-line utilities.
- Use of standardized metadata and mapping files (CMOR, CMIP6 standards) as input for generating outputs.
- Test frameworks for both unit and functional tests, as visible in the tests directories.

##### Design Elements to Incorporate
- Use clear labels for each component and submodule, e.g., “Configuration & Request Parsing”, “Data Preparation”, “Conversion Engine”, “Quality Control”, “Archiving”, “Plugin Loader”, “Workflow Orchestrator”.
- Brief annotations for each component that explain its role in the overall architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% CLI Interfaces Subgraph
  subgraph "CLI Interfaces"
    A1("Archive CLI"):::cli
    A2("Clean CLI"):::cli
    A3("Configure CLI"):::cli
    A4("Convert CLI"):::cli
    A5("Extract CLI"):::cli
    A6("Inventory CLI"):::cli
    A7("Prepare CLI"):::cli
    A8("QC CLI"):::cli
    A9("RequestTransform CLI"):::cli
  end

  %% Processing Pipeline Subgraph
  subgraph "Processing Pipeline"
    B1("Configuration &RequestParsing"):::module
    B2("Data Preparation"):::module
    subgraph "Conversion Modules"
      C1("CDDS Convert"):::module
      C2("MIP Convert"):::module
    end
    B3("Quality Control"):::module
    B4("Archiving &Publishing"):::module
    B5("Request &Inventory"):::module
  end

  %% Supporting Components Subgraph
  subgraph "Supporting Components"
    D1("Plugin Framework"):::plugin
    D2("Workflow Orchestration"):::workflow
    D3("Configuration Management"):::config
    D4("Testing Framework"):::test
  end

  %% Data Flow Connections
  A1 -->|"triggers"| B1
  A2 --> B1
  A3 --> B1
  A4 --> B1
  A5 --> B1
  A6 --> B1
  A7 --> B1
  A8 --> B1
  A9 --> B1

  B1 -->|"prepares"| B2
  B1 -->|"parses"| B5
  B2 -->|"sends data"| C1
  B2 -->|"sends data"| C2
  C1 -->|"processes"| B3
  C2 -->|"processes"| B3
  B3 -->|"validated data"| B4

  %% Plugin connections to core modules
  D1 ---|"extend"| C1
  D1 ---|"extend"| B3
  D1 ---|"extend"| B4

  %% Workflow orchestration connections
  D2 ---|"orchestrates"| B2
  D2 ---|"orchestrates"| C1
  D2 ---|"orchestrates"| C2
  D2 ---|"orchestrates"| B3
  D2 ---|"orchestrates"| B4

  %% Configuration Management supports initial parsing
  D3 -->|"supplies"| B1

  %% Testing Framework (dashed lines indicate testing feedback)
  D4 ---|"tests"| B2
  D4 ---|"tests"| C1
  D4 ---|"tests"| C2
  D4 ---|"tests"| B3
  D4 ---|"tests"| B4

  %% Click Events for CLI Interfaces
  click A1 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/archive/command_line.py"
  click A2 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/clean/command_line.py"
  click A3 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/configure/command_line.py"
  click A4 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/convert/command_line.py"
  click A5 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/extract/command_line.py"
  click A6 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/inventory/command_line.py"
  click A7 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/prepare/command_line.py"
  click A8 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/qc/command_line.py"
  click A9 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/request_transform/command_line.py"

  %% Click Events for Processing Pipeline Components
  click B2 "https://github.com/metoffice/cdds/tree/main/cdds/cdds/prepare"
  click C1 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/convert/command_line.py"
  click C2 "https://github.com/metoffice/cdds/blob/main/mip_convert/mip_convert/command_line.py"
  click B3 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/qc/command_line.py"
  click B4 "https://github.com/metoffice/cdds/blob/main/cdds/cdds/archive/command_line.py"
  click B5 "https://github.com/metoffice/cdds/tree/main/cdds/cdds/configure"

  %% Click Events for Supporting Components
  click D1 "https://github.com/metoffice/cdds/tree/main/cdds/cdds/common/plugins"
  click D2 "https://github.com/metoffice/cdds/tree/main/cdds/cdds/workflows"
  click D3 "https://github.com/metoffice/cdds/tree/main/config"
  click D4 "https://github.com/metoffice/cdds/tree/main/cdds/cdds/tests"

  %% Styles
  classDef cli fill:#FFCCFF,stroke:#000,stroke-width:2px;
  classDef module fill:#CCFFFF,stroke:#000,stroke-width:2px;
  classDef plugin fill:#CCFFCC,stroke:#000,stroke-width:2px;
  classDef workflow fill:#CCCCFF,stroke:#000,stroke-width:2px;
  classDef config fill:#FFF5CC,stroke:#000,stroke-width:2px;
  classDef test fill:#EEEEEE,stroke:#000,stroke-width:2px,stroke-dasharray: 5 5;
```
