# esmvalgroup/esmvalcore architecture

> Architecture diagram of the GitHub repository esmvalgroup/esmvalcore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/esmvalgroup/esmvalcore
- Repository: https://github.com/esmvalgroup/esmvalcore

## Overview

This project, ESMValCore, is a Python library that provides the core functionalities for the ESMValTool—a community tool for running diagnostics and evaluating Earth System Models through “recipes”. The package is designed to work with CMIP (Climate Model Intercomparison Project) data and includes modules for configuration, CMOR table handling, ESGF data searching/downloading, preprocessing of climate data (e.g., regridding, unit conversion, deriving new variables), as well as recipe management to orchestrate analysis workflows.

##### Main Components
- CLI/Entry Point: The project exposes the “esmvaltool” command which acts as the primary interface. This component parses user input (e.g., recipes and configurations) and triggers downstream actions.
- Recipe Engine: Under esmvalcore/_recipe and esmvalcore/experimental, the recipe engine is responsible for reading recipe files (YAML), validating them, orchestrating preprocessor steps, and managing the workflow.
- Preprocessing Module: Located in esmvalcore/preprocessor, this module comprises many submodules (e.g., _regrid, _derive, _mask) that apply various processing functions to climate data.
- Data Checker and CMOR Utilities: Within esmvalcore/cmor, there are functionalities to load, validate, and fix data using CMOR (Climate Model Output Rewriter) tables. This is crucial for ensuring data consistency.
- Configuration & Logging: The esmvalcore/config folder contains configuration objects, validators, and logging setup. It allows the system to be flexible based on user customizations.
- ESGF and Data Access: The esmvalcore/esgf module handles the climate data search, download, and metadata management from external ESGF servers.
- Auxiliary (Experimental and Utility Functions): Supporting modules such as esmvalcore/experimental, esmvalcore/iris_helpers, and esmvalcore/typing offer experiments and helper routines.
- Testing and CI/CD Infrastructure: The extensive tests folder and CI configurations (.github workflows, .circleci, etc.) indicate robust automated testing, which while not part of the runtime architecture, is key for ensuring system reliability.

##### Relationships/Interactions
- The CLI/Entry Point (esmvaltool command) interfaces with the Recipe Engine, initiating the workflow.
- The Recipe Engine coordinates calls to the Preprocessing Module to prepare and clean model data.
- Preprocessing functions rely on CMOR utilities to validate and correct the raw data against standard tables.
- ESGF functions may be invoked to download model/observational datasets used as input to preprocessing and recipe execution.
- The Configuration component underpins all of these layers by supplying parameters and logging settings throughout the system.
- External libraries, such as iris for cube handling and dask (via configuration options), are used by the preprocessor and other modules, and should be indicated as external dependencies.

##### Important Architectural Patterns/Design Principles
- Modular design: Each major functionality (e.g., preprocessing, recipe management, configuration, data access) is divided into its own module or sub-package.
- Separation of Concerns: The design clearly separates data validation (CMOR), preprocessing, recipe execution, and configuration management.
- Extensibility: The existence of “experimental” folders and templated recipe outputs shows that the project is built to allow extensions and custom implementations.
- Test-Driven: A large number of tests and integration setups indicate that the system is developed with robust testing in mind.

##### Relevant Technologies and Libraries
- Python is the base language.
- External dependencies include iris (for data cube handling), dask (for parallel computing), and YAML for configuration/recipes.
- Use of CI/CD tools (GitHub Actions, CircleCI, Codacy, etc.) which support automated testing and deployment.
- Documentation generation tools, as evident from the .readthedocs.yaml and extensive doc folders.

##### Elements
– From CLI to Recipe Engine.
– From Recipe Engine to Preprocessing, CMOR, and ESGF modules.
– Core modules (e.g., configuration, recipe engine) versus utility libraries (ESGF, preprocessing).

##### Detail
- Do not shy away from splitting the system into as many individual components as visible from the file tree (e.g., within preprocessing, detail submodules like _derive, _regrid, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Components
    A("CLI/Entry Point"):::core
    B("Configuration & Logging"):::core
    subgraph "Core Modules"
        C("Recipe Engine"):::core
        C1("esmvalcore/_recipe"):::core
        C2("esmvalcore/experimental/recipe.py"):::experimental
        D("Preprocessing Module"):::core
        E("Data Checker & CMOR Utilities"):::core
        F("ESGF & Data Access"):::core
        G("Auxiliary Functions"):::auxil
        G1("esmvalcore/experimental"):::experimental
        G2("esmvalcore/iris_helpers.py"):::util
        G3("esmvalcore/typing.py"):::util
    end

    %% External Dependencies
    subgraph "External Dependencies"
        H("iris library"):::external
        I("dask library"):::external
        J("YAML"):::external
    end

    %% Testing & CI/CD Infrastructure
    subgraph "Testing & CI/CD"
        K("tests"):::testing
        L(".github/workflows"):::testing
        M(".circleci"):::testing
    end

    %% Relationships / Data Flow
    A -->|"triggers"| C
    A -->|"initiates"| B
    B -->|"configures"| C
    B -->|"configures"| D
    B -->|"configures"| E
    B -->|"configures"| F

    C -->|"orchestrates"| D
    C -->|"validates via"| E
    C -->|"fetches data from"| F
    C -->|"leverages"| G

    D -->|"applies checks using"| E
    F -->|"supplies raw data to"| D

    %% External library usage
    D -->|"uses"| H
    D -->|"uses"| I
    F -->|"parses"| J

    %% Click Events for Core Components and Testing
    click A "https://github.com/esmvalgroup/esmvalcore/blob/main/esmvalcore/_main.py"
    click B "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/config"
    click C1 "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/_recipe"
    click C2 "https://github.com/esmvalgroup/esmvalcore/blob/main/esmvalcore/experimental/recipe.py"
    click D "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/preprocessor"
    click E "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/cmor"
    click F "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/esgf"
    click G1 "https://github.com/esmvalgroup/esmvalcore/tree/main/esmvalcore/experimental"
    click G2 "https://github.com/esmvalgroup/esmvalcore/blob/main/esmvalcore/iris_helpers.py"
    click G3 "https://github.com/esmvalgroup/esmvalcore/blob/main/esmvalcore/typing.py"
    click K "https://github.com/esmvalgroup/esmvalcore/tree/main/tests"
    click L "https://github.com/esmvalgroup/esmvalcore/tree/main/.github/workflows"
    click M "https://github.com/esmvalgroup/esmvalcore/blob/main/.circleci"

    %% Class Definitions for Colors
    classDef core fill:#f9e79f,stroke:#b9770e,stroke-width:2px;
    classDef experimental fill:#fadbd8,stroke:#c0392b,stroke-width:2px;
    classDef auxil fill:#d5f5e3,stroke:#239b56,stroke-width:2px;
    classDef util fill:#fcf3cf,stroke:#b7950b,stroke-width:2px;
    classDef external fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px;
    classDef testing fill:#e8e8e8,stroke:#7f8c8d,stroke-dasharray:5 5,stroke-width:2px;
```
