How metoffice/cdds works
Main Components
• Front-end (command-line interfaces and user interaction): Represented by the numerous “command_line.py” scripts that are spread throughout modules like archive, clean, configure, convert, extract, inventory, prepare, qc, and request_transform.
• Core Processing Modules:
– Ingestion/Preparation: Modules in “prepare” that build directory structures, create request files and process user variable definitions.
– Conversion: The “convert” module (located in both cdds and the separate mip_convert package) handles transformation of raw climate model data into CMOR-standard formats. This stage involves configuration management, processing pipelines, and mapping generation.
– Quality Control (QC): The “qc” directory organizes checks, validators, and dataset verifications (including plugins for different standards such as cf17, cmip6, and cordex).
– Archiving: The “archive” module deals with publishing, storing, and managing historical states of the processed data.
– Data Request & Inventory: Components under “request” and “inventory” handle parsing of metadata/configuration files and manage inventory/DB interactions.
• Plugin Framework: The “common/plugins” and its subdirectories (e.g., cmip6, cmip6_plus, cordex, gcmodeldev) provide the extensibility required for handling various climate model configurations and data streams.
• Workflow Orchestration: The “workflows” directory contains definitions (with files like “flow.cylc” and rose-app.conf) for the overall orchestration in processing stages such as conversion and processing.
• Configuration Management: Directories and files (config/, setup.cfg, and environment*.yml) support consistent configuration and dependency management across the system.
• Testing: Extensive tests (located primarily under cdds/cdds/tests and mip_convert/mip_convert/tests) ensure the integrity and correct behavior of each component.
Relationships and Data Flow
• The system starts with user-supplied configuration and request files that are parsed by “configure” modules.
• Data flows from raw input (model outputs) into the preparation module, then moves into conversion where request parameters and mapping configurations are applied.
• After conversion, quality control checks (using the plugins and QC validators) are run.
• Upon successful QC, data is processed for archiving and publishing via the “archive” module.
• Plugins extend behavior at each step to handle model-specific attributes and mapping configurations.
• Workflows coordinate this progression by invoking various command-line scripts in a defined sequence.
Architectural Patterns and Design Principles
• Modular architecture: Each processing stage (prepare, convert, extract, qc, archive) is isolated in its own directory, making the system extendable.
• Plugin-based extensibility: A dedicated plugin framework allows adding support for new models or experiments easily.
• CLI Driven: Many modules expose command-line interfaces, indicating that the system is designed for batch processing and automation.
• Workflow orchestration: Several “rose-app.conf” files and “flow.cylc” definitions suggest a workflow management system (possibly Cylc) is used to manage execution order and dependencies.
Relevant Technologies and Frameworks
• Python is the primary language, with heavy use of configuration files (JSON, CFG, etc.) and command-line utilities.
• Use of standardized metadata and mapping files (CMOR, CMIP6 standards) as input for generating outputs.
• Test frameworks for both unit and functional tests, as visible in the tests directories.
Design Elements to Incorporate
• Use clear labels for each component and submodule, e.g., “Configuration & Request Parsing”, “Data Preparation”, “Conversion Engine”, “Quality Control”, “Archiving”, “Plugin Loader”, “Workflow Orchestrator”.
• Brief annotations for each component that explain its role in the overall architecture.