# radical-collaboration/facts architecture

> Architecture diagram of the GitHub repository radical-collaboration/facts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/radical-collaboration/facts
- Repository: https://github.com/radical-collaboration/facts

## Overview

##### Project Type and Purpose
- FACTS is an open-source, modular framework written in Python for sea-level projection (global mean, regional, extreme) that supports exploration of deep uncertainty in climate processes.
- It follows a plugin-style, pipeline-driven architecture built on the Radical-PILOT stack to dispatch computational modules to heterogeneous resources.
- Users configure “experiments” via YAML, then invoke a workflow engine (runFACTS.py/FACTS.py) that loads modules, reads data, runs preprocess→fit→postprocess→project stages, and aggregates outputs.

##### File Structure Analysis
- Top-level:
- runFACTS.py / FACTS.py: CLI entrypoints
- docker/: Docker image and develop script
- environment.yml, requirements.txt: conda/pip dependencies
- docs/, notebooks/: user documentation and analysis notebooks
- resources/: resource configuration per cluster or localhost
- experiments/: experiment definitions (config.yml + location lists)
- input_files/, modules-data/: static data repositories
- modules/: hundreds of subdirectories, each representing a physical process (e.g., ipccar5/glaciers, deconto16/AIS, fair/temperature, etc.). Each submodule has:
– preprocess.py, fit.py, postprocess.py, project.py
– pipeline.yml / pipeline.global.yml
– AssignFP.py, ReadFingerprint.py, read_locationfile.py
- Patterns:
- Plugin pattern: each process is a self-contained module adhering to the same interface
- Pipeline orchestration via YAML manifests
- Radical-PILOT dispatch of module tasks to remote resources

##### Key README Insights
- Emphasizes modularity (simple vs. complex models) and dispatch on appropriate resources.
- References Radical-PILOT computing stack.
- Encourages characterization of parametric & structural uncertainties by swapping modules.

##### Components
a. Main Components
- CLI/Workflow Launcher (runFACTS.py, FACTS.py)
- Configuration Manager (YAML parser loading experiments/, resources/)
- Pipeline Orchestrator (engine reading pipeline.yml files)
- Module Registry (plugins in modules/*)
- Radical-PILOT Resource Manager
- Data Repositories:
– Static Input (input_files/, modules-data/)
– Experiment Inputs (experiments/*)
– Intermediate Data (per-module working directories)
– Final Outputs (aggregated projections)
- User Interfaces:
– Documentation (docs/)
– Notebooks (analysis & visualization)
- Containerization Layer (docker/Dockerfile)

b. Relationships and Data Flow
1. User invokes CLI → loads resources/*.yml + experiments/<id>/config.yml
2. CLI → Pipeline Orchestrator reads module pipelines (pipeline.yml)
3. For each module step (preprocess→fit→postprocess→project):
– Module Registry provides code
– Orchestrator packages task, sends to Radical-PILOT
– Radical-PILOT schedules execution on HPC/cluster
– Module reads static & experiment inputs, writes intermediate outputs
4. Orchestrator collects outputs, passes to next stage
5. Final projection results written to outputs folder
6. Notebooks read outputs for visualization
7. Documentation describes architecture + usage

c. Architectural Patterns & Principles
- Modular plugin architecture with uniform interfaces
- Pipeline-as-code: pipeline.yml drives the workflow
- Separation of concerns: configuration, compute, data storage, presentation
- Distributed execution via a resource manager (Radical-PILOT)
- Reproducibility through environment.yml, Docker

d. Key Technologies & Frameworks
- Python 3 scripts & modules
- YAML for pipeline & experiment configuration
- Radical-PILOT for distributed task scheduling
- Conda/pip environment management
- Docker for containerized development
- Sphinx/ReadTheDocs for documentation
- Jupyter notebooks for result exploration

##### Detailed Capture of Architectural Elements
- Hierarchy: CLI → orchestrator → per-module pipeline stages
- How pipeline.yml files live next to module code and drive behavior
- Resource selection via resources/*.yml
- Containerization layer enveloping the CLI and orchestrator if using Docker

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine
    subgraph "Containerization Layer" 
        direction TB
        Dockerfile["Dockerfile"]:::external
        develop_sh["develop.sh"]:::external
    end
    subgraph "Core Engine" 
        direction TB
        CLI["CLI / Workflow Launcher"]:::core
        Config["Configuration Manager"]:::core
        Orchestrator["Pipeline Orchestrator"]:::core
    end
    subgraph "Plugin Boundary" 
        direction TB
        Registry["Module Registry (modules/)"]:::modules
    end
    subgraph "Execution" 
        direction TB
        RPilot["Radical-PILOT Resource Manager"]:::external
        ModuleTasks["Module Tasks"]:::modules
    end
    subgraph "Data Stores" 
        direction TB
        StaticInput["Static Input Data"]:::datastore
        ExperimentInput["Experiment Inputs"]:::datastore
        Intermediate["Intermediate Data"]:::datastore
        FinalOutput["Final Outputs"]:::datastore
    end
    subgraph "User Interfaces" 
        direction TB
        Docs["Documentation"]:::external
        Notebooks["Notebooks"]:::external
    end
    subgraph "Environment Management" 
        direction TB
        Env1["environment.yml"]:::core
        Env2["requirements.txt"]:::core
        Env3["facts_requirments.txt"]:::core
        Scripts["scripts/index-modules-data.sh"]:::core
    end

    %% Connections
    Dockerfile -->|contains| CLI
    develop_sh -->|sets up| CLI
    CLI -->|"loads configs"| Config
    Config -->|"provides settings"| Orchestrator
    Orchestrator -->|"reads pipelines"| Registry
    Orchestrator -->|"dispatch tasks"| RPilot
    RPilot -->|"schedules execution"| ModuleTasks
    ModuleTasks -->|"reads/writes"| StaticInput
    ModuleTasks -->|"reads/writes"| ExperimentInput
    ModuleTasks -->|"writes"| Intermediate
    Intermediate -->|"collected by"| Orchestrator
    Orchestrator -->|"aggregates results"| FinalOutput
    FinalOutput -->|"visualized by"| Notebooks
    FinalOutput -->|"documented in"| Docs

    %% Click Events
    click CLI "https://github.com/radical-collaboration/facts/blob/main/runFACTS.py"
    click CLI "https://github.com/radical-collaboration/facts/blob/main/FACTS.py"
    click Config "https://github.com/radical-collaboration/facts/blob/main/resources/resource.yml"
    click Config "https://github.com/radical-collaboration/facts/blob/main/resources/resource_localhost.yml"
    click Config "https://github.com/radical-collaboration/facts/blob/main/resources/resource_remote_example.yml"
    click Config "https://github.com/radical-collaboration/facts/tree/main/experiments/"
    click Orchestrator "https://github.com/radical-collaboration/facts/blob/main/modules/facts/total/total_workflow.py"
    click Registry "https://github.com/radical-collaboration/facts/tree/main/modules/"
    click RPilot "https://github.com/radical-collaboration/facts/blob/main/resources/resource_remote_example.yml"
    click StaticInput "https://github.com/radical-collaboration/facts/tree/main/input_files/"
    click StaticInput "https://github.com/radical-collaboration/facts/tree/main/modules-data/"
    click ExperimentInput "https://github.com/radical-collaboration/facts/tree/main/experiments/"
    click Dockerfile "https://github.com/radical-collaboration/facts/tree/main/docker/Dockerfile"
    click develop_sh "https://github.com/radical-collaboration/facts/blob/main/docker/develop.sh"
    click Docs "https://github.com/radical-collaboration/facts/tree/main/docs/"
    click Docs "https://github.com/radical-collaboration/facts/tree/main/docs/source/"
    click Notebooks "https://github.com/radical-collaboration/facts/tree/main/notebooks/"
    click Notebooks "https://github.com/radical-collaboration/facts/tree/main/notebooks/GMD_Paper_Notebooks/"
    click Notebooks "https://github.com/radical-collaboration/facts/tree/main/notebooks/P-box/"
    click Env1 "https://github.com/radical-collaboration/facts/blob/main/environment.yml"
    click Env2 "https://github.com/radical-collaboration/facts/blob/main/requirements.txt"
    click Env3 "https://github.com/radical-collaboration/facts/blob/main/facts_requirments.txt"
    click Scripts "https://github.com/radical-collaboration/facts/blob/main/scripts/index-modules-data.sh"

    %% Styles
    classDef core fill:#BEE3F8,stroke:#333,stroke-width:1px;
    classDef modules fill:#C6F6D5,stroke:#333,stroke-width:1px;
    classDef datastore fill:#FBD38D,stroke:#333,stroke-width:1px;
    classDef external fill:#D6BCFA,stroke:#333,stroke-width:1px;
```
