How radical-collaboration/facts works
Project Type and Purpose
• FACTS is an open-source, modular framework written in Python for sea-level projection (global mean, regional, extreme) that supports exploration of deep uncertainty in climate processes.
• It follows a plugin-style, pipeline-driven architecture built on the Radical-PILOT stack to dispatch computational modules to heterogeneous resources.
• Users configure “experiments” via YAML, then invoke a workflow engine (runFACTS.py/FACTS.py) that loads modules, reads data, runs preprocess→fit→postprocess→project stages, and aggregates outputs.
File Structure Analysis
• Top-level:
• runFACTS.py / FACTS.py: CLI entrypoints
• docker/: Docker image and develop script
• environment.yml, requirements.txt: conda/pip dependencies
• docs/, notebooks/: user documentation and analysis notebooks
• resources/: resource configuration per cluster or localhost
• experiments/: experiment definitions (config.yml + location lists)
• input_files/, modules-data/: static data repositories
• modules/: hundreds of subdirectories, each representing a physical process (e.g., ipccar5/glaciers, deconto16/AIS, fair/temperature, etc.). Each submodule has:
– preprocess.py, fit.py, postprocess.py, project.py
– pipeline.yml / pipeline.global.yml
– AssignFP.py, ReadFingerprint.py, read_locationfile.py
• Patterns:
• Plugin pattern: each process is a self-contained module adhering to the same interface
• Pipeline orchestration via YAML manifests
• Radical-PILOT dispatch of module tasks to remote resources
Key README Insights
• Emphasizes modularity (simple vs. complex models) and dispatch on appropriate resources.
• References Radical-PILOT computing stack.
• Encourages characterization of parametric & structural uncertainties by swapping modules.
Components
a. Main Components
• CLI/Workflow Launcher (runFACTS.py, FACTS.py)
• Configuration Manager (YAML parser loading experiments/, resources/)
• Pipeline Orchestrator (engine reading pipeline.yml files)
• Module Registry (plugins in modules/*)
• Radical-PILOT Resource Manager
• Data Repositories:
– Static Input (input_files/, modules-data/)
– Experiment Inputs (experiments/*)
– Intermediate Data (per-module working directories)
– Final Outputs (aggregated projections)
• User Interfaces:
– Documentation (docs/)
– Notebooks (analysis & visualization)
• Containerization Layer (docker/Dockerfile)
b. Relationships and Data Flow
1. User invokes CLI → loads resources/*.yml + experiments/<id>/config.yml
2. CLI → Pipeline Orchestrator reads module pipelines (pipeline.yml)
3. For each module step (preprocess→fit→postprocess→project):
– Module Registry provides code
– Orchestrator packages task, sends to Radical-PILOT
– Radical-PILOT schedules execution on HPC/cluster
– Module reads static & experiment inputs, writes intermediate outputs
4. Orchestrator collects outputs, passes to next stage
5. Final projection results written to outputs folder
6. Notebooks read outputs for visualization
7. Documentation describes architecture + usage
c. Architectural Patterns & Principles
• Modular plugin architecture with uniform interfaces
• Pipeline-as-code: pipeline.yml drives the workflow
• Separation of concerns: configuration, compute, data storage, presentation
• Distributed execution via a resource manager (Radical-PILOT)
• Reproducibility through environment.yml, Docker
d. Key Technologies & Frameworks
• Python 3 scripts & modules
• YAML for pipeline & experiment configuration
• Radical-PILOT for distributed task scheduling
• Conda/pip environment management
• Docker for containerized development
• Sphinx/ReadTheDocs for documentation
• Jupyter notebooks for result exploration
Detailed Capture of Architectural Elements
• Hierarchy: CLI → orchestrator → per-module pipeline stages
• How pipeline.yml files live next to module code and drive behavior
• Resource selection via resources/*.yml
• Containerization layer enveloping the CLI and orchestrator if using Docker