# ibm/terramind architecture

> Architecture diagram of the GitHub repository ibm/terramind, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibm/terramind
- Repository: https://github.com/ibm/terramind

## Overview

##### Project Type and Purpose
- TerraMind is an open-source any-to-any generative foundation model for Earth Observation, delivered as:
- A fine-tuning and generation toolkit (TerraTorch CLI + Python library)
- Example notebooks and configuration files for training and inference
- Pre-trained TerraMind backbones hosted on Hugging Face
- It is not a monolithic web service but a modular ML toolkit based on PyTorch Lightning and Diffusers.

##### File-Structure Analysis
- Top level:
- configs/ : Lightning YAML configs for different datasets and “Thinking-in-Modalities” variants
- examples/ : sample GeoTIFF inputs for DEM, LULC, NDVI, Sentinel-1/2 modalities
- notebooks/ : Jupyter workflows for fine-tuning, generation, tokenizer demos
- README.md : project overview, setup, usage
- No explicit “src/” directory: core logic lives in the external TerraTorch library, imported via pip.

##### README Insights
- TerraMind integrates with TerraTorch CLI commands: `terratorch fit`, `terratorch test`
- Two fine-tuning modes: standard and Thinking-in-Modalities (TiM)
- Uses Hugging Face diffusers (v0.30.0) for generation, six tokenizers for multimodal inputs
- Example datasets handled by generic DataModules (`GenericMultiModalDataModule`, `GenericNonGeoSegmentationDataModule`)

##### Components
a. User Layer
- CLI Interface: `terratorch` commands
- Jupyter Notebooks
b. Configuration Layer
- YAML Config Files (configs/*.yaml)
c. Core Library Layer (TerraTorch)
- Entry-point: CLI parser → constructs Trainer + DataModule + Model
- Data Modules: multimodal vs. single-modal
- Model Backbones: TerraMind base/large/tim variants
- Tokenizers: six modality-specific tokenizers
- Trainer: PyTorch Lightning + loggers (TensorBoard)
- Generation Engine: Hugging Face Diffusers
d. Data Layer
- Local File System: examples/*.tif or user datasets
- External Hubs: Hugging Face Model Hub for pre-trained weights
e. Output Layer
- Model Checkpoints (.ckpt)
- Generated imagery outputs
- Logs & Metrics (TensorBoard)

##### Interactions and Data Flows
- CLI/Notebook → loads YAML config
- Config → instantiates DataModule → reads TIFFs
- Config → selects Model Backbone → loads pre-trained weights from HF
- DataModule + Model → passed into Lightning Trainer
- Trainer ↔ Tokenizer (during pre-processing and generation)
- Trainer → writes checkpoints, logs to TensorBoard
- `terratorch test` or generation notebooks → use model + diffusers to produce new modalities
- Generated outputs can be visualized in notebooks or saved to disk

##### Architectural Patterns
- Plugin-style DataModule abstraction for new datasets
- YAML-driven CLI orchestrating all components
- Separation of concerns: config files, data ingestion, model definition, training loop, generation pipeline
- TiM approach: a sub-component of the backbone that predicts intermediate modalities

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Layer"
        CLI["CLI Interface\n(`terratorch fit`/`terratorch test`)"]:::ui
        Notebooks["Jupyter Notebooks"]:::ui
    end

    subgraph "Configuration Layer"
        Config["YAML Config Files"]:::data
    end

    subgraph "Core Library Layer"
        CLIparser["CLI Parser"]:::compute
        DataModule["DataModule\n(multimodal/single)"]:::compute
        Model["Model Backbones\n(base/large/TiM)"]:::compute
        Tokenizer["Tokenizers\n(6 modalities)"]:::compute
        Trainer["Lightning Trainer\n(PyTorch Lightning)"]:::compute
        Generation["Generation Engine\n(Hugging Face Diffusers)"]:::compute
    end

    subgraph "Data Layer"
        LocalFS["Local File System\n(GeoTIFF inputs)"]:::data
        HFModels["Hugging Face Model Hub"]:::external
    end

    subgraph "Output Layer"
        Checkpoints["Model Checkpoints\n(.ckpt)"]:::data
        Outputs["Generated Imagery"]:::data
        Logs["Logs & Metrics\n(TensorBoard)"]:::data
    end

    CLI -->|"invokes"| CLIparser
    Notebooks -->|"executes"| CLIparser
    CLIparser -->|"loads"| Config
    Config -->|"instantiates"| DataModule
    Config -->|"selects backbone"| Model
    DataModule -->|"reads TIFF"| LocalFS
    Model -->|"loads weights"| HFModels
    DataModule -->|"provides data"| Trainer
    Model -->|"provides model"| Trainer
    Trainer -->|"uses tokenizer"| Tokenizer
    Trainer -->|"trains & logs"| Logs
    Trainer -->|"saves checkpoint"| Checkpoints
    Notebooks -->|"generates via"| Generation
    Model -->|"used by"| Generation
    Generation -->|"produces"| Outputs
    Outputs -->|"visualized in"| Notebooks

    click CLI "https://github.com/ibm/terramind/blob/main/README.md"
    click Notebooks "https://github.com/ibm/terramind/tree/main/notebooks/"
    click Config "https://github.com/ibm/terramind/tree/main/configs/"
    click LocalFS "https://github.com/ibm/terramind/tree/main/examples/"

    classDef ui fill:#FFA500,stroke:#333, color:#000
    classDef compute fill:#ADD8E6,stroke:#333, color:#000
    classDef data fill:#ccffcc,stroke:#333, color:#000
    classDef external fill:#DDA0DD,stroke:#333, color:#000
```
