# earth-data-kit/edk-sar architecture

> Architecture diagram of the GitHub repository earth-data-kit/edk-sar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/earth-data-kit/edk-sar
- Repository: https://github.com/earth-data-kit/edk-sar

## Overview

##### What this project is (type + purpose)

**edk-sar** is a **Python library / workflow wrapper** around **ISCE2** for running **end-to-end InSAR processing** (coregistration → interferogram generation → downstream analysis like phase/displacement), with a strong emphasis on:
- **Reproducible execution via Docker** (ISCE2 + dependencies)
- **Python orchestration** of ISCE2 steps using “workflows”
- **Post-processing and analysis** in modern Python tooling (**xarray** via a custom accessor, GDAL exports, mapping/visualization)

So the “system” is not a classic web app; it’s a **local/CLI + containerized scientific processing pipeline** that interacts with external data providers and produces files (rasters, stacks, outputs).

- A Python control plane (edk_sar)
- A containerized compute runtime (ISCE2)
- External data sources (Copernicus, USGS)
- Local persistent storage (data/tmp/output directories)
- Analysis & visualization layer (xarray accessor, datashader/folium)
- Export tooling (GDAL/COG)

##### Top-level
- `edk_sar/` → main Python package (core library)
- `edk_sar/dockerfiles/docker-compose.yml` → container runtime definition (ISCE2 environment)
- `requirements.txt` → Python dependency boundary
- `InSAR.md` → likely domain notes (optional reference)

##### edk_sar package
- `edk_sar/__init__.py` → public API surface (e.g., `es.init`, `es.workflows...`)
- `edk_sar/constants.py` → global constants (e.g., wavelength, paths, etc.) to annotate in component notes
- `edk_sar/workflows/` → main pipeline orchestration
- `workflows/base/*`:
- `runner.py` + `helpers.py` → shared orchestration utilities (command execution, path mgmt, env mgmt)
- `create_netrc.sh` → credentials provisioning for download (external services)
- `get_dem.sh`, `get_aux_file.sh` → external data acquisition steps
- `workflows/coregister/*`:
- `generate_run_files.sh`, `execute_run_files.sh`, `runner.py` → staged execution model
- `workflows/interferograms/*`:
- similar pattern to coregister
- `edk_sar/xarray_accessor.py` → `.edk` accessor (analysis API layer on top of xarray)
- `edk_sar/edk_datashader.py` → visualization/rendering component (map tiles / datashader pipeline)

This structure strongly suggests: **Python orchestrates, shell scripts perform discrete ops, ISCE2 runs inside Docker, outputs become rasters that xarray consumes and exports.**

##### “Control plane” (Python library layer)
- imports `edk_sar`
- calls `es.init(".env")`
- calls `es.workflows.coregister.run(slc_path)` and `es.workflows.interferograms.run(slc_path)`
- later uses `xarray` + `.edk` accessor for analysis and export

- **Init/Config** (`__init__.py`, `.env` loading)
- **Workflows Orchestrator** (`edk_sar/workflows/**/runner.py`, `base/runner.py`)
- **ISCE2 Framework Adapter** (`edk_sar/frameworks/isce2.py`)
- **Analysis API** (`xarray_accessor.py`)
- **Visualization/Rendering** (`edk_datashader.py`)

##### “Compute/runtime plane” (Docker + ISCE2)
- references `docker-compose.yml`
- contains **ISCE2 Toolchain** (and likely SNAPHU/unwrapping tools, GDAL inside container depending on setup)

Make it explicit that workflow runners call into Dockerized commands (even if the exact invocation isn’t visible here, README states Docker-based execution is core).

##### Data sources (external dependencies)
- **Copernicus Dataspace** (Sentinel-1 SLC downloads/auth)
- **USGS** (DEM/aux data/auth, depending on what scripts fetch)

Them to “Download/Provision” steps in `workflows/base/*`:
- `create_netrc.sh` (credentials) → enables downloads
- `get_dem.sh`, `get_aux_file.sh` → fetch required supporting datasets

##### Storage (file-based architecture is central)
- `data/` (created by user; likely holds SLCs, stacks, DEMs, geometry)
- `tmp/` (scratch space)
- Output directories (stack, merged, geom_reference, interferograms, etc. — even if not in tree, README hints at paths like `edk_sar/data/stack/merged/geom_reference/`)

- Downloads → `data/`
- ISCE2 processing → reads from `data/` / writes to stack outputs
- xarray accessor → reads rasters/ENVI/VRT/etc. from outputs
- GDAL export → writes GeoTIFF/COG outputs

##### Analysis & export
- `geocode()` depends on geometry rasters (lon/lat VRTs)
- `plot()` depends on folium/datashader stack
- `export()` depends on GDAL (COG LZW compression)

##### `coregister.runner` → `base.helpers/base.runner` → Docker/ISCE2
- reads SLC inputs from `data/`
- writes “coregistered stack” outputs into `data/stack/...`
##### `interferograms.runner` → Docker/ISCE2
- reads coregistered products
- writes interferogram products (`*.int`, coherence, etc.)
##### User / Notebook → `xarray.open_dataarray(...)`
7. `DataArray.edk.geocode()` → uses geometry rasters (`lon.rdr.vrt`, `lat.rdr.vrt`) and GDAL tooling

##### Data acquisition subflow
- `edk_sar.init` / workflow setup → `create_netrc.sh`
→ enables authenticated access to **USGS/Copernicus**
- `get_dem.sh` → downloads DEM → stored in `data/` → used by ISCE2 steps

- **Control calls** (Python → runner → docker exec)
- **Data flow** (external → local storage → ISCE2 → outputs → xarray → exports)

##### Architectural patterns to explicitly show

- **Facade/Adapter**: `edk_sar/frameworks/isce2.py` wraps ISCE2 complexity.
- **Workflow/Orchestration**: `workflows/*/runner.py` coordinate multi-step execution; shell scripts generate/execute run files.
- **Batch pipeline + file-based contracts**: intermediate artifacts on disk are the integration boundary between stages.
- **Reproducible runtime boundary**: Docker isolates ISCE2 dependencies from host Python env.
- **Extension point**: custom `xarray` accessor (`.edk`) is the “user-friendly API” for geocoding/plot/export.

##### “Don’t overthink it” rule applied to this repo (what to split into separate components)
- `coregister` workflow runner
- `interferograms` workflow runner
- `base` runner/helpers
- credential provisioning (`create_netrc.sh`)
- DEM/aux download steps
- ISCE2 adapter layer
- Docker runtime
- external providers (USGS, Copernicus)
- filesystem volumes (data/tmp/output)
- xarray accessor (geocode/plot/export)
- GDAL export dependency
- visualization subsystem (datashader/folium)

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% (1) Context / Container view
  %% =========================
  subgraph "Context: edk-sar end-to-end InSAR processing (local + Docker)"
    direction TB

    User["Data scientist / analyst\n(Python/CLI/Notebook)"]:::actor

    subgraph "Host machine (Python control plane)"
      direction TB

      ExecSH["Convenience runner\nexec.sh"]:::entry

      subgraph "edk_sar (Python package)"
        direction TB
        Pkg["edk_sar"]:::pkg
        Init["Public API + init/config\n(es.init, es.workflows.*)"]:::python
        Consts["Constants / defaults"]:::python

        subgraph "Workflows (orchestration)"
          direction TB
          WFpkg["workflows package API"]:::python

          subgraph "Base workflow utilities"
            direction TB
            BaseRunner["base.runner\n(shared orchestration)"]:::python
            BaseHelpers["base.helpers\n(cmd/path/env utils)"]:::python
            Netrc["create_netrc.sh\n(credentials provisioning)"]:::script
            GetDEM["get_dem.sh\n(DEM download)"]:::script
            GetAUX["get_aux_file.sh\n(aux download)"]:::script
          end

          subgraph "Coregister workflow"
            direction TB
            CoregRunner["coregister.runner\n(stage orchestration)"]:::python
            CoregGen["generate_run_files.sh\n(build runfiles)"]:::script
            CoregExec["execute_run_files.sh\n(run runfiles)"]:::script
          end

          subgraph "Interferograms workflow"
            direction TB
            IntfRunner["interferograms.runner\n(stage orchestration)"]:::python
            IntfGen["generate_run_files.sh\n(build runfiles)"]:::script
            IntfExec["execute_run_files.sh\n(run runfiles)"]:::script
          end
        end

        subgraph "Analysis / Visualization (post-processing)"
          direction TB
          XRAcc["xarray accessor\nDataArray.edk: geocode/plot/export"]:::python
          Viz["Datashader/Folium rendering\n(map tiles / overlays)"]:::python
          GDAL["GDAL tooling\n(GeoTIFF/COG export)"]:::dep
        end

        ISCE2Adapter["ISCE2 framework adapter/facade"]:::python
      end

      Req["Python deps boundary\nrequirements.txt"]:::doc
      Doc["Domain notes\nInSAR.md"]:::doc
      ExamplesNB["Example notebook\nsentinel1_mt_etna.ipynb"]:::example
      ExamplesPY["Example script\nsentinel1_mt_etna.py"]:::example
    end

    subgraph "Docker runtime (reproducible compute plane)"
      direction TB
      Compose["docker-compose.yml\n(ISCE2 environment)"]:::docker
      ISCE2["ISCE2 toolchain\n(coregistration, interferograms, etc.)"]:::compute
    end

    subgraph "External data providers"
      direction TB
      Copernicus["Copernicus Dataspace\n(Sentinel-1 SLC)"]:::external
      USGS["USGS\n(DEM/aux)"]:::external
    end

    subgraph "Local filesystem (file-based contracts)"
      direction TB
      DataVol[("data/\nSLCs, DEM, aux, stacks, geometry")]:::storage
      TmpVol[("tmp/\nscratch")]:::storage
      OutVol[("outputs/\ninterferograms, exports")]:::storage
    end
  end

  %% =========================
  %% Control flow (solid)
  %% =========================
  User -->|"runs"| ExecSH
  User -->|"imports/calls"| Init
  Init -->|"configures"| Consts
  Init -->|"exposes"| WFpkg
  Init -->|"enables analysis"| XRAcc

  WFpkg -->|"uses"| BaseRunner
  BaseRunner -->|"uses"| BaseHelpers
  BaseRunner -->|"provisions auth"| Netrc
  BaseRunner -->|"acquires DEM"| GetDEM
  BaseRunner -->|"acquires aux"| GetAUX

  WFpkg -->|"orchestrates"| CoregRunner
  CoregRunner -->|"generates"| CoregGen
  CoregRunner -->|"executes"| CoregExec

  WFpkg -->|"orchestrates"| IntfRunner
  IntfRunner -->|"generates"| IntfGen
  IntfRunner -->|"executes"| IntfExec

  CoregExec -->|"docker exec"| ISCE2Adapter
  IntfExec -->|"docker exec"| ISCE2Adapter
  ISCE2Adapter -->|"runs inside"| Compose
  Compose -->|"starts"| ISCE2

  XRAcc -->|"geocode()"| GDAL
  XRAcc -->|"plot()"| Viz
  XRAcc -->|"export() (COG/LZW)"| GDAL

  %% =========================
  %% Data flow (dashed)
  %% =========================
  Netrc -.->|"writes credentials"| DataVol

  Copernicus -.->|"downloads SLCs"| DataVol
  USGS -.->|"downloads DEM/aux"| DataVol
  GetDEM -.->|"stores DEM"| DataVol
  GetAUX -.->|"stores aux"| DataVol

  ISCE2 -.->|"reads SLC/DEM/aux"| DataVol
  ISCE2 -.->|"uses scratch"| TmpVol
  ISCE2 -.->|"writes coregistered stack\n(data/stack/...)"| DataVol
  ISCE2 -.->|"writes interferograms (*.int, coh)\n& geometry (lon/lat VRT)"| OutVol

  XRAcc -.->|"reads rasters/geometry"| DataVol
  XRAcc -.->|"reads interferograms"| OutVol
  GDAL -.->|"writes GeoTIFF/COG"| OutVol
  Viz -.->|"renders overlays from arrays"| OutVol

  ExamplesNB -.->|"demonstrates pipeline"| Init
  ExamplesPY -.->|"demonstrates pipeline"| Init
  Req -.->|"pins deps for host env"| Pkg
  Doc -.->|"documents processing notes"| Pkg

  %% =========================
  %% Click events (component mapping)
  %% =========================
  click Pkg "https://github.com/earth-data-kit/edk-sar/tree/master/edk_sar/"
  click Init "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/__init__.py"
  click Consts "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/constants.py"

  click Compose "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/dockerfiles/docker-compose.yml"
  click ExecSH "https://github.com/earth-data-kit/edk-sar/blob/master/exec.sh"

  click ISCE2Adapter "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/frameworks/isce2.py"

  click WFpkg "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/__init__.py"
  click BaseRunner "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/base/runner.py"
  click BaseHelpers "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/base/helpers.py"
  click Netrc "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/base/create_netrc.sh"
  click GetDEM "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/base/get_dem.sh"
  click GetAUX "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/base/get_aux_file.sh"

  click CoregRunner "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/coregister/runner.py"
  click CoregGen "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/coregister/generate_run_files.sh"
  click CoregExec "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/coregister/execute_run_files.sh"

  click IntfRunner "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/interferograms/runner.py"
  click IntfGen "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/interferograms/generate_run_files.sh"
  click IntfExec "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/workflows/interferograms/execute_run_files.sh"

  click XRAcc "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/xarray_accessor.py"
  click Viz "https://github.com/earth-data-kit/edk-sar/blob/master/edk_sar/edk_datashader.py"

  click ExamplesNB "https://github.com/earth-data-kit/edk-sar/blob/master/examples/sentinel1_mt_etna.ipynb"
  click ExamplesPY "https://github.com/earth-data-kit/edk-sar/blob/master/examples/sentinel1_mt_etna.py"

  click Doc "https://github.com/earth-data-kit/edk-sar/blob/master/InSAR.md"
  click Req "https://github.com/earth-data-kit/edk-sar/blob/master/requirements.txt"

  %% =========================
  %% Styles
  %% =========================
  classDef actor fill:#111827,stroke:#0b1220,color:#ffffff,stroke-width:2px
  classDef pkg fill:#1f6feb,stroke:#0b3d91,color:#ffffff,stroke-width:2px
  classDef python fill:#2f81f7,stroke:#0b3d91,color:#ffffff
  classDef script fill:#f59e0b,stroke:#92400e,color:#111827,stroke-width:1.5px
  classDef docker fill:#10b981,stroke:#065f46,color:#052e2b,stroke-width:2px
  classDef compute fill:#34d399,stroke:#065f46,color:#052e2b,stroke-width:2px
  classDef external fill:#9ca3af,stroke:#374151,color:#111827
  classDef storage fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef dep fill:#a78bfa,stroke:#5b21b6,color:#111827
  classDef doc fill:#fde68a,stroke:#92400e,color:#111827
  classDef example fill:#fb7185,stroke:#9f1239,color:#111827
  classDef entry fill:#f97316,stroke:#9a3412,color:#111827,stroke-width:2px
```
