How earth-data-kit/edk-sar works
What this project is (type + purpose)
edk-sar is a Python library / workflow wrapper around ISCE2 for running end-to-end InSAR processing (coregistration → interferogram generation → downstream analysis like phase/displacement), with a strong emphasis on:
• Reproducible execution via Docker (ISCE2 + dependencies)
• Python orchestration of ISCE2 steps using “workflows”
• Post-processing and analysis in modern Python tooling (xarray via a custom accessor, GDAL exports, mapping/visualization)
So the “system” is not a classic web app; it’s a local/CLI + containerized scientific processing pipeline that interacts with external data providers and produces files (rasters, stacks, outputs).
• A Python control plane (edk_sar)
• A containerized compute runtime (ISCE2)
• External data sources (Copernicus, USGS)
• Local persistent storage (data/tmp/output directories)
• Analysis & visualization layer (xarray accessor, datashader/folium)
• Export tooling (GDAL/COG)
Top-level
• edk_sar/ → main Python package (core library)
• edk_sar/dockerfiles/docker-compose.yml → container runtime definition (ISCE2 environment)
• requirements.txt → Python dependency boundary
• InSAR.md → likely domain notes (optional reference)
edk_sar package
• edk_sar/__init__.py → public API surface (e.g., es.init, es.workflows...)
• edk_sar/constants.py → global constants (e.g., wavelength, paths, etc.) to annotate in component notes
• edk_sar/workflows/ → main pipeline orchestration
• workflows/base/*:
• runner.py + helpers.py → shared orchestration utilities (command execution, path mgmt, env mgmt)
• create_netrc.sh → credentials provisioning for download (external services)
• get_dem.sh, get_aux_file.sh → external data acquisition steps
• workflows/coregister/*:
• generate_run_files.sh, execute_run_files.sh, runner.py → staged execution model
• workflows/interferograms/*:
• similar pattern to coregister
• edk_sar/xarray_accessor.py → .edk accessor (analysis API layer on top of xarray)
• edk_sar/edk_datashader.py → visualization/rendering component (map tiles / datashader pipeline)
This structure strongly suggests: Python orchestrates, shell scripts perform discrete ops, ISCE2 runs inside Docker, outputs become rasters that xarray consumes and exports.
“Control plane” (Python library layer)
• imports edk_sar
• calls es.init(".env")
• calls es.workflows.coregister.run(slc_path) and es.workflows.interferograms.run(slc_path)
• later uses xarray + .edk accessor for analysis and export
• Init/Config (__init__.py, .env loading)
• Workflows Orchestrator (edk_sar/workflows/**/runner.py, base/runner.py)
• ISCE2 Framework Adapter (edk_sar/frameworks/isce2.py)
• Analysis API (xarray_accessor.py)
• Visualization/Rendering (edk_datashader.py)
“Compute/runtime plane” (Docker + ISCE2)
• references docker-compose.yml
• contains ISCE2 Toolchain (and likely SNAPHU/unwrapping tools, GDAL inside container depending on setup)
Make it explicit that workflow runners call into Dockerized commands (even if the exact invocation isn’t visible here, README states Docker-based execution is core).
Data sources (external dependencies)
• Copernicus Dataspace (Sentinel-1 SLC downloads/auth)
• USGS (DEM/aux data/auth, depending on what scripts fetch)
Them to “Download/Provision” steps in workflows/base/*:
• create_netrc.sh (credentials) → enables downloads
• get_dem.sh, get_aux_file.sh → fetch required supporting datasets
Storage (file-based architecture is central)
• data/ (created by user; likely holds SLCs, stacks, DEMs, geometry)
• tmp/ (scratch space)
• Output directories (stack, merged, geom_reference, interferograms, etc. — even if not in tree, README hints at paths like edk_sar/data/stack/merged/geom_reference/)
• Downloads → data/
• ISCE2 processing → reads from data/ / writes to stack outputs
• xarray accessor → reads rasters/ENVI/VRT/etc. from outputs
• GDAL export → writes GeoTIFF/COG outputs
Analysis & export
• geocode() depends on geometry rasters (lon/lat VRTs)
• plot() depends on folium/datashader stack
• export() depends on GDAL (COG LZW compression)
coregister.runner → base.helpers/base.runner → Docker/ISCE2
• reads SLC inputs from data/
• writes “coregistered stack” outputs into data/stack/...
interferograms.runner → Docker/ISCE2
• reads coregistered products
• writes interferogram products (*.int, coherence, etc.)
User / Notebook → xarray.open_dataarray(...)
7. DataArray.edk.geocode() → uses geometry rasters (lon.rdr.vrt, lat.rdr.vrt) and GDAL tooling
Data acquisition subflow
• edk_sar.init / workflow setup → create_netrc.sh
→ enables authenticated access to USGS/Copernicus
• get_dem.sh → downloads DEM → stored in data/ → used by ISCE2 steps
• Control calls (Python → runner → docker exec)
• Data flow (external → local storage → ISCE2 → outputs → xarray → exports)
Architectural patterns to explicitly show
• Facade/Adapter: edk_sar/frameworks/isce2.py wraps ISCE2 complexity.
• Workflow/Orchestration: workflows/*/runner.py coordinate multi-step execution; shell scripts generate/execute run files.
• Batch pipeline + file-based contracts: intermediate artifacts on disk are the integration boundary between stages.
• Reproducible runtime boundary: Docker isolates ISCE2 dependencies from host Python env.
• Extension point: custom xarray accessor (.edk) is the “user-friendly API” for geocoding/plot/export.
“Don’t overthink it” rule applied to this repo (what to split into separate components)
• coregister workflow runner
• interferograms workflow runner
• base runner/helpers
• credential provisioning (create_netrc.sh)
• DEM/aux download steps
• ISCE2 adapter layer
• Docker runtime
• external providers (USGS, Copernicus)
• filesystem volumes (data/tmp/output)
• xarray accessor (geocode/plot/export)
• GDAL export dependency
• visualization subsystem (datashader/folium)