# podaac/data-subscriber architecture

> Architecture diagram of the GitHub repository podaac/data-subscriber, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/podaac/data-subscriber
- Repository: https://github.com/podaac/data-subscriber

## Overview

This package provides two command-line workflows for PO.DAAC data: an on-demand bulk downloader and a recurring subscriber that tracks newly ingested granules. Both search CMR and download matching data; the subscriber also persists an update timestamp and can route requests through Harmony subsetting. The sampled entry points show these orchestration paths and their calls into shared access utilities. Some transfer and state-management implementation details are in omitted excerpts, so the graph shows only supported high-level relationships.

Main entry points: [`subscriber/podaac_access.py`](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_access.py) (PO.DAAC access), [`subscriber/podaac_data_subscriber.py`](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_subscriber.py) (Data subscriber), [`subscriber/podaac_data_downloader.py`](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_downloader.py) (Bulk downloader), [`subscriber/subsetting.py`](https://github.com/podaac/data-subscriber/blob/main/subscriber/subsetting.py) (Subset routing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflows["Download workflows"]
  node_downloader["Bulk downloader"]
  node_subscriber["Data subscriber"]
end

subgraph group_discovery["Search and access"]
  node_access["PO.DAAC access<br/>[podaac_access.py]"]
  node_auth["Earthdata auth<br/>[podaac_access.py]"]
  node_tokenformatter["Token redaction<br/>[token_formatter.py]"]
end

subgraph group_transfer["Transfer and state"]
  node_output[("Local output")]
end

subgraph group_subset["Harmony subsetting"]
  node_subsetrouting["Subset routing<br/>[subsetting.py]"]
end

node_user(("CLI user"))
node_cmr["CMR search"]
node_earthdata["Earthdata Login"]
node_filearchive["Data archive"]
node_harmony["Harmony jobs"]
node_github["Release API"]

node_user -->|"runs"| node_downloader
node_user -->|"runs"| node_subscriber
node_downloader -->|"gets token"| node_auth
node_subscriber -->|"gets token"| node_auth
node_auth -->|"requests tokens"| node_earthdata
node_downloader -->|"searches CMR"| node_access
node_subscriber -->|"searches CMR"| node_access
node_access -->|"queries collections"| node_cmr
node_downloader -->|"retrieves granules"| node_access
node_subscriber -->|"retrieves granules"| node_access
node_access -->|"downloads data"| node_filearchive
node_downloader -->|"writes files"| node_output
node_subscriber -->|"writes files"| node_output
node_subscriber -->|"reads update state"| node_output
node_subscriber -->|"writes update state"| node_output
node_downloader -->|"routes subset requests"| node_subsetrouting
node_subscriber -->|"routes subset requests"| node_subsetrouting
node_subsetrouting -->|"checks support"| node_access
node_subsetrouting -->|"finds or submits job"| node_access
node_access -->|"submits and monitors"| node_harmony
node_subsetrouting -->|"downloads results"| node_access
node_access -->|"saves job state"| node_output
node_downloader -->|"formats logs"| node_tokenformatter
node_subscriber -->|"formats logs"| node_tokenformatter
node_downloader -.->|"checks releases"| node_access
node_subscriber -.->|"checks releases"| node_access
node_access -.->|"checks latest release"| node_github

click node_downloader "https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_downloader.py"
click node_subscriber "https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_subscriber.py"
click node_access "https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_access.py"
click node_auth "https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_access.py"
click node_subsetrouting "https://github.com/podaac/data-subscriber/blob/main/subscriber/subsetting.py"
click node_tokenformatter "https://github.com/podaac/data-subscriber/blob/main/subscriber/token_formatter.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_downloader,node_subscriber,node_user toneBlue
class node_access,node_auth,node_tokenformatter toneAmber
class node_output,node_github toneMint
class node_subsetrouting toneRose
class node_cmr,node_earthdata,node_filearchive,node_harmony toneIndigo
```

## Components

### Download workflows

- [Bulk downloader](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_downloader.py): `subscriber/podaac_data_downloader.py`
- [Data subscriber](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_data_subscriber.py): `subscriber/podaac_data_subscriber.py`

### Search and access

- [PO.DAAC access](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_access.py): `subscriber/podaac_access.py`
- [Earthdata auth](https://github.com/podaac/data-subscriber/blob/main/subscriber/podaac_access.py): `subscriber/podaac_access.py`
- [Token redaction](https://github.com/podaac/data-subscriber/blob/main/subscriber/token_formatter.py): `subscriber/token_formatter.py`

### Transfer and state

- **Local output**

### Harmony subsetting

- [Subset routing](https://github.com/podaac/data-subscriber/blob/main/subscriber/subsetting.py): `subscriber/subsetting.py`

### Other components

- **CLI user**
- **CMR search**
- **Earthdata Login**
- **Data archive**
- **Harmony jobs**
- **Release API**

## Connections

- CLI user → Bulk downloader: runs
- CLI user → Data subscriber: runs
- Bulk downloader → Earthdata auth: gets token
- Data subscriber → Earthdata auth: gets token
- Earthdata auth → Earthdata Login: requests tokens
- Bulk downloader → PO.DAAC access: searches CMR
- Data subscriber → PO.DAAC access: searches CMR
- PO.DAAC access → CMR search: queries collections
- Bulk downloader → PO.DAAC access: retrieves granules
- Data subscriber → PO.DAAC access: retrieves granules
- PO.DAAC access → Data archive: downloads data
- Bulk downloader → Local output: writes files
- Data subscriber → Local output: writes files
- Data subscriber → Local output: reads update state
- Data subscriber → Local output: writes update state
- Bulk downloader → Subset routing: routes subset requests
- Data subscriber → Subset routing: routes subset requests
- Subset routing → PO.DAAC access: checks support
- Subset routing → PO.DAAC access: finds or submits job
- PO.DAAC access → Harmony jobs: submits and monitors
- Subset routing → PO.DAAC access: downloads results
- PO.DAAC access → Local output: saves job state
- Bulk downloader → Token redaction: formats logs
- Data subscriber → Token redaction: formats logs
- Bulk downloader → PO.DAAC access: checks releases
- Data subscriber → PO.DAAC access: checks releases
- PO.DAAC access → Release API: checks latest release