# wdgph/workflow-wsi architecture

> Architecture diagram of the GitHub repository wdgph/workflow-wsi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wdgph/workflow-wsi
- Repository: https://github.com/wdgph/workflow-wsi

## Overview

##### Project type and purpose
- This is a containerized data‐pipeline project (“WSI Data Pipeline”) whose goal is to automatically retrieve COVID-19 and influenza wastewater surveillance data from an ArcGIS REST endpoint and perform standardization and trend analysis.
- It consists of two main containers (“retrieval” and “processing”), built and published via GitHub Actions, and typically orchestrated by Kubeflow Pipelines / Argo Workflows.

##### File structure
- Top level:
- `.github/workflows/` – CI/CD workflows that build and publish container images.
- `processing/` – contains Dockerfile and `process.R` (data cleaning, weighting, trend analysis).
- `retrieval/` – contains Dockerfile and `getdata.py` (downloads ArcGIS data to CSV).
- No monolithic codebase: it’s a two‐stage pipeline realized as separate containers.
- No database or UI code here; primary artifacts are CSV files.

##### README for additional insights
- Retrieval container inputs: ArcGIS REST URL + credentials via environment variables.
- Processing container inputs: raw CSV + weights + optional patch; outputs multiple CSVs by disease/site or region.
- Orchestration: Kubeflow Pipelines (Argo) manages the two‐step flow.
- CI/CD: GitHub Actions builds images and pushes to GitHub Container Registry.

##### Detailed capture of architectural elements
- GitHub Actions workflow steps building and pushing images
- Versioned images in the Container Registry
- If desired, note optional patch and weighting file inputs to processing
- Scalability (e.g., multiple parallel retrieval tasks)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD and Registry
    CI["GitHub Actions CI/CD"]:::ci
    CR["Container Registry\n(GitHub Packages)"]:::orchestration

    CI -->|"build & push images"| CR

    %% Orchestrator
    ORC["Orchestrator\n(Kubeflow Pipelines / Argo)"]:::orchestration
    CR -->|"pull images"| ORC

    %% External Services
    SECRET["Secret Management\n(Env Vars: ARCGIS_USER/PASSWORD)"]:::external
    EXT["ArcGIS REST API"]:::external

    %% Pipeline Containers
    RET["Retrieval Container\n(Docker)"]:::compute
    PROC["Processing Container\n(Docker)"]:::compute

    %% Storage
    STORAGE["Artifact Storage\n(CSV Outputs)"]:::storage

    %% Data Flow
    ORC -->|"run retrieval step"| RET
    SECRET -->|"provide credentials"| RET
    EXT -->|"return data"| RET
    RET -->|"raw CSV"| STORAGE

    ORC -->|"run processing step"| PROC
    STORAGE -->|"raw CSV + weights + patch"| PROC
    PROC -->|"processed CSVs"| STORAGE

    %% Click Events
    click RET "https://github.com/wdgph/workflow-wsi/tree/main/retrieval/dockerfile"
    click RET "https://github.com/wdgph/workflow-wsi/blob/main/retrieval/getdata.py"
    click PROC "https://github.com/wdgph/workflow-wsi/tree/main/processing/dockerfile"
    click PROC "https://github.com/wdgph/workflow-wsi/blob/main/processing/process.R"
    click CI "https://github.com/wdgph/workflow-wsi/blob/main/ .github/workflows/build-retrieval.yaml"
    click CI "https://github.com/wdgph/workflow-wsi/blob/main/ .github/workflows/build-processing.yaml"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef orchestration fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef ci fill:#FF8C00,stroke:#333,stroke-width:1px
```
