How wdgph/workflow-wsi works
Project type and purpose
• This is a containerized data‐pipeline project (“WSI Data Pipeline”) whose goal is to automatically retrieve COVID-19 and influenza wastewater surveillance data from an ArcGIS REST endpoint and perform standardization and trend analysis.
• It consists of two main containers (“retrieval” and “processing”), built and published via GitHub Actions, and typically orchestrated by Kubeflow Pipelines / Argo Workflows.
File structure
• Top level:
• .github/workflows/ – CI/CD workflows that build and publish container images.
• processing/ – contains Dockerfile and process.R (data cleaning, weighting, trend analysis).
• retrieval/ – contains Dockerfile and getdata.py (downloads ArcGIS data to CSV).
• No monolithic codebase: it’s a two‐stage pipeline realized as separate containers.
• No database or UI code here; primary artifacts are CSV files.
README for additional insights
• Retrieval container inputs: ArcGIS REST URL + credentials via environment variables.
• Processing container inputs: raw CSV + weights + optional patch; outputs multiple CSVs by disease/site or region.
• Orchestration: Kubeflow Pipelines (Argo) manages the two‐step flow.
• CI/CD: GitHub Actions builds images and pushes to GitHub Container Registry.
Detailed capture of architectural elements
• GitHub Actions workflow steps building and pushing images
• Versioned images in the Container Registry
• If desired, note optional patch and weighting file inputs to processing
• Scalability (e.g., multiple parallel retrieval tasks)