How worldfishcenter/peskas.zanzibar.data.pipeline works
Project type and purpose
• This is an R-package-based data and modeling pipeline (“peskas.zanzibar.data.pipeline”) for small-scale fisheries analytics in Zanzibar.
• It provides modular R functions for ingestion, preprocessing, validation, modeling, summarization, mapping, storage, and reporting, and is deployed and orchestrated via GitHub Actions and Docker.
File structure
• Top-level files: DESCRIPTION, NAMESPACE (R package metadata); README.Rmd/README.md; Dockerfile (dev), Dockerfile.prod; docker-compose.yaml; inst/conf.yml (environment config); .github/workflows/*.yaml (CI/CD definitions).
• R/: the core functions, organized by responsibility (ingestion.R, preprocessing.R, model-*.R, summarize-data.R, storage.R, validation.R, kepler-mapping.R, export.R, utils-*.R).
• inst/: static assets and auxiliary code (geoJSON files, RDS parameters, Python script for Kepler mapping, conf.yml template, QMD/HTML report templates).
• man/: autogenerated documentation for every exported function.
• Workflows: data-pipeline.yaml defines the pipeline jobs; other workflows handle testing, coverage, pkgdown site, releases.
README for additional insights
• The pipeline runs in three environments (“default”/.github, “local”, and “production”) controlled by inst/conf.yml and R_CONFIG_ACTIVE.
• GitHub Actions jobs build a production Docker image, then sequentially call package functions for each step, passing artifacts via cloud storage.
• Artifacts versioned with add_version(); retrieved by next job via cloud_object_name().
• Uses the logger package for logging.
a. Main components
• Code Repository (GitHub) containing R package, Dockerfiles, workflows, configs
• CI/CD Engine (GitHub Actions) that triggers on pushes or schedules
• Container Build & Runtime
– Dev container (Dockerfile + docker-compose for RStudio)
– Prod container (Dockerfile.prod)
• Configuration Module (inst/conf.yml + config package)
• Pipeline Steps (containerized R processes invoking package functions)
1. Ingestion
2. Preprocessing
3. Validation
4. Modeling
5. Summarization & Mapping
6. Reporting/Export
• Cloud Storage Service (object store for intermediate/final artifacts)
• External Services (authentication via GitHub Secrets/env vars, cloud credentials)
• Artifacts (versioned data files, GeoJSON, Parquet, reports, pkgdown site)
b. Relationships & data flows
• Developer → pushes code → GitHub Actions
• Actions → build Docker image → spin up container → each job runs R script → reads previous artifacts from cloud → writes new artifacts back to cloud → next job picks them up
• Config layer injects environment variables into each container run
• Final artifacts published (reports to inst/reports, website to pkgdown, data to cloud buckets)
c. Architectural patterns & principles
• Modular design enforced by R package structure (functions as building blocks)
• Single-responsibility per pipeline function
• CI/CD pipeline as orchestrator (akin to workflow engine)
• Infrastructure as code (Dockerfiles, workflows, conf.yml)
• Immutable artifacts & versioning for reproducibility
• Separation of environments (dev/test/prod) via config
d. Key technologies & libraries
• R / tidyverse / config / logger
• Docker (rocker/geospatial base for dev, lightweight base for prod)
• GitHub Actions (CI/CD)
• Cloud storage (e.g., AWS S3/GCS via cloud_storage_authenticate, upload/download functions)
• Kepler mapping (inst/kepler_wcs_surveys.py + kepler-mapping.R)
• pkgdown (documentation site)