How worldfishcenter/peskas.kenya.data.pipeline works
Project type and purpose
• This is an R-package–based data pipeline for the Peskas Kenya fisheries project. Its goal is to orchestrate end-to-end data collection, processing, validation, analytics, export, and reporting & visualization of small-scale fishing data.
File structure
• Top-level R/ directory: individual R scripts implementing pipeline stages (ingestion.R, preprocessing.R, validation.R, analytics functions embedded in various files, export.R, storage.R).
• inst/: assets shipped with the package (geojson boundaries, config.yml, report templates and extensions).
• man/: documentation for every exported function—indicates a function-per-operation modular design.
• Dockerfile.prod, GitHub workflows: CI/CD and containerization for scheduled or on-demand pipeline runs.
• config.yml in inst/: holds environment-specific parameters (cloud credentials, paths, thresholds).
Key insights from README
• The pipeline comprises six modules: Data Collection → Pre-processing → Validation → Analytics → Data Export → Visualization.
• It reads configuration via read_config(), uses cloud upload/download functions, and outputs processed summaries for dashboards.
• CI/CD is implemented via GitHub Actions (R-CMD check, coverage, data‐pipeline runs), and production deployment via Docker.
Elements
• Clear labels for each component (e.g., “Ingestion Module”, “Validation Module”).