# worldfishcenter/peskas.kenya.data.pipeline architecture

> Architecture diagram of the GitHub repository worldfishcenter/peskas.kenya.data.pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/worldfishcenter/peskas.kenya.data.pipeline
- Repository: https://github.com/worldfishcenter/peskas.kenya.data.pipeline

## Overview

##### Project type and purpose
- This is an R-package–based data pipeline for the Peskas Kenya fisheries project. Its goal is to orchestrate end-to-end data collection, processing, validation, analytics, export, and reporting & visualization of small-scale fishing data.

##### File structure
- Top-level R/ directory: individual R scripts implementing pipeline stages (ingestion.R, preprocessing.R, validation.R, analytics functions embedded in various files, export.R, storage.R).
- inst/: assets shipped with the package (geojson boundaries, config.yml, report templates and extensions).
- man/: documentation for every exported function—indicates a function-per-operation modular design.
- Dockerfile.prod, GitHub workflows: CI/CD and containerization for scheduled or on-demand pipeline runs.
- config.yml in inst/: holds environment-specific parameters (cloud credentials, paths, thresholds).

##### Key insights from README
- The pipeline comprises six modules: Data Collection → Pre-processing → Validation → Analytics → Data Export → Visualization.
- It reads configuration via read_config(), uses cloud upload/download functions, and outputs processed summaries for dashboards.
- CI/CD is implemented via GitHub Actions (R-CMD check, coverage, data‐pipeline runs), and production deployment via Docker.

##### Elements
- Clear labels for each component (e.g., “Ingestion Module”, “Validation Module”).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    DS["Data Sources\n(GPS, Kobo, CSV, Cloud Buckets)"]:::datasource

    %% Configuration
    Config["Configuration Store\n(inst/config.yml)"]:::config
    click Config "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/inst/config.yml"

    %% Pipeline Stages
    subgraph "Pipeline Stages" 
        direction TB
        Ingest["Ingestion Module"]:::stage
        Preproc["Pre-processing Module"]:::stage
        Validate["Validation Module"]:::stage
        Analytics["Analytics Module"]:::stage
        Export["Export Module"]:::stage
        Report["Reporting/Visualization"]:::stage
    end

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment" 
        direction TB
        CI["GitHub Actions"]:::cicd
        Docker["Docker Container"]:::cicd
    end

    %% External Services
    CloudBucket["Cloud Storage\n(External)"]:::external
    AuthService["Authentication Service"]:::external
    Dashboard["Dashboard Service"]:::external

    %% Connections
    DS -->|reads| Ingest
    Config -->|used by| Ingest
    Ingest -->|output data| Preproc
    Config -->|used by| Preproc
    Preproc -->|cleaned data| Validate
    Config -->|used by| Validate
    Validate -->|validated data| Analytics
    Config -->|used by| Analytics
    Analytics -->|metrics| Export
    Config -->|used by| Export
    Export -->|reports & summaries| Report
    Config -->|used by| Report

    Export -->|upload/download| CloudBucket
    CloudBucket -->|download raw| Ingest

    Report -->|serves| Dashboard

    CI -->|triggers| Docker
    Docker -->|runs pipeline| Ingest

    %% Click Events for modules and files
    click Ingest "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/ingestion.R"
    click Ingest "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/merge-landings.R"
    click Preproc "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/preprocessing.R"
    click Preproc "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/utils-pipe.R"
    click Validate "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/validation.R"
    click Validate "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/validation-functions.R"
    click Analytics "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/utils.R"
    click Analytics "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/globals.R"
    click Analytics "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/utils-tidy-evals.R"
    click Export "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/export.R"
    click Export "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/storage.R"
    click Report "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/tree/main/inst/reports"
    click Report "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/tree/main/pkgdown/"
    click Report "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/.github/workflows/pkgdown.yaml"
    click CI "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/.github/workflows/R-CMD-check.yaml"
    click CI "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/.github/workflows/data-pipeline.yaml"
    click CI "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/.github/workflows/test-coverage.yaml"
    click CI "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/.github/workflows/release.yaml"
    click CI "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/Dockerfile.prod"
    click Export "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/man/upload_cloud_file.Rd"
    click Export "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/man/download_cloud_file.Rd"
    click Config "https://github.com/worldfishcenter/peskas.kenya.data.pipeline/blob/main/R/utils.R"

    %% Styles
    classDef stage fill:#cce5ff,stroke:#337ab7,stroke-width:1px
    classDef external fill:#d5f5e3,stroke:#27ae60,stroke-width:1px
    classDef config fill:#f9e79f,stroke:#d68910,stroke-width:1px
    classDef datasource fill:#f0f3f4,stroke:#7f8c8d,stroke-width:1px,stroke-dasharray: 5 5
    classDef cicd fill:#f5cba7,stroke:#d35400,stroke-width:1px
```
