# worldfishcenter/peskas.zanzibar.data.pipeline architecture

> Architecture diagram of the GitHub repository worldfishcenter/peskas.zanzibar.data.pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/worldfishcenter/peskas.zanzibar.data.pipeline
- Repository: https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline

## Overview

##### Project type and purpose
- This is an R-package-based data and modeling pipeline (“peskas.zanzibar.data.pipeline”) for small-scale fisheries analytics in Zanzibar.
- It provides modular R functions for ingestion, preprocessing, validation, modeling, summarization, mapping, storage, and reporting, and is deployed and orchestrated via GitHub Actions and Docker.

##### File structure
- Top-level files: DESCRIPTION, NAMESPACE (R package metadata); README.Rmd/README.md; Dockerfile (dev), Dockerfile.prod; docker-compose.yaml; inst/conf.yml (environment config); .github/workflows/*.yaml (CI/CD definitions).
- R/: the core functions, organized by responsibility (ingestion.R, preprocessing.R, model-*.R, summarize-data.R, storage.R, validation.R, kepler-mapping.R, export.R, utils-*.R).
- inst/: static assets and auxiliary code (geoJSON files, RDS parameters, Python script for Kepler mapping, conf.yml template, QMD/HTML report templates).
- man/: autogenerated documentation for every exported function.
- Workflows: data-pipeline.yaml defines the pipeline jobs; other workflows handle testing, coverage, pkgdown site, releases.

##### README for additional insights
- The pipeline runs in three environments (“default”/.github, “local”, and “production”) controlled by inst/conf.yml and R_CONFIG_ACTIVE.
- GitHub Actions jobs build a production Docker image, then sequentially call package functions for each step, passing artifacts via cloud storage.
- Artifacts versioned with add_version(); retrieved by next job via cloud_object_name().
- Uses the logger package for logging.

a. Main components
- Code Repository (GitHub) containing R package, Dockerfiles, workflows, configs
- CI/CD Engine (GitHub Actions) that triggers on pushes or schedules
- Container Build & Runtime
– Dev container (Dockerfile + docker-compose for RStudio)
– Prod container (Dockerfile.prod)
- Configuration Module (inst/conf.yml + config package)
- Pipeline Steps (containerized R processes invoking package functions)
1. Ingestion
2. Preprocessing
3. Validation
4. Modeling
5. Summarization & Mapping
6. Reporting/Export
- Cloud Storage Service (object store for intermediate/final artifacts)
- External Services (authentication via GitHub Secrets/env vars, cloud credentials)
- Artifacts (versioned data files, GeoJSON, Parquet, reports, pkgdown site)

b. Relationships & data flows
- Developer → pushes code → GitHub Actions
- Actions → build Docker image → spin up container → each job runs R script → reads previous artifacts from cloud → writes new artifacts back to cloud → next job picks them up
- Config layer injects environment variables into each container run
- Final artifacts published (reports to inst/reports, website to pkgdown, data to cloud buckets)

c. Architectural patterns & principles
- Modular design enforced by R package structure (functions as building blocks)
- Single-responsibility per pipeline function
- CI/CD pipeline as orchestrator (akin to workflow engine)
- Infrastructure as code (Dockerfiles, workflows, conf.yml)
- Immutable artifacts & versioning for reproducibility
- Separation of environments (dev/test/prod) via config

d. Key technologies & libraries
- R / tidyverse / config / logger
- Docker (rocker/geospatial base for dev, lightweight base for prod)
- GitHub Actions (CI/CD)
- Cloud storage (e.g., AWS S3/GCS via cloud_storage_authenticate, upload/download functions)
- Kepler mapping (inst/kepler_wcs_surveys.py + kepler-mapping.R)
- pkgdown (documentation site)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer triggers pipeline
    Developer["Developer"]:::external
    Developer -->|"push code"| CodeRepo

    %% Code Repository
    subgraph "Code Repository"
        CodeRepo["Root Repository"]:::code
    end

    %% CI/CD Engine
    subgraph "CI/CD Engine"
        CICDEngine["GitHub Actions"]:::code
        DataPipeline["Data Pipeline Workflow"]:::code
    end
    CodeRepo -->|contains workflows| CICDEngine
    CICDEngine -->|runs| DataPipeline

    %% Containers
    subgraph "Containers"
        DevDocker["Dev Dockerfile"]:::container
        DevCompose["docker-compose.yaml"]:::container
        ProdContainer["Prod Dockerfile"]:::container
    end
    CICDEngine -->|build| DevDocker
    CICDEngine -->|build| DevCompose
    CICDEngine -->|build| ProdContainer

    %% Configuration
    ConfigModule["Configuration Module\n(inst/conf.yml)"]:::code
    DevDocker -->|env vars| ConfigModule
    DevCompose -->|env vars| ConfigModule
    ProdContainer -->|env vars| ConfigModule
    DataPipeline -->|loads| ConfigModule

    %% Pipeline Steps
    subgraph "Pipeline Steps"
        Ingestion["Ingestion Job"]:::code
        Preprocessing["Preprocessing Job"]:::code
        Validation["Validation Job"]:::code
        Modeling["Modeling Job"]:::code
        SummarizationMapping["Summarize & Mapping"]:::code
        ReportingExport["Reporting/Export"]:::code
    end
    DataPipeline --> Ingestion
    Ingestion --> Preprocessing
    Preprocessing --> Validation
    Validation --> Modeling
    Modeling --> SummarizationMapping
    SummarizationMapping --> ReportingExport

    %% Cloud Storage
    CloudStorage["Cloud Storage"]:::datastore

    %% Storage I/O
    StorageCloudIO["Storage Module"]:::code
    Ingestion -->|writes artifacts| StorageCloudIO
    Preprocessing -->|reads/writes| StorageCloudIO
    Validation -->|reads/writes| StorageCloudIO
    Modeling -->|reads/writes| StorageCloudIO
    SummarizationMapping -->|reads/writes| StorageCloudIO
    ReportingExport -->|reads/writes| StorageCloudIO
    StorageCloudIO -->|stores to| CloudStorage
    StorageCloudIO -->|retrieves from| CloudStorage

    %% Static Assets
    StaticAssets["Static Assets"]:::code
    SummarizationMapping -->|uses| StaticAssets

    %% Artifacts
    subgraph "Artifacts"
        Reports["Reports (.html, .qmd)"]:::artifact
        GeoJSONs["GeoJSON outputs"]:::artifact
        Parquet["Parquet files"]:::artifact
        Website["Pkgdown Site"]:::artifact
    end
    ReportingExport --> Reports
    SummarizationMapping --> GeoJSONs
    StorageCloudIO --> Parquet
    CICDEngine -->|deploys| Website

    %% External Services
    ExternalServices["External Services\n(Secrets, Cloud Auth)"]:::external
    CICDEngine -.->|uses| ExternalServices
    StorageCloudIO -.->|auth| ExternalServices

    %% Click Events
    click CodeRepo "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/tree/main/./"
    click CICDEngine "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/tree/main/.github/workflows/"
    click DataPipeline "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/.github/workflows/data-pipeline.yaml"
    click DevDocker "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/tree/main/./Dockerfile"
    click DevCompose "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/./docker-compose.yaml"
    click ProdContainer "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/./Dockerfile.prod"
    click ConfigModule "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/./inst/conf.yml"
    click Ingestion "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/ingestion.R"
    click Preprocessing "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/preprocessing.R"
    click Validation "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/validation.R"
    click Modeling "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/model-fishery.R"
    click SummarizationMapping "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/summarize-data.R"
    click ReportingExport "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/export.R"
    click StorageCloudIO "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/blob/main/R/storage.R"
    click StaticAssets "https://github.com/worldfishcenter/peskas.zanzibar.data.pipeline/tree/main/inst/"

    %% Styles
    classDef code fill:#BBDEFB,stroke:#000,stroke-width:1px
    classDef container fill:#C8E6C9,stroke:#000,stroke-width:1px
    classDef datastore fill:#FFE0B2,stroke:#000,stroke-width:1px
    classDef artifact fill:#E0E0E0,stroke:#000,stroke-width:1px
    classDef external stroke-dasharray: 5 5,fill:none,stroke:#000,stroke-width:1px
```
