# worldfishcenter/peskas.timor.data.pipeline architecture

> Architecture diagram of the GitHub repository worldfishcenter/peskas.timor.data.pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/worldfishcenter/peskas.timor.data.pipeline
- Repository: https://github.com/worldfishcenter/peskas.timor.data.pipeline

## Overview

##### Project Type and Purpose
- This is an R package structured as a data pipeline. Its purpose is to implement, deploy, and execute data ingestion, cleaning, preprocessing, modeling, and reporting workflows for small-scale fisheries analytics.
- The README outlines that the pipeline is deployed as a series of interconnected GitHub Actions jobs, orchestrating containerized execution via Docker. It also emphasizes the use of environment-specific configurations (remote, local, production) and cloud storage as storage for versioned artifacts.

##### Analyzing the File Structure
- Top-level files indicate an R package: DESCRIPTION, NAMESPACE, documentation (man/), and R/ directories contain function files (e.g., calculate-nutrients.R, ingest-landings.R, etc.).
- The presence of GitHub Actions workflows under .github/workflows (e.g., data-pipeline.yaml, data-report.yaml, etc.) shows the CI/CD pipeline is integral.
- Dockerfiles (Dockerfile and Dockerfile.prod) and docker-compose.yaml indicate separate containerized environments for development and production.
- Inst/ directory holds auxiliary code and configuration—for example, inst/conf.yml (the environment configuration file) and scripts (e.g., kepler_mapper.py, which suggests auxiliary mapping/reporting functionality).

a. Main Components:
- Core Pipeline Functions: Represented by various R scripts (e.g., ingestion, cleaning, preprocessing, merging, modeling, and validation functions). This is the heart of the R package.
- GitHub Actions Workflows: Each YAML file under .github/workflows represents a distinct job in the CI/CD pipeline (e.g., data ingestion, data reporting, uploading artifacts, etc.).
- Docker Containers: There are two primary container definitions—one for development (Dockerfile) running an RStudio server via rocker/geospatial and one for production (Dockerfile.prod) for lightweight execution.
- Configuration Management: The inst/conf.yml file used by the config package to determine parameters based on the environment (local/cloud/production).
- External Services and Cloud Storage: The pipeline stores outputs (artifacts) in cloud storage containers; this component links to external APIs (Google Drive, possibly Dataverse, cloud storage authentication via environment variables, etc.).
- Testing and Documentation: Presence of tests/ and pkgdown configurations indicates that automated testing and website documentation are also integral parts of the architecture.

b. Relationships and Interactions:
- The workflow begins with the GitHub Actions triggering the job (e.g., data-pipeline.yaml). This workflow calls R functions (each implemented as separate scripts) that perform tasks such as data ingestion and processing.
- The execution environment is defined by Docker containers. The development container supports interactive coding (RStudio) while the production container supports efficient job execution.
- Configuration is loaded at runtime using the config package. Based on the R_CONFIG_ACTIVE environment variable, the pipeline selects between remote, local, or production configurations.
- Artifacts produced in one stage are versioned (with add_version()) and are then stored in cloud storage for subsequent jobs to retrieve (using cloud_object_name()).
- Logging is handled by the logger package to record events, especially in production.

c. Architectural Patterns and Design Principles:
- Modular Function Decomposition: The pipeline is broken down into many single-responsibility functions or “stages” (e.g., ingest, preprocess, validate, model). This promotes code reuse and easier testing.
- Containerization: Using Docker for both development and production environments ensures consistency across deployments.
- Environment-Based Configuration: Decoupling settings and secret management by switching configurations based on the runtime environment.
- CI/CD Pipeline Automation: GitHub Actions orchestrate the sequential execution of tasks (with dependencies clearly defined between them).

d. Relevant Technologies, Frameworks, and Libraries:
- R and Tidyverse: For data manipulation and standard coding practices.
- GitHub Actions: For automating builds, tests, and deployments.
- Docker: For containerizing the application.
- config package: For environment-specific configuration management.
- logger package: For logging production events.
- Additional utilities: The usethis package to facilitate setup, and pkgdown for generating documentation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    GA["GitHub Actions Workflows"]:::cdicolor
    %% Docker Containers Subgraph
    subgraph "Docker Containers"
        D1["Development (Dockerfile)"]:::container
        D2["Production (Dockerfile.prod)"]:::container
        D3["Orchestration (docker-compose.yaml)"]:::container
    end
    %% Core Pipeline Functions
    CP["Core Pipeline Functions"]:::rmodule
    %% Auxiliary Components
    CM["Configuration Management"]:::config
    LOG["Logging (logger package)"]:::rmodule
    %% External Services Subgraph
    subgraph "External Services"
        ES1["Cloud Storage (R/cloud-storage.R)"]:::external
        ES2["Google Drive (R/google-drive.R)"]:::external
        ES3["Dataverse Export (R/export-dataverse.R)"]:::external
    end
    %% Testing and Documentation Subgraph
    subgraph "Testing and Documentation"
        T["Testing (tests/)"]:::test
        DOC["Documentation (man/)"]:::test
        PKG["Pkgdown (_pkgdown.yml)"]:::test
    end

    %% Connections
    GA -->|"triggers"| D1
    GA -->|"triggers"| D2
    GA -->|"triggers"| D3

    D1 -->|"runs"| CP
    D2 -->|"runs"| CP
    D3 -->|"orchestrates"| CP

    CP -->|"loads"| CM
    CP -->|"logs"| LOG
    CP -->|"stores artifacts"| ES1
    CP -->|"integrates with"| ES2
    CP -->|"exports to"| ES3

    GA -->|"initiates"| T
    GA -->|"initiates"| DOC
    GA -->|"initiates"| PKG

    %% Class Definitions for Colors
    classDef cdicolor fill:#f9f,stroke:#333,stroke-width:2px;
    classDef container fill:#ccf,stroke:#333,stroke-width:2px;
    classDef rmodule fill:#cfc,stroke:#333,stroke-width:2px;
    classDef config fill:#cff,stroke:#333,stroke-width:2px;
    classDef external fill:#fc9,stroke:#333,stroke-width:2px;
    classDef test fill:#ffc,stroke:#333,stroke-width:2px;

    %% Click Events
    click GA "https://github.com/worldfishcenter/peskas.timor.data.pipeline/tree/main/.github/workflows/"
    click D1 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/tree/main/Dockerfile"
    click D2 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/Dockerfile.prod"
    click D3 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/docker-compose.yaml"
    click CP "https://github.com/worldfishcenter/peskas.timor.data.pipeline/tree/main/R/"
    click CM "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/inst/conf.yml"
    click ES1 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/R/cloud-storage.R"
    click ES2 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/R/google-drive.R"
    click ES3 "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/R/export-dataverse.R"
    click T "https://github.com/worldfishcenter/peskas.timor.data.pipeline/tree/main/tests/"
    click DOC "https://github.com/worldfishcenter/peskas.timor.data.pipeline/tree/main/man/"
    click PKG "https://github.com/worldfishcenter/peskas.timor.data.pipeline/blob/main/_pkgdown.yml"
```
