How worldfishcenter/peskas.timor.data.pipeline works
Project Type and Purpose
• This is an R package structured as a data pipeline. Its purpose is to implement, deploy, and execute data ingestion, cleaning, preprocessing, modeling, and reporting workflows for small-scale fisheries analytics.
• The README outlines that the pipeline is deployed as a series of interconnected GitHub Actions jobs, orchestrating containerized execution via Docker. It also emphasizes the use of environment-specific configurations (remote, local, production) and cloud storage as storage for versioned artifacts.
Analyzing the File Structure
• Top-level files indicate an R package: DESCRIPTION, NAMESPACE, documentation (man/), and R/ directories contain function files (e.g., calculate-nutrients.R, ingest-landings.R, etc.).
• The presence of GitHub Actions workflows under .github/workflows (e.g., data-pipeline.yaml, data-report.yaml, etc.) shows the CI/CD pipeline is integral.
• Dockerfiles (Dockerfile and Dockerfile.prod) and docker-compose.yaml indicate separate containerized environments for development and production.
• Inst/ directory holds auxiliary code and configuration—for example, inst/conf.yml (the environment configuration file) and scripts (e.g., kepler_mapper.py, which suggests auxiliary mapping/reporting functionality).
a. Main Components:
• Core Pipeline Functions: Represented by various R scripts (e.g., ingestion, cleaning, preprocessing, merging, modeling, and validation functions). This is the heart of the R package.
• GitHub Actions Workflows: Each YAML file under .github/workflows represents a distinct job in the CI/CD pipeline (e.g., data ingestion, data reporting, uploading artifacts, etc.).
• Docker Containers: There are two primary container definitions—one for development (Dockerfile) running an RStudio server via rocker/geospatial and one for production (Dockerfile.prod) for lightweight execution.
• Configuration Management: The inst/conf.yml file used by the config package to determine parameters based on the environment (local/cloud/production).
• External Services and Cloud Storage: The pipeline stores outputs (artifacts) in cloud storage containers; this component links to external APIs (Google Drive, possibly Dataverse, cloud storage authentication via environment variables, etc.).
• Testing and Documentation: Presence of tests/ and pkgdown configurations indicates that automated testing and website documentation are also integral parts of the architecture.
b. Relationships and Interactions:
• The workflow begins with the GitHub Actions triggering the job (e.g., data-pipeline.yaml). This workflow calls R functions (each implemented as separate scripts) that perform tasks such as data ingestion and processing.
• The execution environment is defined by Docker containers. The development container supports interactive coding (RStudio) while the production container supports efficient job execution.
• Configuration is loaded at runtime using the config package. Based on the R_CONFIG_ACTIVE environment variable, the pipeline selects between remote, local, or production configurations.
• Artifacts produced in one stage are versioned (with add_version()) and are then stored in cloud storage for subsequent jobs to retrieve (using cloud_object_name()).
• Logging is handled by the logger package to record events, especially in production.
c. Architectural Patterns and Design Principles:
• Modular Function Decomposition: The pipeline is broken down into many single-responsibility functions or “stages” (e.g., ingest, preprocess, validate, model). This promotes code reuse and easier testing.
• Containerization: Using Docker for both development and production environments ensures consistency across deployments.
• Environment-Based Configuration: Decoupling settings and secret management by switching configurations based on the runtime environment.
• CI/CD Pipeline Automation: GitHub Actions orchestrate the sequential execution of tasks (with dependencies clearly defined between them).
d. Relevant Technologies, Frameworks, and Libraries:
• R and Tidyverse: For data manipulation and standard coding practices.
• GitHub Actions: For automating builds, tests, and deployments.
• Docker: For containerizing the application.
• config package: For environment-specific configuration management.
• logger package: For logging production events.
• Additional utilities: The usethis package to facilitate setup, and pkgdown for generating documentation.