# ardandxb/data-pipelines architecture

> Architecture diagram of the GitHub repository ardandxb/data-pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ardandxb/data-pipelines
- Repository: https://github.com/ardandxb/data-pipelines

## Overview

##### Main Components
- Prefect Cloud: This is the orchestration engine where deployments are scheduled. Indicate it as the central control plane for scheduling flow-runs.
- Prefect Agent: Represent this component as the worker that polls for flow runs. Note that it is containerized (with a Dockerfile in the agent/image directory) and deployed via Kubernetes (agent/k8s directory contains deployment configurations for multiple environments).
- Data Flows: The flows are written in Python (located in the flows directory). They encompass various tasks like data pulling, processing (e.g., raw_verra_data.py, build_all.py) and managing artifacts.
- GitHub Repository: Highlight that the flows are stored in GitHub. The agents “download” the flow code from GitHub when executing tasks.
- Artifact Storage (S3 or local): Separate components that manage the storage of processed artifacts (referenced in flows, via environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.).
- CI/CD Pipeline: The .github/workflows directory shows that GitHub Actions is used for both continuous integration (ci.yaml) and deployments (deploy.yaml). This pipeline automates tasks like uploading deployments to Prefect Cloud.

##### Relationships and Interactions
- Prefect Cloud schedules flow-runs which are then picked up by the Prefect agents.
- Agents, running in Docker containers on Kubernetes clusters, download the latest flow code from the GitHub repository.
- Once running, flows perform operations such as extracting data, processing metrics (carbon metrics, offsets, etc.), and interacting with external storage like S3.
- CI/CD pipelines ensure that changes in the repository (both for flows and the agent image) are tested, built, and deployed into the appropriate environment (dev/prod).
- There is a bidirectional flow of configuration information: environment variables or .env files in the flows directory allow for the customization of execution behavior.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The repository explicitly separates deployment/agent code (agent directory) from business logic/data flows (flows directory).
- Containerization and Orchestration: The use of Docker (agent/image/Dockerfile) and Kubernetes manifests (agent/k8s) indicates that deployment is containerized and orchestrated.
- CI/CD: Automated testing/deployment using GitHub Actions highlights Continuous Integration and Delivery practices.
- Configurability: Environment variables and templated configuration files (like .env.dist and multiple kustomization.yaml files) allow for flexible deployment to different environments (development and production).

## Diagram (Mermaid)

```mermaid
flowchart TD
    PC["Prefect Cloud Scheduler"]:::cloud
    PA["Dockerized Prefect Agent"]:::container
    GR["GitHub Repository (Flow Code)"]:::process
    DF["Data Flows (Python Scripts)"]:::process
    S3["Artifact Storage (S3)"]:::storage
    CI["CI/CD Pipeline"]:::cicd
    KD["Kubernetes Deployments"]:::container

    %% Kubernetes Deployment Subgraph
    subgraph "Kubernetes Deployments"
        DEV["Dev Environment"]:::container
        PROD["Prod Environment"]:::container
    end

    %% Relationships
    PC -->|"schedules"| PA
    PA -->|"downloads_code"| GR
    PA -->|"executes"| DF
    DF -->|"updates"| S3
    CI -->|"triggers_build_deploy"| PA
    CI -->|"deploys_to"| KD
    PA -->|"builds_image_for"| KD

    %% Click Events
    click PA "https://github.com/ardandxb/data-pipelines/tree/main/agent/"
    click GR "https://github.com/ardandxb/data-pipelines/tree/main/flows/"
    click DF "https://github.com/ardandxb/data-pipelines/tree/main/flows/"
    click S3 "https://github.com/ardandxb/data-pipelines/blob/main/flows/.env.dist"
    click CI "https://github.com/ardandxb/data-pipelines/tree/main/.github/workflows/"
    click KD "https://github.com/ardandxb/data-pipelines/tree/main/agent/k8s/"
    click DEV "https://github.com/ardandxb/data-pipelines/tree/main/agent/k8s/dev"
    click PROD "https://github.com/ardandxb/data-pipelines/tree/main/agent/k8s/prod"

    %% Styles
    classDef cloud fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef container fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef process fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef storage fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef cicd fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
