How ardandxb/data-pipelines works
Main Components
• Prefect Cloud: This is the orchestration engine where deployments are scheduled. Indicate it as the central control plane for scheduling flow-runs.
• Prefect Agent: Represent this component as the worker that polls for flow runs. Note that it is containerized (with a Dockerfile in the agent/image directory) and deployed via Kubernetes (agent/k8s directory contains deployment configurations for multiple environments).
• Data Flows: The flows are written in Python (located in the flows directory). They encompass various tasks like data pulling, processing (e.g., raw_verra_data.py, build_all.py) and managing artifacts.
• GitHub Repository: Highlight that the flows are stored in GitHub. The agents “download” the flow code from GitHub when executing tasks.
• Artifact Storage (S3 or local): Separate components that manage the storage of processed artifacts (referenced in flows, via environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.).
• CI/CD Pipeline: The .github/workflows directory shows that GitHub Actions is used for both continuous integration (ci.yaml) and deployments (deploy.yaml). This pipeline automates tasks like uploading deployments to Prefect Cloud.
Relationships and Interactions
• Prefect Cloud schedules flow-runs which are then picked up by the Prefect agents.
• Agents, running in Docker containers on Kubernetes clusters, download the latest flow code from the GitHub repository.
• Once running, flows perform operations such as extracting data, processing metrics (carbon metrics, offsets, etc.), and interacting with external storage like S3.
• CI/CD pipelines ensure that changes in the repository (both for flows and the agent image) are tested, built, and deployed into the appropriate environment (dev/prod).
• There is a bidirectional flow of configuration information: environment variables or .env files in the flows directory allow for the customization of execution behavior.
Architectural Patterns and Design Principles
• Separation of Concerns: The repository explicitly separates deployment/agent code (agent directory) from business logic/data flows (flows directory).
• Containerization and Orchestration: The use of Docker (agent/image/Dockerfile) and Kubernetes manifests (agent/k8s) indicates that deployment is containerized and orchestrated.
• CI/CD: Automated testing/deployment using GitHub Actions highlights Continuous Integration and Delivery practices.
• Configurability: Environment variables and templated configuration files (like .env.dist and multiple kustomization.yaml files) allow for flexible deployment to different environments (development and production).