# prefecthq/prefect-recipes architecture

> Architecture diagram of the GitHub repository prefecthq/prefect-recipes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prefecthq/prefect-recipes
- Repository: https://github.com/prefecthq/prefect-recipes

## Overview

##### Project Type & Purpose
- This is an open-source “recipes” repository for Prefect 2.0: a collection of reusable workflow patterns, example flows, configuration helpers, infrastructure-as-code modules, and CI/CD pipelines.
- It is not a classic full-stack web app, but rather a monorepo exposing:
- Python flow definitions under flows-starter and flows-advanced
- Configuration utilities (e.g., dot-env secrets loader)
- DevOps modules (Terraform for AWS/Azure/GCP, Docker containers, Helm charts)
- GitHub Actions workflows for build & deploy
- Documentation under docs/

##### File-Structure Analysis
- Top directories:
- configuration/ — helper scripts to load secrets
- devops/ — IaC (Terraform), Docker containers, GitHub Actions YAML
- flows-starter/ & flows-advanced/ — Python examples (ETL, ML, serverless, event-driven)
- docs/ — MkDocs configuration and markdown guides
- prefect-v1-legacy/ — legacy Prefect 1.0 examples (can be shaded out)
- CI config: .github/workflows/python.yml, publish-docs.yml, Terraform sanity checks, docker build workflows
- Package metadata: setup.py, requirements.txt, mkdocs.yml

##### Key Insights from README
- Central theme: “recipes” illustrating typical Prefect use cases, from basic flows to cloud deployment
- Heavily oriented toward Infrastructure as Code (Terraform modules per cloud), containerization, and GitHub Actions
- Audience: contributors & end users seeking patterns to integrate Prefect with AWS/Azure/GCP, DBs, S3, Airbyte, Snowflake, etc.

- Code Repository (monorepo)
- CI/CD Layer (GitHub Actions)
- Container Registry (GCP Artifact Registry / AWS ECR)
- I aC Modules (Terraform for AWS, Azure, GCP; Helm charts)
- Runtime Orchestrator (Prefect Orion server + UI)
- Agents (ECS/Fargate, AKS, GKE, VM)
- Flow Definitions (Python scripts under flows-starter & flows-advanced)
- Configuration Blocks (dot-env, Secrets Blocks)
- External Services:
– Object storage (S3, Blob Storage, GCS)
– Data warehouses (Snowflake, dbt)
– Data connectors (Airbyte)
- Monitoring & Observability (Prometheus, Grafana via Helm)

1. Developer commits → Code Repo
2. Push → Triggers GitHub Actions →
– Docker build → push image → Container Registry
– Terraform plan/apply → provision Prefect server + agent infra
– MkDocs build → publish docs
3. Prefect Orion registers new deployments pointing to container images & flow code
4. Agents poll Orion → pull container image → execute flows
5. Flows retrieve configuration from Blocks (dot-env secrets) → connect to external services (S3, Snowflake, Airbyte)
6. Metrics / Logs → Monitoring stack (Prometheus & Grafana)

c. Architectural Patterns & Principles:
- Modular separation: code vs. infrastructure vs. examples vs. docs
- Infrastructure as Code (reusable Terraform modules for multiple clouds)
- Containerize workloads for portability
- CI/CD for automated build, test, deploy, and doc publishing
- Extensibility: recipes structured as plug-and-play patterns

d. Key Technologies & Frameworks (annotate on each component):
– GitHub Actions (YAML workflows)
– Docker & Dockerfile
– Terraform (HCL modules under devops/infrastructure-as-code)
– Helm charts (prefect2-agent)
– Prefect 2.0 (Orion server, deployments, blocks)
– Python (flow definitions, dot-env loader)
– Cloud Providers: AWS (ECS, EKS, S3, IAM), Azure (AKS, VM, Blob), GCP (GKE, MIG)
– Monitoring: Prometheus + Grafana Helm chart

## Diagram (Mermaid)

```mermaid
graph TB
    %% Monorepo
    subgraph "Monorepo" 
        codeRepo["Code Repository"]:::code
        packageMeta["Package Metadata & Config"]:::code
        docs["Documentation"]:::code
    end

    %% CI/CD Layer
    subgraph "CI/CD Layer" 
        githubActions["GitHub Actions"]:::ci
    end

    %% DevOps - Infrastructure as Code
    subgraph "DevOps (IaC)" 
        AWSModule["Terraform AWS Modules"]:::infra
        AzureModule["Terraform Azure Modules"]:::infra
        GCPModule["Terraform GCP Modules"]:::infra
        HelmCharts["Helm Charts (Prefect Agent)"]:::infra
    end

    %% Container Build & Registry
    subgraph "Container Build & Registry"
        DockerScripts["Docker Deployment Scripts"]:::ci
        workflowGCP["Docker Build & Push (GCP)"]:::ci
        workflowAWS["Deploy to AWS ECR"]:::ci
        ContainerGCP["GCP Artifact Registry"]:::infra
        ContainerAWS["AWS ECR"]:::infra
    end

    %% Prefect Runtime
    subgraph "Prefect Runtime" 
        PrefectOrion["Prefect Orion Server"]:::runtime
        subgraph "Agents"
            ECSAgentInfra["ECS Agent Infra"]:::runtime
            ECSWorkerInfra["ECS Worker Infra"]:::runtime
            AKSAgentInfra["AKS Agent Infra"]:::runtime
            VMAgentInfra["VM Agent Infra"]:::runtime
            MIGAgentInfra["GCP MIG Agent Infra"]:::runtime
        end
    end

    %% Flows Definitions
    subgraph "Flows" 
        flowsStarter["Flows Starter Examples"]:::code
        flowsAdvanced["Flows Advanced Examples"]:::code
    end

    %% Configuration Utilities
    subgraph "Configuration"
        config["Dot-Env Secrets Loader"]:::config
    end

    %% External Services
    subgraph "External Services"
        S3["S3 / Blob / GCS"]:::external
        Snowflake["Snowflake"]:::external
        Airbyte["Airbyte"]:::external
    end

    %% Monitoring
    monitoring["Prometheus & Grafana"]:::monitoring

    %% Connections
    codeRepo -->|"push"| githubActions
    githubActions -->|"build docker"| DockerScripts
    githubActions -->|"push to registry"| workflowGCP
    githubActions -->|"push to registry"| workflowAWS
    githubActions -->|"terraform apply"| AWSModule
    githubActions -->|"terraform apply"| AzureModule
    githubActions -->|"terraform apply"| GCPModule
    githubActions -->|"deploy helm charts"| HelmCharts
    githubActions -->|"publish docs"| docs

    AWSModule --> OrionInfraProvision["Prefect Orion Infra"]
    AWSModule --> ECSAgentInfra
    AWSModule --> ECSWorkerInfra
    AzureModule --> AKSAgentInfra
    AzureModule --> VMAgentInfra
    GCPModule --> MIGAgentInfra

    DockerScripts --> ContainerGCP
    DockerScripts --> ContainerAWS
    workflowGCP --> ContainerGCP
    workflowAWS --> ContainerAWS

    PrefectOrion -->|"registers deployments"| flowsStarter
    PrefectOrion -->|"registers deployments"| flowsAdvanced
    PrefectOrion -->|"schedules runs"| ECSAgentInfra
    PrefectOrion -->|"schedules runs"| AKSAgentInfra
    PrefectOrion -->|"schedules runs"| VMAgentInfra
    PrefectOrion -->|"schedules runs"| MIGAgentInfra

    ECSAgentInfra -->|"exec flows"| config
    ECSWorkerInfra -->|"exec flows"| config
    AKSAgentInfra -->|"exec flows"| config
    VMAgentInfra -->|"exec flows"| config
    MIGAgentInfra -->|"exec flows"| config

    config -->|"provides secrets"| S3
    config -->|"provides secrets"| Snowflake
    config -->|"provides secrets"| Airbyte

    PrefectOrion -->|"metrics"| monitoring
    ECSAgentInfra -->|"logs & metrics"| monitoring
    AKSAgentInfra -->|"logs & metrics"| monitoring
    VMAgentInfra -->|"logs & metrics"| monitoring
    MIGAgentInfra -->|"logs & metrics"| monitoring

    %% Click Events
    click codeRepo "https://github.com/prefecthq/prefect-recipes/tree/main//"
    click githubActions "https://github.com/prefecthq/prefect-recipes/tree/main/.github/workflows/"
    click DockerScripts "https://github.com/prefecthq/prefect-recipes/tree/main/devops/deployments/docker/"
    click workflowGCP "https://github.com/prefecthq/prefect-recipes/blob/main/devops/github-actions/docker-build-push-gcp-artifact-registry.yaml"
    click workflowAWS "https://github.com/prefecthq/prefect-recipes/blob/main/devops/github-actions/prefect-deploy-aws-ecr.yaml"
    click AWSModule "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/"
    click AzureModule "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/azure/"
    click GCPModule "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/gcp/"
    click HelmCharts "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/helm/prefect2-agent/"
    click OrionInfraProvision "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/orion-server/"
    click ECSAgentInfra "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent/"
    click ECSWorkerInfra "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-worker/"
    click AKSAgentInfra "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/azure/prefect-agent-on-aks/"
    click VMAgentInfra "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/azure/prefect-agent-on-avm/"
    click MIGAgentInfra "https://github.com/prefecthq/prefect-recipes/tree/main/devops/infrastructure-as-code/gcp/prefect2-mig/"
    click config "https://github.com/prefecthq/prefect-recipes/tree/main/configuration/dot-env-secrets/"
    click flowsStarter "https://github.com/prefecthq/prefect-recipes/tree/main/flows-starter/"
    click flowsAdvanced "https://github.com/prefecthq/prefect-recipes/tree/main/flows-advanced/"
    click docs "https://github.com/prefecthq/prefect-recipes/tree/main/docs/"

    %% Styles
    classDef code fill:#D0E1FF,stroke:#0E4DA4,color:#0E4DA4
    classDef ci fill:#E8D5FF,stroke:#6B3099,color:#6B3099
    classDef infra fill:#D0FFD8,stroke:#1F7A1F,color:#1F7A1F
    classDef runtime fill:#FFE5B4,stroke:#CC7A00,color:#CC7A00
    classDef config fill:#FFF3B0,stroke:#D39E00,color:#D39E00
    classDef external fill:#F0F0F0,stroke:#666666,color:#666666
    classDef monitoring fill:#FFD1D1,stroke:#A80000,color:#A80000
```
