# 52north/pygeoapi_k8s-manager architecture

> Architecture diagram of the GitHub repository 52north/pygeoapi_k8s-manager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/52north/pygeoapi_k8s-manager
- Repository: https://github.com/52north/pygeoapi_k8s-manager

## Overview

1. **What this repository is**
   - This is not a standalone application; it is a Python extension for **pygeoapi** that replaces/extends pygeoapi’s manager layer with Kubernetes-backed execution.
   - The core package lives under `src/pygeoapi_k8s_manager/` and is wired into pygeoapi via the `server.manager` configuration in `pygeoapi-config.yaml`.

2. **Top-level architecture**
   - The system has three main runtime parts:
     - a **pygeoapi-facing manager** that receives process execution requests,
     - a **Kubernetes job execution layer** that runs containerized workloads,
     - an optional **finalizer controller** that watches finished jobs, extracts results/logs, and persists them to object storage.
   - The repository also includes Kubernetes deployment manifests and a kind-based local cluster setup for development.

3. **Request and execution flow**
   - A client calls a pygeoapi process endpoint.
   - `KubernetesManager` (`src/pygeoapi_k8s_manager/manager.py`) translates that request into a Kubernetes Job.
   - The job runs the selected container image/process wrapper in the cluster.
   - The manager and job lifecycle are coupled through Kubernetes API objects and pod/job status.

4. **Process subsystem**
   - `src/pygeoapi_k8s_manager/process/` contains the process implementations:
     - `generic_image.py`: a generic wrapper that can execute arbitrary container images on Kubernetes.
     - `hello_world_k8s.py`: a sample/test process used by the repository’s tests and usage examples.
   - These processes are the bridge between pygeoapi process inputs and the Kubernetes job spec.

5. **Input and authentication model**
   - Dynamic inputs are passed into job pods through the environment variable `PYGEOAPI_K8S_MANAGER_INPUTS`.
   - Each job pod also receives `PYGEOAPI_JOB_ID`, which ties the Kubernetes execution back to the pygeoapi job identity.
   - `GenericImageProcessor` requires an auth token input named `token` by default, validated against `PYGEOAPI_K8S_MANAGER_API_TOKEN`; this can be disabled or overridden per processor.

6. **Output/result handling**
   - The repository uses log markers to encode result metadata and payload:
     - `PYGEOAPI_K8S_MANAGER_RESULT_MIMETYPE`
     - `PYGEOAPI_K8S_MANAGER_RESULT_START`
   - This means the result is not returned directly from the job container through an API channel; instead, the finalizer parses logs to reconstruct the result content and content type.

7. **Finalizer controller and storage boundary**
   - `src/pygeoapi_k8s_manager/finalizer.py` implements `KubernetesFinalizerController`.
   - It watches job state, fetches logs, extracts the final result, and uploads log/result artifacts to an **S3-compatible bucket**.
   - Required storage configuration is supplied through `PYGEOAPI_K8S_MANAGER_FINALIZER_BUCKET_*` environment variables.
   - If finalizer configuration is missing or invalid, cleanup can be blocked, leaving Kubernetes resources for manual intervention.

8. **Kubernetes infrastructure and permissions**
   - `k8s-manifests/` contains the production deployment pieces: deployment, service, ingress, configmap, and RBAC permissions.
   - The README’s RBAC table is important architecturally: the manager needs access to jobs, pods, pod logs/status, and events.
   - `k8s-kind/` provides a separate local cluster definition plus a MinIO-backed setup for testing the finalizer/storage path in kind.

9. **Configuration and deployment model**
   - `pygeoapi-config.yaml` is the main integration point showing how pygeoapi loads this manager and enables the finalizer controller.
   - `Dockerfile` packages the extension into a container based on pygeoapi, and the project is versioned independently from pygeoapi.
   - The repo is designed to be deployed as a containerized service inside Kubernetes, not as a library consumed only in-process.

10. **Testing and quality gates**
   - `tests/` covers manager behavior, utility functions, finalizer logic, and the two process implementations.
   - `.github/workflows/` contains CI for build, lint/format/test, and stale issue handling.
   - This suggests the main integration risks are around Kubernetes job orchestration, result parsing from logs, and finalizer behavior rather than UI or HTTP routing.

11. **Operational characteristics**
   - The architecture is Kubernetes-native and event-driven at the cluster boundary: execution happens via Jobs, status is observed from the API, and cleanup/persistence is tied to finalizer behavior.
   - Sensitive data includes the API token and potentially log contents, which is why the finalizer bucket is explicitly expected to be private.

12. **How to diagram it**
   - Draw pygeoapi as the external API surface.
   - Show `KubernetesManager` as the central orchestration component.
   - Show a Kubernetes Job/pod execution box fed by env vars (`PYGEOAPI_K8S_MANAGER_INPUTS`, `PYGEOAPI_JOB_ID`).
   - Show a side path from job logs/status to `KubernetesFinalizerController`, then to S3-compatible object storage.
   - Add Kubernetes cluster resources/RBAC as the infrastructure boundary around the manager and job execution.

Main entry points: [`src/pygeoapi_k8s_manager/manager.py`](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/manager.py) (Kubernetes Manager), [`src/pygeoapi_k8s_manager/process/__init__.py`](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/__init__.py) (Process API), [`src/pygeoapi_k8s_manager/finalizer.py`](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/finalizer.py) (Finalizer), [`pygeoapi-config.yaml`](https://github.com/52north/pygeoapi_k8s-manager/blob/main/pygeoapi-config.yaml) (Config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_integration["pygeoapi integration"]
  node_pygeoapi(("pygeoapi<br/>external API"))
  node_manager["Kubernetes Manager<br/>process manager<br/>[manager.py]"]
  node_process_api["Process API<br/>process bridge<br/>[__init__.py]"]
  node_generic_image["Generic Image<br/>container process<br/>[generic_image.py]"]
  node_hello_world["Hello World<br/>sample process<br/>[hello_world_k8s.py]"]
  node_util["Utilities<br/>helpers<br/>[util.py]"]
end

subgraph group_runtime["Kubernetes runtime"]
  node_k8s_api[("K8s API<br/>cluster control")]
  node_job_pods["Job Pods<br/>job runtime"]
  node_inputs_env["Inputs env<br/>env contract"]
end

subgraph group_ops["Finalization and storage"]
  node_finalizer["Finalizer<br/>log watcher<br/>[finalizer.py]"]
  node_log_markers["Log markers<br/>result protocol"]
  node_bucket[("Object Storage<br/>s3-compatible bucket")]
end

subgraph group_delivery["Deployment and local dev"]
  node_config["Config"]
  node_deployment["Deployment<br/>k8s manifest<br/>[deployment.yaml]"]
  node_permissions["RBAC<br/>k8s permissions<br/>[permissions.yaml]"]
  node_service_ingress["Service/Ingress<br/>exposure<br/>[service.yaml]"]
  node_kind_cluster["Kind Setup<br/>local cluster<br/>[kind-cluster.yaml]"]
  node_minio[("MinIO<br/>local storage<br/>[minio.yaml]")]
end

node_pygeoapi -->|"process request"| node_manager
node_config -->|"loads"| node_manager
node_config -->|"enables"| node_finalizer
node_manager -->|"dispatches"| node_process_api
node_process_api -->|"implements"| node_generic_image
node_process_api -->|"implements"| node_hello_world
node_process_api -.->|"uses"| node_util
node_manager -->|"creates jobs"| node_k8s_api
node_manager -->|"requires"| node_permissions
node_permissions -->|"authorizes"| node_k8s_api
node_k8s_api -->|"runs"| node_job_pods
node_job_pods -->|"receives"| node_inputs_env
node_job_pods -->|"emits"| node_log_markers
node_job_pods -->|"status/logs"| node_finalizer
node_finalizer -->|"uploads"| node_bucket
node_finalizer -->|"watches"| node_k8s_api
node_deployment -->|"deploys"| node_manager
node_service_ingress -->|"exposes"| node_manager
node_kind_cluster -->|"includes"| node_minio
node_kind_cluster -->|"hosts"| node_manager
node_minio -.->|"mimics"| node_bucket

click node_config "https://github.com/52north/pygeoapi_k8s-manager/blob/main/pygeoapi-config.yaml"
click node_manager "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/manager.py"
click node_process_api "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/__init__.py"
click node_generic_image "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/generic_image.py"
click node_hello_world "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/hello_world_k8s.py"
click node_util "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/util.py"
click node_finalizer "https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/finalizer.py"
click node_deployment "https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/deployment.yaml"
click node_permissions "https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/permissions.yaml"
click node_service_ingress "https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/service.yaml"
click node_kind_cluster "https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-kind/kind-cluster.yaml"
click node_minio "https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-kind/minio.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pygeoapi,node_manager,node_process_api,node_generic_image,node_hello_world,node_util toneBlue
class node_k8s_api,node_job_pods,node_inputs_env toneAmber
class node_finalizer,node_log_markers,node_bucket toneMint
class node_config,node_deployment,node_permissions,node_service_ingress,node_kind_cluster,node_minio toneRose
```

## Components

### pygeoapi integration

pygeoapi-facing manager and process bridge

- **pygeoapi**: external API · Upstream process API surface that invokes the manager
- [Kubernetes Manager](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/manager.py): process manager · Translates pygeoapi process requests into Kubernetes jobs · `src/pygeoapi_k8s_manager/manager.py`
- [Process API](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/__init__.py): process bridge · Shared process interface used by the manager to resolve implementations · `src/pygeoapi_k8s_manager/process/__init__.py`
- [Generic Image](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/generic_image.py): container process · Runs arbitrary container images as Kubernetes-backed processes · `src/pygeoapi_k8s_manager/process/generic_image.py`
- [Hello World](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/process/hello_world_k8s.py): sample process · Example process used in tests and usage examples · `src/pygeoapi_k8s_manager/process/hello_world_k8s.py`
- [Utilities](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/util.py): helpers · Shared helpers for request and execution plumbing · `src/pygeoapi_k8s_manager/util.py`

### Kubernetes runtime

Execution and cluster-facing control path

- **K8s API**: cluster control · Kubernetes API used to create and observe jobs, pods, and events
- **Job Pods**: job runtime · Containerized workload execution inside the cluster
- **Inputs env**: env contract · Carries dynamic inputs and job identity into pods

### Finalization and storage

Log parsing and artifact persistence

- [Finalizer](https://github.com/52north/pygeoapi_k8s-manager/blob/main/src/pygeoapi_k8s_manager/finalizer.py): log watcher · Watches completed jobs, extracts results from logs, and persists artifacts · `src/pygeoapi_k8s_manager/finalizer.py`
- **Log markers**: result protocol · Encodes result mimetype and payload boundaries in container logs
- **Object Storage**: s3-compatible bucket · Private storage for results and logs written by the finalizer

### Deployment and local dev

Manifests, config, and cluster setup

- [Config](https://github.com/52north/pygeoapi_k8s-manager/blob/main/pygeoapi-config.yaml): pygeoapi config · Enables the custom manager and finalizer integration · `pygeoapi-config.yaml`
- [Deployment](https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/deployment.yaml): k8s manifest · Runs the manager as a Kubernetes service · `k8s-manifests/deployment.yaml`
- [RBAC](https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/permissions.yaml): k8s permissions · Grants access to jobs, pods, logs, statuses, and events · `k8s-manifests/permissions.yaml`
- [Service/Ingress](https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-manifests/service.yaml): exposure · Exposes the manager inside or outside the cluster · `k8s-manifests/service.yaml`
- [Kind Setup](https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-kind/kind-cluster.yaml): local cluster · Local Kubernetes environment for development and validation · `k8s-kind/kind-cluster.yaml`
- [MinIO](https://github.com/52north/pygeoapi_k8s-manager/blob/main/k8s-kind/minio.yaml): local storage · Kind-backed object storage used to exercise the finalizer path · `k8s-kind/minio.yaml`

## Connections

- pygeoapi → Kubernetes Manager: process request · pygeoapi invokes the custom manager for process execution
- Config → Kubernetes Manager: loads · pygeoapi configuration wires in the Kubernetes manager
- Config → Finalizer: enables · configuration activates the optional finalizer controller
- Kubernetes Manager → Process API: dispatches · Manager resolves and runs process implementations
- Process API → Generic Image: implements · Generic container-backed process implementation
- Process API → Hello World: implements · Sample process implementation
- Process API → Utilities: uses · Shared helper functions support process behavior
- Kubernetes Manager → K8s API: creates jobs · Manager talks to Kubernetes to create and monitor jobs
- Kubernetes Manager → RBAC: requires · RBAC boundary needed for cluster operations
- RBAC → K8s API: authorizes · Grants access to job, pod, log, and event resources
- K8s API → Job Pods: runs · Kubernetes schedules the job pod that executes the workload
- Job Pods → Inputs env: receives · Inputs and job id are injected through environment variables
- Job Pods → Log markers: emits · Result metadata and payload are written to logs using markers
- Job Pods → Finalizer: status/logs · Completed jobs are observed and inspected by the finalizer
- Finalizer → Object Storage: uploads · Result and log artifacts are persisted to object storage
- Finalizer → K8s API: watches · Finalizer monitors job state and fetches logs through the API
- Deployment → Kubernetes Manager: deploys · Kubernetes deployment runs the service container
- Service/Ingress → Kubernetes Manager: exposes · Service and ingress provide network access to the manager
- Kind Setup → MinIO: includes · Local development cluster includes storage for finalizer testing
- Kind Setup → Kubernetes Manager: hosts · Kind provides the local runtime for the manager stack
- MinIO → Object Storage: mimics · Local object storage stands in for the production bucket
