How 52north/pygeoapi_k8s-manager works
1. What this repository is
- This is not a standalone application; it is a Python extension for pygeoapi that replaces/extends pygeoapi’s manager layer with Kubernetes-backed execution.
- The core package lives under src/pygeoapi_k8s_manager/ and is wired into pygeoapi via the server.manager configuration in pygeoapi-config.yaml.
2. Top-level architecture
- The system has three main runtime parts:
- a pygeoapi-facing manager that receives process execution requests,
- a Kubernetes job execution layer that runs containerized workloads,
- an optional finalizer controller that watches finished jobs, extracts results/logs, and persists them to object storage.
- The repository also includes Kubernetes deployment manifests and a kind-based local cluster setup for development.
3. Request and execution flow
- A client calls a pygeoapi process endpoint.
- KubernetesManager (src/pygeoapi_k8s_manager/manager.py) translates that request into a Kubernetes Job.
- The job runs the selected container image/process wrapper in the cluster.
- The manager and job lifecycle are coupled through Kubernetes API objects and pod/job status.
4. Process subsystem
- src/pygeoapi_k8s_manager/process/ contains the process implementations:
- generic_image.py: a generic wrapper that can execute arbitrary container images on Kubernetes.
- hello_world_k8s.py: a sample/test process used by the repository’s tests and usage examples.
- These processes are the bridge between pygeoapi process inputs and the Kubernetes job spec.
5. Input and authentication model
- Dynamic inputs are passed into job pods through the environment variable PYGEOAPI_K8S_MANAGER_INPUTS.
- Each job pod also receives PYGEOAPI_JOB_ID, which ties the Kubernetes execution back to the pygeoapi job identity.
- GenericImageProcessor requires an auth token input named token by default, validated against PYGEOAPI_K8S_MANAGER_API_TOKEN; this can be disabled or overridden per processor.
6. Output/result handling
- The repository uses log markers to encode result metadata and payload:
- PYGEOAPI_K8S_MANAGER_RESULT_MIMETYPE
- PYGEOAPI_K8S_MANAGER_RESULT_START
- This means the result is not returned directly from the job container through an API channel; instead, the finalizer parses logs to reconstruct the result content and content type.
7. Finalizer controller and storage boundary
- src/pygeoapi_k8s_manager/finalizer.py implements KubernetesFinalizerController.
- It watches job state, fetches logs, extracts the final result, and uploads log/result artifacts to an S3-compatible bucket.
- Required storage configuration is supplied through PYGEOAPI_K8S_MANAGER_FINALIZER_BUCKET_* environment variables.
- If finalizer configuration is missing or invalid, cleanup can be blocked, leaving Kubernetes resources for manual intervention.
8. Kubernetes infrastructure and permissions
- k8s-manifests/ contains the production deployment pieces: deployment, service, ingress, configmap, and RBAC permissions.
- The README’s RBAC table is important architecturally: the manager needs access to jobs, pods, pod logs/status, and events.
- k8s-kind/ provides a separate local cluster definition plus a MinIO-backed setup for testing the finalizer/storage path in kind.
9. Configuration and deployment model
- pygeoapi-config.yaml is the main integration point showing how pygeoapi loads this manager and enables the finalizer controller.
- Dockerfile packages the extension into a container based on pygeoapi, and the project is versioned independently from pygeoapi.
- The repo is designed to be deployed as a containerized service inside Kubernetes, not as a library consumed only in-process.
10. Testing and quality gates
- tests/ covers manager behavior, utility functions, finalizer logic, and the two process implementations.
- .github/workflows/ contains CI for build, lint/format/test, and stale issue handling.
- This suggests the main integration risks are around Kubernetes job orchestration, result parsing from logs, and finalizer behavior rather than UI or HTTP routing.
11. Operational characteristics
- The architecture is Kubernetes-native and event-driven at the cluster boundary: execution happens via Jobs, status is observed from the API, and cleanup/persistence is tied to finalizer behavior.
- Sensitive data includes the API token and potentially log contents, which is why the finalizer bucket is explicitly expected to be private.
12. How to diagram it
- Draw pygeoapi as the external API surface.
- Show KubernetesManager as the central orchestration component.
- Show a Kubernetes Job/pod execution box fed by env vars (PYGEOAPI_K8S_MANAGER_INPUTS, PYGEOAPI_JOB_ID).
- Show a side path from job logs/status to KubernetesFinalizerController, then to S3-compatible object storage.
- Add Kubernetes cluster resources/RBAC as the infrastructure boundary around the manager and job execution.
Main entry points: manager.py Kubernetes Manager · __init__.py Process API · finalizer.py Finalizer · pygeoapi-config.yaml Config
Components 18
pygeoapi integration
pygeoapi-facing manager and process bridge
- pygeoapi · Upstream process API surface that invokes the manager
- Kubernetes Manager · Translates pygeoapi process requests into Kubernetes jobs
src/pygeoapi_k8s_manager/manager.py - Process API · Shared process interface used by the manager to resolve implementations
src/pygeoapi_k8s_manager/process/__init__.py - Generic Image · Runs arbitrary container images as Kubernetes-backed processes
src/pygeoapi_k8s_manager/process/generic_image.py - Hello World · Example process used in tests and usage examples
src/pygeoapi_k8s_manager/process/hello_world_k8s.py - Utilities · Shared helpers for request and execution plumbing
src/pygeoapi_k8s_manager/util.py
Kubernetes runtime
Execution and cluster-facing control path
- K8s API · Kubernetes API used to create and observe jobs, pods, and events
- Job Pods · Containerized workload execution inside the cluster
- Inputs env · Carries dynamic inputs and job identity into pods
Finalization and storage
Log parsing and artifact persistence
- Finalizer · Watches completed jobs, extracts results from logs, and persists artifacts
src/pygeoapi_k8s_manager/finalizer.py - Log markers · Encodes result mimetype and payload boundaries in container logs
- Object Storage · Private storage for results and logs written by the finalizer
Deployment and local dev
Manifests, config, and cluster setup
- Config · Enables the custom manager and finalizer integration
pygeoapi-config.yaml - Deployment · Runs the manager as a Kubernetes service
k8s-manifests/deployment.yaml - RBAC · Grants access to jobs, pods, logs, statuses, and events
k8s-manifests/permissions.yaml - Service/Ingress · Exposes the manager inside or outside the cluster
k8s-manifests/service.yaml - Kind Setup · Local Kubernetes environment for development and validation
k8s-kind/kind-cluster.yaml - MinIO · Kind-backed object storage used to exercise the finalizer path
k8s-kind/minio.yaml
Connections 21
- pygeoapi → Kubernetes Manager: process request · pygeoapi invokes the custom manager for process execution
- Config → Kubernetes Manager: loads · pygeoapi configuration wires in the Kubernetes manager
- Config → Finalizer: enables · configuration activates the optional finalizer controller
- Kubernetes Manager → Process API: dispatches · Manager resolves and runs process implementations
- Process API → Generic Image: implements · Generic container-backed process implementation
- Process API → Hello World: implements · Sample process implementation
- Process API → Utilities: uses · Shared helper functions support process behavior
- Kubernetes Manager → K8s API: creates jobs · Manager talks to Kubernetes to create and monitor jobs
- Kubernetes Manager → RBAC: requires · RBAC boundary needed for cluster operations
- RBAC → K8s API: authorizes · Grants access to job, pod, log, and event resources
- K8s API → Job Pods: runs · Kubernetes schedules the job pod that executes the workload
- Job Pods → Inputs env: receives · Inputs and job id are injected through environment variables
- Job Pods → Log markers: emits · Result metadata and payload are written to logs using markers
- Job Pods → Finalizer: status/logs · Completed jobs are observed and inspected by the finalizer
- Finalizer → Object Storage: uploads · Result and log artifacts are persisted to object storage
- Finalizer → K8s API: watches · Finalizer monitors job state and fetches logs through the API
- Deployment → Kubernetes Manager: deploys · Kubernetes deployment runs the service container
- Service/Ingress → Kubernetes Manager: exposes · Service and ingress provide network access to the manager
- Kind Setup → MinIO: includes · Local development cluster includes storage for finalizer testing
- Kind Setup → Kubernetes Manager: hosts · Kind provides the local runtime for the manager stack
- MinIO → Object Storage: mimics · Local object storage stands in for the production bucket