How apache/flink-statefun-docker works
Project Type and Purpose
• This is an open-source tooling repository whose sole purpose is to generate, build, and publish Docker images for Apache Flink Stateful Functions across multiple versions and Java runtimes (Java 8 and Java 11).
• It is not a running application but rather a configuration-driven build system: a collection of versioned Dockerfiles, a templating mechanism, and a release script that updates and publishes images to Docker Hub.
File Structure Analysis
• Top-level artifacts:
• gpg_keys.txt – maps each Flink Stateful Functions release to its GPG signing key
• -version.sh – a helper script that bootstraps new version directories from a template
• template/ – contains the canonical Dockerfile, entrypoint, and configuration structure used as the basis for all versioned images
• directories named by version (e.g., 2.0.0, 2.2.0-java8, 3.3.0-java11) – each holds:
– Dockerfile
– docker-entry-point.sh
– flink-distribution/ (with bin/ and conf/flink-conf.yaml)
• Patterns:
• Template-based generation: the template/ directory defines the layout and is copied and parameterized for each new release.
• Version-partitioned directories: maintain separate configurations per Flink & Java version.
• Release script: orchestrates adding keys and instantiating new version folders.
Key Insights from the README
• Workflow steps for adding a new release:
a. GPG key to gpg_keys.txt
b. Run add-version.sh with statefun-version and flink-version arguments
c. Build and push resulting images (with tags like x.y.z, x.y.z-java8, latest)
• Target Docker Hub repo: apache/flink-statefun
• Publishing conventions for Java 8 vs Java 11 images
Main Components
a. Source Repository (GitHub)
• Contains: gpg_keys.txt, template/, add-version.sh, versioned directories
b. Templating Engine (add-version.sh)
• Reads: template/, gpg_keys.txt, user params (statefun + flink versions)
• Outputs: new version directory with Dockerfile & entrypoint scripts
c. Docker Build System
• Inputs: version directory (Dockerfile + flink-distribution/)
• Produces: Docker image tagged appropriately
d. Docker Registry (Docker Hub: apache/flink-statefun)
• Stores published images under their tags
e. External Artifacts
• Flink Stateful Functions distributions (downloaded or baked into images)
• GPG key servers (used to verify package signatures)
Relationships and Data Flows
• Developer/CI → GitHub: push changes (gpg_keys.txt, new version dir)
• -version.sh → file system: reads template + keys → writes new version folder
• Docker daemon → reads Dockerfile & flink-distribution → builds image
• Docker daemon → pushes image → Docker Hub registry
• Consumer (users or downstream pipelines) → pulls image from Docker Hub
Architectural Patterns and Principles
• Template-driven code generation: central template ensures consistency across versions.
• Directory per version: isolation of configuration; easy parallel maintenance.
• Scripted release pipeline: reproducible, automatable steps for new releases.
• Single responsibility: each directory is self-contained for one image build.
[GitHub Repo] → [add-version.sh] → [Versioned Directory: Dockerfile + flink-distribution] → [Docker Build] → [Docker Hub Registry] → [Consumers]
• Blue = source and templates (GitHub Repo, template/)
• Green = build logic (add-version.sh, Docker Build)
• Orange = artifacts (Version directories, Docker images)
• Purple = external services (Docker Hub, GPG key servers)
Essential Elements to Capture
• Clear labels for:
• Template directory vs version directories
• Release script parameters (–s statefun-version, –f flink-version)
• Docker tags conventions (x.y.z, x.y.z-java8, latest)
• -version.sh reading and writing files
• Docker build pulling configs and pushing images
• Consumers pulling images from the registry
• Static config (template, gpg_keys.txt)
• Dynamic generation (scripts)
• Build/runtime artifacts (images)
• External dependencies (registry, GPG servers)