# apache/flink-playgrounds architecture

> Architecture diagram of the GitHub repository apache/flink-playgrounds, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/flink-playgrounds
- Repository: https://github.com/apache/flink-playgrounds

## Overview

This repository is a collection of runnable Apache Flink learning playgrounds, not one unified application. Users explore operational behavior in the Operations Playground, build a Kafka-to-MySQL spend report in the Table Walkthrough, and run a Kafka-to-Elasticsearch payment-processing example in the PyFlink Walkthrough. The repository also supplies sample event generation and a click-count job for the operations exercise. The sampled job and generator sources show concrete Kafka flows; README statements describe the other walkthrough capabilities. The SpendReport aggregation itself is deliberately unimplemented in the sampled source.

Main entry points: [`docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java) (Click Count Job), [`docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java) (Transaction Producer), [`table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`](https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java) (Spend Report Job), [`pyflink-walkthrough/payment_msg_proccessing.py`](https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/payment_msg_proccessing.py) (Payment Processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ops["Operations Playground"]
  node_opsenv["Ops Environment"]
  node_opsjob["Click Count Job"]
  node_clickrecords["Click Records<br/>[ClickEvent.java]"]
  node_backpressure["Backpressure Map"]
  node_counter["Window Counter"]
  node_statistics["Window Statistics"]
  node_statsrecord["Statistics Record"]
  node_clickcodec["Click Codecs"]
  node_statscodec["Statistics Codec"]
  node_ops_kafka[("Kafka Cluster")]
  node_clickgen["Click Event Generator"]
  node_opscontrol(("Job Operator"))
end

subgraph group_table["Table Walkthrough"]
  node_tablejob["Spend Report Job<br/>[SpendReport.java]"]
  node_spendstub["Report Stub<br/>[SpendReport.java]"]
  node_transactions["Transaction Events<br/>[Transaction.java]"]
  node_mysql[("MySQL Report Store")]
  node_dashboard["Grafana Dashboard<br/>[dashboard.json]"]
  node_tableenv["Table Services"]
end

subgraph group_pyflink["PyFlink Walkthrough"]
  node_pyjob["Payment Processing"]
  node_pygen["Source Data Generator"]
  node_pykafka[("Kafka Input")]
  node_es[("Elasticsearch")]
  node_kibana["Kibana Views<br/>[export.ndjson]"]
end

subgraph group_shared["Sample Data and Jobs"]
  node_datagen["Transaction Producer<br/>[Producer.java]"]
  node_supplier["Transaction Supplier"]
  node_serializer["Transaction Serializer"]
  node_throttler["Rate Throttler<br/>[Throttler.java]"]
  node_tx_kafka[("Transaction Kafka")]
end

node_operator(("Playground User"))

node_operator -->|"explores operations"| node_opsenv
node_operator -->|"runs walkthrough"| node_tableenv
node_operator -->|"runs example"| node_pyjob
node_opsjob -->|"reads and writes"| node_ops_kafka
node_opsjob -->|"processes"| node_clickrecords
node_opsjob -->|"optionally maps"| node_backpressure
node_opsjob -->|"aggregates windows"| node_counter
node_opsjob -->|"collects windows"| node_statistics
node_statistics -->|"creates record"| node_statsrecord
node_opsjob -->|"configures input"| node_clickcodec
node_opsjob -->|"configures output"| node_statscodec
node_opsjob -->|"supports operation"| node_opscontrol
node_opscontrol -->|"operates jobs"| node_opsenv
node_opsenv -->|"includes cluster"| node_ops_kafka
node_clickgen -->|"produces events"| node_ops_kafka
node_tablejob -->|"reads records"| node_transactions
node_tablejob -->|"calls report"| node_spendstub
node_tablejob -->|"writes report"| node_mysql
node_transactions -.->|"published to"| node_tx_kafka
node_mysql -->|"visualized in"| node_dashboard
node_tableenv -->|"runs job"| node_tablejob
node_pygen -.->|"generates input"| node_pykafka
node_pyjob -->|"reads events"| node_pykafka
node_pyjob -->|"writes results"| node_es
node_es -->|"visualized in"| node_kibana
node_datagen -->|"gets transactions"| node_supplier
node_datagen -->|"configures encoding"| node_serializer
node_datagen -->|"throttles sends"| node_throttler
node_datagen -->|"publishes records"| node_tx_kafka

click node_opsenv "https://github.com/apache/flink-playgrounds/tree/master/operations-playground"
click node_opsjob "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java"
click node_clickrecords "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEvent.java"
click node_backpressure "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java"
click node_counter "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/CountingAggregator.java"
click node_statistics "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java"
click node_statsrecord "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatistics.java"
click node_clickcodec "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventDeserializationSchema.java"
click node_statscodec "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatisticsSerializationSchema.java"
click node_tablejob "https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java"
click node_spendstub "https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java"
click node_transactions "https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/Transaction.java"
click node_dashboard "https://github.com/apache/flink-playgrounds/blob/master/docker/grafana-spend-report-init/dashboard.json"
click node_tableenv "https://github.com/apache/flink-playgrounds/tree/master/table-walkthrough"
click node_pyjob "https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/payment_msg_proccessing.py"
click node_pygen "https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/generator/generate_source_data.py"
click node_kibana "https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/kibana/export.ndjson"
click node_datagen "https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java"
click node_supplier "https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSupplier.java"
click node_serializer "https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSerializer.java"
click node_throttler "https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Throttler.java"
click node_clickgen "https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_opsenv,node_opsjob,node_clickrecords,node_backpressure,node_counter,node_statistics,node_statsrecord,node_clickcodec,node_statscodec,node_ops_kafka,node_clickgen,node_opscontrol,node_operator toneBlue
class node_tablejob,node_spendstub,node_transactions,node_mysql,node_dashboard,node_tableenv toneAmber
class node_pyjob,node_pygen,node_pykafka,node_es,node_kibana toneMint
class node_datagen,node_supplier,node_serializer,node_throttler,node_tx_kafka toneRose
```

## Components

### Operations Playground

- [Ops Environment](https://github.com/apache/flink-playgrounds/tree/master/operations-playground): `operations-playground`
- [Click Count Job](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`
- [Click Records](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEvent.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEvent.java`
- [Backpressure Map](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java`
- [Window Counter](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/CountingAggregator.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/CountingAggregator.java`
- [Window Statistics](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java`
- [Statistics Record](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatistics.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatistics.java`
- [Click Codecs](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventDeserializationSchema.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventDeserializationSchema.java`
- [Statistics Codec](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatisticsSerializationSchema.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatisticsSerializationSchema.java`
- **Kafka Cluster**
- [Click Event Generator](https://github.com/apache/flink-playgrounds/blob/master/docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java): `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java`
- **Job Operator**

### Table Walkthrough

- [Spend Report Job](https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java): `table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`
- [Report Stub](https://github.com/apache/flink-playgrounds/blob/master/table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java): `table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`
- [Transaction Events](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/Transaction.java): `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/Transaction.java`
- **MySQL Report Store**
- [Grafana Dashboard](https://github.com/apache/flink-playgrounds/blob/master/docker/grafana-spend-report-init/dashboard.json): `docker/grafana-spend-report-init/dashboard.json`
- [Table Services](https://github.com/apache/flink-playgrounds/tree/master/table-walkthrough): `table-walkthrough`

### PyFlink Walkthrough

- [Payment Processing](https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/payment_msg_proccessing.py): `pyflink-walkthrough/payment_msg_proccessing.py`
- [Source Data Generator](https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/generator/generate_source_data.py): `pyflink-walkthrough/generator/generate_source_data.py`
- **Kafka Input**
- **Elasticsearch**
- [Kibana Views](https://github.com/apache/flink-playgrounds/blob/master/pyflink-walkthrough/kibana/export.ndjson): `pyflink-walkthrough/kibana/export.ndjson`

### Sample Data and Jobs

- [Transaction Producer](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java): `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`
- [Transaction Supplier](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSupplier.java): `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSupplier.java`
- [Transaction Serializer](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSerializer.java): `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSerializer.java`
- [Rate Throttler](https://github.com/apache/flink-playgrounds/blob/master/docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Throttler.java): `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Throttler.java`
- **Transaction Kafka**

### Other components

- **Playground User**

## Connections

- Playground User → Ops Environment: explores operations (evidence: `README.md`)
- Playground User → Table Services: runs walkthrough (evidence: `README.md`)
- Playground User → Payment Processing: runs example (evidence: `README.md`)
- Click Count Job → Kafka Cluster: reads and writes (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Click Records: processes (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Backpressure Map: optionally maps (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Window Counter: aggregates windows (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Window Statistics: collects windows (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Window Statistics → Statistics Record: creates record (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java`)
- Click Count Job → Click Codecs: configures input (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Statistics Codec: configures output (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java`)
- Click Count Job → Job Operator: supports operation (evidence: `README.md`)
- Job Operator → Ops Environment: operates jobs (evidence: `README.md`)
- Ops Environment → Kafka Cluster: includes cluster (evidence: `README.md`)
- Click Event Generator → Kafka Cluster: produces events (evidence: `docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java`)
- Spend Report Job → Transaction Events: reads records (evidence: `table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`)
- Spend Report Job → Report Stub: calls report (evidence: `table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`)
- Spend Report Job → MySQL Report Store: writes report (evidence: `table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java`)
- Transaction Events → Transaction Kafka: published to (evidence: `README.md`)
- MySQL Report Store → Grafana Dashboard: visualized in (evidence: `README.md`)
- Table Services → Spend Report Job: runs job (evidence: `README.md`)
- Source Data Generator → Kafka Input: generates input
- Payment Processing → Kafka Input: reads events (evidence: `README.md`)
- Payment Processing → Elasticsearch: writes results (evidence: `README.md`)
- Elasticsearch → Kibana Views: visualized in (evidence: `README.md`)
- Transaction Producer → Transaction Supplier: gets transactions (evidence: `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`)
- Transaction Producer → Transaction Serializer: configures encoding (evidence: `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`)
- Transaction Producer → Rate Throttler: throttles sends (evidence: `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`)
- Transaction Producer → Transaction Kafka: publishes records (evidence: `docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java`)