How apache/flink-playgrounds works
This repository is a collection of runnable Apache Flink learning playgrounds, not one unified application. Users explore operational behavior in the Operations Playground, build a Kafka-to-MySQL spend report in the Table Walkthrough, and run a Kafka-to-Elasticsearch payment-processing example in the PyFlink Walkthrough. The repository also supplies sample event generation and a click-count job for the operations exercise. The sampled job and generator sources show concrete Kafka flows; README statements describe the other walkthrough capabilities. The SpendReport aggregation itself is deliberately unimplemented in the sampled source.
Main entry points: ClickEventCount.java Click Count Job · Producer.java Transaction Producer · SpendReport.java Spend Report Job · payment_msg_proccessing.py Payment Processing
Components 29
Operations Playground
- Ops Environment
operations-playground - Click Count Job
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Records
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEvent.java - Backpressure Map
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/BackpressureMap.java - Window Counter
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/CountingAggregator.java - Window Statistics
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java - Statistics Record
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatistics.java - Click Codecs
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventDeserializationSchema.java - Statistics Codec
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/records/ClickEventStatisticsSerializationSchema.java - Kafka Cluster
- Click Event Generator
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java - Job Operator
Table Walkthrough
- Spend Report Job
table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java - Report Stub
table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java - Transaction Events
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/Transaction.java - MySQL Report Store
- Grafana Dashboard
docker/grafana-spend-report-init/dashboard.json - Table Services
table-walkthrough
PyFlink Walkthrough
- Payment Processing
pyflink-walkthrough/payment_msg_proccessing.py - Source Data Generator
pyflink-walkthrough/generator/generate_source_data.py - Kafka Input
- Elasticsearch
- Kibana Views
pyflink-walkthrough/kibana/export.ndjson
Sample Data and Jobs
- Transaction Producer
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java - Transaction Supplier
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSupplier.java - Transaction Serializer
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/model/TransactionSerializer.java - Rate Throttler
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Throttler.java - Transaction Kafka
Other components
- Playground User
Connections 29
- Playground User → Ops Environment: explores operations
README.md - Playground User → Table Services: runs walkthrough
README.md - Playground User → Payment Processing: runs example
README.md - Click Count Job → Kafka Cluster: reads and writes
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Click Records: processes
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Backpressure Map: optionally maps
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Window Counter: aggregates windows
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Window Statistics: collects windows
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Window Statistics → Statistics Record: creates record
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/functions/ClickEventStatisticsCollector.java - Click Count Job → Click Codecs: configures input
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Statistics Codec: configures output
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventCount.java - Click Count Job → Job Operator: supports operation
README.md - Job Operator → Ops Environment: operates jobs
README.md - Ops Environment → Kafka Cluster: includes cluster
README.md - Click Event Generator → Kafka Cluster: produces events
docker/ops-playground-image/java/flink-playground-clickcountjob/src/main/java/org/apache/flink/playgrounds/ops/clickcount/ClickEventGenerator.java - Spend Report Job → Transaction Events: reads records
table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java - Spend Report Job → Report Stub: calls report
table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java - Spend Report Job → MySQL Report Store: writes report
table-walkthrough/src/main/java/org/apache/flink/playgrounds/spendreport/SpendReport.java - Transaction Events → Transaction Kafka: published to
README.md - MySQL Report Store → Grafana Dashboard: visualized in
README.md - Table Services → Spend Report Job: runs job
README.md - Source Data Generator → Kafka Input: generates input
- Payment Processing → Kafka Input: reads events
README.md - Payment Processing → Elasticsearch: writes results
README.md - Elasticsearch → Kibana Views: visualized in
README.md - Transaction Producer → Transaction Supplier: gets transactions
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java - Transaction Producer → Transaction Serializer: configures encoding
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java - Transaction Producer → Rate Throttler: throttles sends
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java - Transaction Producer → Transaction Kafka: publishes records
docker/data-generator/src/main/java/org/apache/flink/playground/datagen/Producer.java