How jaehyeon-kim/flink-demos works
This repository is a collection of Apache Flink demos and learning exercises, not one integrated application. Its principal workflows show data entering through Kafka, files, or custom sources; being transformed with DataStream or Table API operations; and producing aggregates or other results in sinks. The graph keeps the README’s distinct projects and capabilities separate. Some connections are documented only in the README, so unsampled implementations are represented at their existing project-directory paths rather than inferred in detail.
Main entry points: app.py Taxi Producer · s4_11_session_windows_sql.py Session Windows · local/processor.py Local Fraud App · remote/processor.py Managed Fraud App
Components 30
AWS Pipelines
- Taxi Producer
real-time-streaming-aws/producer/app.py - Kafka Cluster
- Taxi Flink Flows
real-time-streaming-aws - AWS Data Services
- Kafka Consumer
real-time-streaming-aws/consumer/app.py - Local Fraud App
fraud-detection/local/processor.py - Managed Fraud App
fraud-detection/remote/processor.py - Fraud Topics
- DynamoDB
- Local Window App
pyflink-getting-started-on-aws/local/processor.py - Managed Window App
pyflink-getting-started-on-aws/remote/processor.py - Kafka Topics
- Taxi Ride Records
real-time-streaming-aws/producer/app.py
PyFlink Learning
- DataStream Course
building-pyflink-apps/src - PyFlink Tutorials
pyflink-doc
Book Examples
- Sensor Aggregation
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt - Sensor Source
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/sensor/SensorSource.kt - DataStream Examples
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter5 - Time And Windows
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter6 - State Examples
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter7 - Custom Connectors
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter8/CustomConnectors.kt - PyFlink Book Port
stream-processing-with-pyflink/src - Average Readings
stream-processing-with-flink/src/main/kotlin/me/jaehyeon/chapter1/AverageSensorReadings.kt
Course Exercises
- Table Sources
pyflink-udemy/s3_02_python_source.py - Table Operations
pyflink-udemy/s4_01_projections.py - Session Windows
pyflink-udemy/s4_11_session_windows_sql.py - SQL UDF
sql-cookbook/src/python_udf.py - Sales Items Topic
- Processed Sales Topic
Other components
- Demo User
Connections 27
- Demo User → Taxi Producer: runs
- Taxi Producer → Taxi Ride Records: generates
- Taxi Producer → Kafka Cluster: publishes
- Kafka Cluster → Taxi Flink Flows: feeds
- Taxi Flink Flows → AWS Data Services: moves data
- Kafka Cluster → Kafka Consumer: triggers
- Local Fraud App → Fraud Topics: reads topics
- Managed Fraud App → Fraud Topics: reads topics
- Local Fraud App → DynamoDB: writes flagged
- Managed Fraud App → DynamoDB: writes flagged
- Local Window App → Kafka Topics: reads and writes
- Managed Window App → Kafka Topics: reads and writes
- Sensor Aggregation → Sensor Source: creates source
- Sensor Aggregation → Average Readings: computes
- Demo User → DataStream Course: studies
- Demo User → PyFlink Tutorials: studies
- Demo User → PyFlink Book Port: studies
- Demo User → DataStream Examples: studies
- Demo User → Time And Windows: studies
- Demo User → State Examples: studies
- Demo User → Custom Connectors: studies
- Demo User → Table Sources: studies
- Demo User → Table Operations: studies
- Demo User → Session Windows: studies
- Demo User → SQL UDF: studies
- Sales Items Topic → Session Windows: supplies rows
- Session Windows → Processed Sales Topic: writes aggregate