How acquirel/flinklearn works
This repository is a collection of Apache Flink streaming examples, with a Scala click-log analytics workflow as its most complete product path. The workflow consumes Kafka JSON, converts it into typed messages, enriches records with time and user-history fields, then runs channel analytics backed by HBase. Other documented-by-tree examples cover Kafka/HBase/HDFS/socket connectors, windowing, state, broadcast operators, metrics, and Table/UDF JSON conversion. Source excerpts support the Scala workflow and UDF wiring; links among unsampled examples are omitted rather than inferred.
Main entry points: DataToWideTask.scala Wide enrichment · App.scala Flink job entry · ChannelPvUvTask.scala Channel PV/UV · HBaseUtil.scala HBase access
Components 22
Input and models
- Kafka input
src/main/scala/com/kali/flink/App.scala - Click messages
src/main/scala/com/kali/flink/bean/Message.scala - Wide click record
src/main/scala/com/kali/flink/bean/ClickLogWide.scala
Stream analytics
- Wide enrichment
src/main/scala/com/kali/flink/task/DataToWideTask.scala - Time formatting
src/main/scala/com/kali/flink/util/TimeUtil.scala - Channel hot stats
src/main/scala/com/kali/flink/task/ChannelHotTask.scala - Channel PV/UV
src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala - Flink job entry
src/main/scala/com/kali/flink/App.scala
State and sinks
- HBase access
src/main/scala/com/kali/flink/util/HBaseUtil.scala - HBase
Flink examples
- Data generator
src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java - Generated user
src/main/java/com/kali/flink/core/connector/generator/User.java - Broadcast operators
src/main/java/com/kali/flink/core/operator/broadcast - Dimension record
src/main/java/com/kali/flink/core/operator/broadcast/Entity/DimData.java - Window examples
src/main/java/com/kali/flink/core/operator/window - Window record
src/main/java/com/kali/flink/core/operator/window/Entity/WordCountCart.java - State and timers
src/main/java/com/kali/flink/core/operator/state - Metrics examples
src/main/java/com/kali/flink/core/metrics - Connector demos
src/main/java/com/kali/flink/core/connector - Scala connectors
src/main/scala/com/kali/flink/connector
Table functions
- JSON table demo
src/main/java/com/kali/flink/stream/table/UseUdf.java - JSON to row UDF
src/main/java/com/kali/flink/stream/udf/JsonToRow.java
Connections 19
- Kafka input → Click messages: parses into
src/main/scala/com/kali/flink/App.scala - Flink job entry → Kafka input: creates source
src/main/scala/com/kali/flink/App.scala - Flink job entry → Wide enrichment: invokes
src/main/scala/com/kali/flink/App.scala - Wide enrichment → Click messages: maps records
src/main/scala/com/kali/flink/task/DataToWideTask.scala - Wide enrichment → Wide click record: constructs
src/main/scala/com/kali/flink/task/DataToWideTask.scala - Wide enrichment → Time formatting: formats dates
src/main/scala/com/kali/flink/task/DataToWideTask.scala - Wide enrichment → HBase access: reads and updates
src/main/scala/com/kali/flink/task/DataToWideTask.scala - HBase access → HBase: reads and writes
- Flink job entry → Channel hot stats: invokes
src/main/scala/com/kali/flink/App.scala - Flink job entry → Channel PV/UV: optional invocation
src/main/scala/com/kali/flink/App.scala - Channel hot stats → Wide click record: consumes stream
src/main/scala/com/kali/flink/task/ChannelHotTask.scala - Channel hot stats → HBase access: uses storage
src/main/scala/com/kali/flink/task/ChannelHotTask.scala - Channel PV/UV → Wide click record: consumes stream
src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala - Channel PV/UV → Time formatting: formats windows
src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala - Channel PV/UV → HBase access: reads and writes
src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala - Data generator → Generated user: generates
src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java - Broadcast operators → Dimension record: uses dimension
src/main/java/com/kali/flink/core/operator/broadcast/BroadcastTest.java - Window examples → Window record: uses record
src/main/java/com/kali/flink/core/operator/window/processtime/StreamingTumblingTimeWindow.java - JSON table demo → JSON to row UDF: registers function
src/main/java/com/kali/flink/stream/table/UseUdf.java