# acquirel/flinklearn architecture

> Architecture diagram of the GitHub repository acquirel/flinklearn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/acquirel/flinklearn
- Repository: https://github.com/acquirel/flinklearn

## Overview

This repository is a collection of Apache Flink streaming examples, with a Scala click-log analytics workflow as its most complete product path. The workflow consumes Kafka JSON, converts it into typed messages, enriches records with time and user-history fields, then runs channel analytics backed by HBase. Other documented-by-tree examples cover Kafka/HBase/HDFS/socket connectors, windowing, state, broadcast operators, metrics, and Table/UDF JSON conversion. Source excerpts support the Scala workflow and UDF wiring; links among unsampled examples are omitted rather than inferred.

Main entry points: [`src/main/scala/com/kali/flink/task/DataToWideTask.scala`](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/DataToWideTask.scala) (Wide enrichment), [`src/main/scala/com/kali/flink/App.scala`](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/App.scala) (Flink job entry), [`src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala`](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala) (Channel PV/UV), [`src/main/scala/com/kali/flink/util/HBaseUtil.scala`](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/util/HBaseUtil.scala) (HBase access)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and models"]
  node_kafka["Kafka input<br/>[App.scala]"]
  node_message["Click messages<br/>[Message.scala]"]
  node_wide_record["Wide click record<br/>[ClickLogWide.scala]"]
end

subgraph group_analytics["Stream analytics"]
  node_wide_task["Wide enrichment"]
  node_time["Time formatting<br/>[TimeUtil.scala]"]
  node_hot_task["Channel hot stats"]
  node_pvuv_task["Channel PV/UV"]
  node_app["Flink job entry<br/>[App.scala]"]
end

subgraph group_storage["State and sinks"]
  node_hbase_util["HBase access<br/>[HBaseUtil.scala]"]
  node_hbase[("HBase")]
end

subgraph group_examples["Flink examples"]
  node_generator["Data generator"]
  node_user["Generated user<br/>[User.java]"]
  node_broadcast["Broadcast operators"]
  node_dim_data["Dimension record<br/>[DimData.java]"]
  node_window_examples["Window examples"]
  node_word_count["Window record<br/>[WordCountCart.java]"]
  node_state_examples["State and timers"]
  node_metrics["Metrics examples"]
  node_connectors["Connector demos"]
  node_scala_connectors["Scala connectors"]
end

subgraph group_table["Table functions"]
  node_udf_demo["JSON table demo<br/>[UseUdf.java]"]
  node_json_udf["JSON to row UDF<br/>[JsonToRow.java]"]
end

node_kafka -->|"parses into"| node_message
node_app -->|"creates source"| node_kafka
node_app -->|"invokes"| node_wide_task
node_wide_task -->|"maps records"| node_message
node_wide_task -->|"constructs"| node_wide_record
node_wide_task -->|"formats dates"| node_time
node_wide_task -->|"reads and updates"| node_hbase_util
node_hbase_util -.->|"reads and writes"| node_hbase
node_app -->|"invokes"| node_hot_task
node_app -.->|"optional invocation"| node_pvuv_task
node_hot_task -.->|"consumes stream"| node_wide_record
node_hot_task -.->|"uses storage"| node_hbase_util
node_pvuv_task -->|"consumes stream"| node_wide_record
node_pvuv_task -->|"formats windows"| node_time
node_pvuv_task -->|"reads and writes"| node_hbase_util
node_generator -->|"generates"| node_user
node_broadcast -->|"uses dimension"| node_dim_data
node_window_examples -->|"uses record"| node_word_count
node_udf_demo -->|"registers function"| node_json_udf

click node_kafka "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/App.scala"
click node_message "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/bean/Message.scala"
click node_wide_task "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/DataToWideTask.scala"
click node_wide_record "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/bean/ClickLogWide.scala"
click node_time "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/util/TimeUtil.scala"
click node_hbase_util "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/util/HBaseUtil.scala"
click node_hot_task "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/ChannelHotTask.scala"
click node_pvuv_task "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala"
click node_app "https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/App.scala"
click node_generator "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java"
click node_user "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/connector/generator/User.java"
click node_broadcast "https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/broadcast"
click node_dim_data "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/operator/broadcast/Entity/DimData.java"
click node_window_examples "https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/window"
click node_word_count "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/operator/window/Entity/WordCountCart.java"
click node_state_examples "https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/state"
click node_metrics "https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/metrics"
click node_connectors "https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/connector"
click node_scala_connectors "https://github.com/acquirel/flinklearn/tree/master/src/main/scala/com/kali/flink/connector"
click node_udf_demo "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/stream/table/UseUdf.java"
click node_json_udf "https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/stream/udf/JsonToRow.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_kafka,node_message,node_wide_record toneBlue
class node_wide_task,node_time,node_hot_task,node_pvuv_task,node_app toneAmber
class node_hbase_util,node_hbase toneMint
class node_generator,node_user,node_broadcast,node_dim_data,node_window_examples,node_word_count,node_state_examples,node_metrics,node_connectors,node_scala_connectors toneRose
class node_udf_demo,node_json_udf toneIndigo
```

## Components

### Input and models

- [Kafka input](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/App.scala): `src/main/scala/com/kali/flink/App.scala`
- [Click messages](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/bean/Message.scala): `src/main/scala/com/kali/flink/bean/Message.scala`
- [Wide click record](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/bean/ClickLogWide.scala): `src/main/scala/com/kali/flink/bean/ClickLogWide.scala`

### Stream analytics

- [Wide enrichment](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/DataToWideTask.scala): `src/main/scala/com/kali/flink/task/DataToWideTask.scala`
- [Time formatting](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/util/TimeUtil.scala): `src/main/scala/com/kali/flink/util/TimeUtil.scala`
- [Channel hot stats](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/ChannelHotTask.scala): `src/main/scala/com/kali/flink/task/ChannelHotTask.scala`
- [Channel PV/UV](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala): `src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala`
- [Flink job entry](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/App.scala): `src/main/scala/com/kali/flink/App.scala`

### State and sinks

- [HBase access](https://github.com/acquirel/flinklearn/blob/master/src/main/scala/com/kali/flink/util/HBaseUtil.scala): `src/main/scala/com/kali/flink/util/HBaseUtil.scala`
- **HBase**

### Flink examples

- [Data generator](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java): `src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java`
- [Generated user](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/connector/generator/User.java): `src/main/java/com/kali/flink/core/connector/generator/User.java`
- [Broadcast operators](https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/broadcast): `src/main/java/com/kali/flink/core/operator/broadcast`
- [Dimension record](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/operator/broadcast/Entity/DimData.java): `src/main/java/com/kali/flink/core/operator/broadcast/Entity/DimData.java`
- [Window examples](https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/window): `src/main/java/com/kali/flink/core/operator/window`
- [Window record](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/core/operator/window/Entity/WordCountCart.java): `src/main/java/com/kali/flink/core/operator/window/Entity/WordCountCart.java`
- [State and timers](https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/operator/state): `src/main/java/com/kali/flink/core/operator/state`
- [Metrics examples](https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/metrics): `src/main/java/com/kali/flink/core/metrics`
- [Connector demos](https://github.com/acquirel/flinklearn/tree/master/src/main/java/com/kali/flink/core/connector): `src/main/java/com/kali/flink/core/connector`
- [Scala connectors](https://github.com/acquirel/flinklearn/tree/master/src/main/scala/com/kali/flink/connector): `src/main/scala/com/kali/flink/connector`

### Table functions

- [JSON table demo](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/stream/table/UseUdf.java): `src/main/java/com/kali/flink/stream/table/UseUdf.java`
- [JSON to row UDF](https://github.com/acquirel/flinklearn/blob/master/src/main/java/com/kali/flink/stream/udf/JsonToRow.java): `src/main/java/com/kali/flink/stream/udf/JsonToRow.java`

## Connections

- Kafka input → Click messages: parses into (evidence: `src/main/scala/com/kali/flink/App.scala`)
- Flink job entry → Kafka input: creates source (evidence: `src/main/scala/com/kali/flink/App.scala`)
- Flink job entry → Wide enrichment: invokes (evidence: `src/main/scala/com/kali/flink/App.scala`)
- Wide enrichment → Click messages: maps records (evidence: `src/main/scala/com/kali/flink/task/DataToWideTask.scala`)
- Wide enrichment → Wide click record: constructs (evidence: `src/main/scala/com/kali/flink/task/DataToWideTask.scala`)
- Wide enrichment → Time formatting: formats dates (evidence: `src/main/scala/com/kali/flink/task/DataToWideTask.scala`)
- Wide enrichment → HBase access: reads and updates (evidence: `src/main/scala/com/kali/flink/task/DataToWideTask.scala`)
- HBase access → HBase: reads and writes
- Flink job entry → Channel hot stats: invokes (evidence: `src/main/scala/com/kali/flink/App.scala`)
- Flink job entry → Channel PV/UV: optional invocation (evidence: `src/main/scala/com/kali/flink/App.scala`)
- Channel hot stats → Wide click record: consumes stream (evidence: `src/main/scala/com/kali/flink/task/ChannelHotTask.scala`)
- Channel hot stats → HBase access: uses storage (evidence: `src/main/scala/com/kali/flink/task/ChannelHotTask.scala`)
- Channel PV/UV → Wide click record: consumes stream (evidence: `src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala`)
- Channel PV/UV → Time formatting: formats windows (evidence: `src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala`)
- Channel PV/UV → HBase access: reads and writes (evidence: `src/main/scala/com/kali/flink/task/ChannelPvUvTask.scala`)
- Data generator → Generated user: generates (evidence: `src/main/java/com/kali/flink/core/connector/generator/GeneratorSource.java`)
- Broadcast operators → Dimension record: uses dimension (evidence: `src/main/java/com/kali/flink/core/operator/broadcast/BroadcastTest.java`)
- Window examples → Window record: uses record (evidence: `src/main/java/com/kali/flink/core/operator/window/processtime/StreamingTumblingTimeWindow.java`)
- JSON table demo → JSON to row UDF: registers function (evidence: `src/main/java/com/kali/flink/stream/table/UseUdf.java`)