# didi/knowstreaming architecture

> Architecture diagram of the GitHub repository didi/knowstreaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/didi/knowstreaming
- Repository: https://github.com/didi/knowstreaming

## Overview

##### Project type & purpose
- This is a full-stack, cloud-native Kafka management & monitoring platform (“Know Streaming”).
- Provides GUI (frontend) for Kafka cluster ops, a Java backend for business logic, real-time metrics collection, scheduled tasks, persistence into MySQL/Elasticsearch, and Docker/Helm-based deployment.

##### Key top-level modules (file structure clues)
- km-console: React/TypeScript micro-frontends (config-manager-fe, layout-clusters-fe) packaged via Lerna & Webpack.
- km-rest: Spring Boot REST API layer exposing HTTP endpoints.
- km-biz, km-core, km-collector, km-task: Java modules for business logic, core services, metrics collection, scheduling dispatch tasks.
- km-persistence: data access layer (MyBatis DAOs to MySQL & ES).
- km-common: shared DTO/VO/entity definitions, utilities, annotations.
- km-enterprise, km-extends: plugin modules for enterprise features (HA mirror, account, monitor).
- km-dist: Docker Compose & Helm charts for multi-container deployment.
- km-biz & km-core: service/business orchestration.
- km-task & km-collector: background tasks & metric sinks.

##### Architecture & component mapping
a. Frontend (km-console)
- React apps (config-manager-fe, layout-clusters-fe) served by a Node/webpack layer or static web server.
- Communicates via REST (HTTP/JSON) to backend.
b. API Gateway / REST Layer (km-rest)
- Spring MVC controllers handle incoming requests, route to service layer.
- Security/interceptor for permissions.
c. Business & Orchestration (km-biz, km-core)
- Implements managers & service interfaces for clusters, brokers, topics, ACLs, jobs, etc.
- Applies patterns: Factory, Strategy (reassign strategies), Event–Listener for cluster changes.
d. Persistence (km-persistence)
- MySQL DAOs via MyBatis for configuration & operational data.
- Elasticsearch DAOs for time-series metrics & logs (ES templates loaded at init).
e. Collectors & Tasks
- km-collector: metric collectors poll Kafka, Connect, MirrorMaker, JMX, ZK and push to ES.
- km-task: scheduled dispatch tasks manage metadata sync & health checks (Kafka, Connect, MM2).
f. Plugins / Extensions
- km-enterprise/km-ha: high-availability mirroring logic.
- km-extends: account integration (LDAP), Prometheus sink for monitoring.
g. External systems
- Kafka clusters (Zookeeper-backed or KRaft), Zookeeper ensembles, JMX interfaces.
- MySQL, Elasticsearch.
- LDAP (for authentication in account plugin).
- Prometheus (monitoring plugin).

##### High-level relationships & data flows
- UI → REST API (HTTP) → Service layer → Persistence / Collector / Task modules.
- Scheduled km-task triggers metadata sync & health checks → writes to persistence & emits events.
- km-collector continuously polls metrics → writes to Elasticsearch.
- Business operations (create topic, rebalance) → Service → Kafka Admin clients → Kafka clusters.
- Cluster state watchers (km-core flusher) subscribe to ZK change events → update local caches & DB.
- Plugin modules extend core via Spring’s component/annotation scanning (km-common).

##### Architectural patterns & technologies
- Layered architecture: Controller → Service → DAO.
- Event-driven ZK watchers using Observer pattern.
- Factory pattern for handler extension (BaseExtendFactory).
- Micro-frontend (Lerna) for independent UI modules.
- Spring Boot, Spring MVC, MyBatis, Maven multi-module.
- Docker Compose & Kubernetes Helm for deployment.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% UI Layer
    subgraph "User Interface" 
        direction TB
        ConfigFE["Config Manager FE"]:::ui
        ClusterFE["Cluster Layout FE"]:::ui
    end

    %% API Layer
    subgraph "REST API Layer" 
        direction TB
        REST["km-rest (Spring Boot Controllers)"]:::api
    end

    %% Business & Core Layer
    subgraph "Business & Core Services" 
        direction TB
        Common["km-common (DTOs/Utilities)"]:::common
        Biz["km-biz (Managers & Services)"]:::biz
        Core["km-core (ZK Watchers & Factories)"]:::core
    end

    %% Persistence Layer
    subgraph "Persistence Layer" 
        direction TB
        Persistence["km-persistence (MyBatis DAOs)"]:::persistence
        MySQL[(MySQL Database)]:::db
        ES[(Elasticsearch)]:::db
    end

    %% Collectors & Tasks Layer
    subgraph "Collectors & Tasks" 
        direction TB
        Collector["km-collector (Metric Collectors)"]:::async
        Task["km-task (Scheduler & Dispatch)"]:::async
    end

    %% Plugins Layer
    subgraph "Plugin Modules" 
        direction TB
        HA["km-enterprise (HA Mirror Logic)"]:::plugin
        LDAP["km-extends (Account/LDAP Integration)"]:::plugin
        Prom["km-extends (Prometheus Sink)"]:::plugin
    end

    %% Deployment Layer
    subgraph "Deployment" 
        direction TB
        Docker["Docker Compose"]:::infra
        Helm["Kubernetes Helm Charts"]:::infra
    end

    %% External Systems
    subgraph "External Systems" 
        direction TB
        Kafka["Kafka Clusters"]:::external
        ZK["Zookeeper"]:::external
        JMXsys["JMX Interfaces"]:::external
        LDAPsys["LDAP Server"]:::external
        Promsys["Prometheus"]:::external
    end

    %% Connections
    ConfigFE -->|"HTTP/JSON"| REST
    ClusterFE -->|"HTTP/JSON"| REST
    REST -->|"invoke"| Biz
    REST -->|"invoke"| Core
    Biz -->|"uses"| Common
    Core -->|"uses"| Common
    Biz -->|"persist"| Persistence
    Core -->|"persist"| Persistence
    Persistence -->|store| MySQL
    Persistence -->|index| ES
    Collector -.->|writes| ES
    Task -.->|writes| Persistence
    Task -.->|emits events to| Biz
    Biz -->|"AdminClient"| Kafka
    Core -->|"watch events"| ZK
    Collector -.->|polls| Kafka
    Collector -.->|polls| ZK
    Collector -.->|polls| JMXsys
    LDAP -->|auth| LDAPsys
    Prom -->|push metrics to| Promsys
    HA -->|integrates| Biz
    HA -->|uses| Core

    %% Deployment
    Docker --> REST
    Docker --> Biz
    Docker --> Core
    Docker --> Persistence
    Docker --> Collector
    Docker --> Task
    Docker --> HA
    Docker --> LDAP
    Docker --> Prom
    Helm --> Docker

    %% Click Events
    click ConfigFE "https://github.com/didi/knowstreaming/tree/master/km-console/packages/config-manager-fe"
    click ClusterFE "https://github.com/didi/knowstreaming/tree/master/km-console/packages/layout-clusters-fe"
    click REST "https://github.com/didi/knowstreaming/tree/master/km-rest/src/main/java/com/xiaojukeji/know/streaming/km/rest"
    click Biz "https://github.com/didi/knowstreaming/tree/master/km-biz/src/main/java/com/xiaojukeji/know/streaming/km/biz"
    click Core "https://github.com/didi/knowstreaming/tree/master/km-core/src/main/java/com/xiaojukeji/know/streaming/km/core"
    click Common "https://github.com/didi/knowstreaming/tree/master/km-common/src/main/java/com/xiaojukeji/know/streaming/km/common"
    click Persistence "https://github.com/didi/knowstreaming/tree/master/km-persistence/src/main/java/com/xiaojukeji/know/streaming/km/persistence"
    click Collector "https://github.com/didi/knowstreaming/tree/master/km-collector/src/main/java/com/xiaojukeji/know/streaming/km/collector"
    click Task "https://github.com/didi/knowstreaming/tree/master/km-task/src/main/java/com/xiaojukeji/know/streaming/km/task"
    click HA "https://github.com/didi/knowstreaming/tree/master/km-enterprise/km-ha/src/main/java/com/xiaojukeji/know/streaming/km/ha"
    click LDAP "https://github.com/didi/knowstreaming/tree/master/km-extends/km-account/src/main/java/com/xiaojukeji/know/streaming/km/account"
    click Prom "https://github.com/didi/knowstreaming/tree/master/km-extends/km-monitor/src/main/java/com/xiaojukeji/know/streaming/km/monitor"
    click Docker "https://github.com/didi/knowstreaming/tree/master/km-dist/docker"
    click Helm "https://github.com/didi/knowstreaming/tree/master/km-dist/helm"
    click ROOT "https://github.com/didi/knowstreaming/blob/master/pom.xml"

    %% Styles
    classDef ui fill:#D0E8FF,stroke:#003366,color:#003366;
    classDef api fill:#DFFFE0,stroke:#228B22,color:#228B22;
    classDef biz fill:#FFF5CC,stroke:#DAA520,color:#DAA520;
    classDef core fill:#FFF5CC,stroke:#DAA520,color:#DAA520;
    classDef common fill:#F0F0F0,stroke:#808080,color:#808080;
    classDef persistence fill:#FFE8CC,stroke:#FF8C00,color:#FF8C00;
    classDef db fill:#FFF0F0,stroke:#FF0000,color:#FF0000,shape:cylinder;
    classDef async fill:#E8D0FF,stroke:#800080,color:#800080,stroke-dasharray: 5 5;
    classDef plugin fill:#E0FFFF,stroke:#008B8B,color:#008B8B;
    classDef infra fill:#F0FFF0,stroke:#006400,color:#006400;
    classDef external fill:#F0F0F0,stroke:#696969,color:#696969,shape:cloud;
```
