How didi/knowstreaming works
Project type & purpose
• This is a full-stack, cloud-native Kafka management & monitoring platform (“Know Streaming”).
• Provides GUI (frontend) for Kafka cluster ops, a Java backend for business logic, real-time metrics collection, scheduled tasks, persistence into MySQL/Elasticsearch, and Docker/Helm-based deployment.
Key top-level modules (file structure clues)
• km-console: React/TypeScript micro-frontends (config-manager-fe, layout-clusters-fe) packaged via Lerna & Webpack.
• km-rest: Spring Boot REST API layer exposing HTTP endpoints.
• km-biz, km-core, km-collector, km-task: Java modules for business logic, core services, metrics collection, scheduling dispatch tasks.
• km-persistence: data access layer (MyBatis DAOs to MySQL & ES).
• km-common: shared DTO/VO/entity definitions, utilities, annotations.
• km-enterprise, km-extends: plugin modules for enterprise features (HA mirror, account, monitor).
• km-dist: Docker Compose & Helm charts for multi-container deployment.
• km-biz & km-core: service/business orchestration.
• km-task & km-collector: background tasks & metric sinks.
Architecture & component mapping
a. Frontend (km-console)
• React apps (config-manager-fe, layout-clusters-fe) served by a Node/webpack layer or static web server.
• Communicates via REST (HTTP/JSON) to backend.
b. API Gateway / REST Layer (km-rest)
• Spring MVC controllers handle incoming requests, route to service layer.
• Security/interceptor for permissions.
c. Business & Orchestration (km-biz, km-core)
• Implements managers & service interfaces for clusters, brokers, topics, ACLs, jobs, etc.
• Applies patterns: Factory, Strategy (reassign strategies), Event–Listener for cluster changes.
d. Persistence (km-persistence)
• MySQL DAOs via MyBatis for configuration & operational data.
• Elasticsearch DAOs for time-series metrics & logs (ES templates loaded at init).
e. Collectors & Tasks
• km-collector: metric collectors poll Kafka, Connect, MirrorMaker, JMX, ZK and push to ES.
• km-task: scheduled dispatch tasks manage metadata sync & health checks (Kafka, Connect, MM2).
f. Plugins / Extensions
• km-enterprise/km-ha: high-availability mirroring logic.
• km-extends: account integration (LDAP), Prometheus sink for monitoring.
g. External systems
• Kafka clusters (Zookeeper-backed or KRaft), Zookeeper ensembles, JMX interfaces.
• MySQL, Elasticsearch.
• LDAP (for authentication in account plugin).
• Prometheus (monitoring plugin).
High-level relationships & data flows
• UI → REST API (HTTP) → Service layer → Persistence / Collector / Task modules.
• Scheduled km-task triggers metadata sync & health checks → writes to persistence & emits events.
• km-collector continuously polls metrics → writes to Elasticsearch.
• Business operations (create topic, rebalance) → Service → Kafka Admin clients → Kafka clusters.
• Cluster state watchers (km-core flusher) subscribe to ZK change events → update local caches & DB.
• Plugin modules extend core via Spring’s component/annotation scanning (km-common).
Architectural patterns & technologies
• Layered architecture: Controller → Service → DAO.
• Event-driven ZK watchers using Observer pattern.
• Factory pattern for handler extension (BaseExtendFactory).
• Micro-frontend (Lerna) for independent UI modules.
• Spring Boot, Spring MVC, MyBatis, Maven multi-module.
• Docker Compose & Kubernetes Helm for deployment.