# weops-lab/bk-lite architecture

> Architecture diagram of the GitHub repository weops-lab/bk-lite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weops-lab/bk-lite
- Repository: https://github.com/weops-lab/bk-lite

## Overview

##### Project Type and Purpose
- BlueKing Lite is a full-stack, microservices-based operations product. It includes:
- A React/Next.js “web” frontend
- A Django-based “server” API layer (multiple Django apps: cmdb, monitor, node_mgmt, opspilot, system_mgmt, console_mgmt)
- Standalone microservices (in “agents”, “metis”, “pilot” folders) for data collection, AI inference, conversational assistant
- Deployment manifests (Docker Compose, Kubernetes) for infra services
- Edge-deployed agents (fusion-collector, nats-executor, telegraf)

##### Break Down the Major Components
a. Frontend Layer (“web” directory)
- Next.js application
- Served via Traefik (or similar API gateway)
b. API Gateway & Authentication
- Traefik (ingress) routes UI requests to backends
- Keycloak deployed (infra/keycloak) for SSO/OAuth
c. Backend Services (Django “server” directory)
- Multiple Django apps, each exposing REST endpoints and NATS RPC
- Celery workers (for async jobs) using broker (Redis/RabbitMQ)
d. Messaging & RPC
- NATS for internal RPC between services and to agents
- RabbitMQ for pilot event bus (optional)
e. Edge Agents (“agents” directory)
- fusion-collector: SNMP/Telegraf based data collector pushing metrics to central Telegraf/VictoriaMetrics
- nats-executor: executes remote commands via NATS RPC on hosts
- stargazer: cloud-resource comparator and monitor agent
f. AI & Inference Microservices (“metis”, “pilot”)
- Metis: LLM-based anomaly detection, chunking, RAG, embedding, summarize APIs
- Pilot: Rasa-style conversational assistant bridging messaging channels (WeChat, DingTalk) with LLMs
g. Data Stores
- PostgreSQL clusters (per Django app)
- Neo4j for CMDB graph (optional)
- VictoriaMetrics (timeseries)
- Elasticsearch (search for RAG)
- Redis for caching, Celery broker
- MinIO or S3 for document storage
h. Infrastructure & Orchestration
- Docker Compose (dev) & Kubernetes (prod) manifests under “deploy”
- Longhorn for persistent volumes
- Monitoring: Prometheus, Grafana

##### Relationships and Data Flows
- User Browser → Traefik → Next.js UI
- UI → REST/GraphQL → Django API
- Django Apps ↔ NATS RPC → edge agents (for remote deploy/collect)
- Django Apps → Celery → async tasks → broker (Redis/RabbitMQ)
- fusion-collector/Telegraf agents → push metrics → central Telegraf → VictoriaMetrics
- Metis & Pilot expose their own HTTP/gRPC endpoints consumed by UI or other services
- Django apps read/write to PostgreSQL, Redis, Elasticsearch, Neo4j, MinIO

##### Architectural Patterns & Technologies
- Microservices pattern (small, focused services)
- Event-driven / message-based communication via NATS and RabbitMQ
- CQRS for metric collection vs. configuration management
- Externalize configuration via Helm/Kustomize (deploy/kubernetes)
- AI-first modules: embed-service, anomaly-detector, RAG service, orchestration agent (plan-execute, reactive agents)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer" 
        direction TB
        Browser["User Browser"]:::external
        Traefik["Traefik (API Gateway)"]:::gateway
        NextJS["Next.js UI"]:::frontend
    end

    %% Authentication
    subgraph "Authentication Services"
        direction TB
        Keycloak["Keycloak (SSO/OAuth)"]:::infra
    end

    %% Backend Services
    subgraph "Backend Services (Django API)" 
        direction TB
        CMDB["CMDB Service"]:::backend
        Monitor["Monitor Service"]:::backend
        NodeMgmt["Node Mgmt Service"]:::backend
        OpsPilot["OpsPilot Service"]:::backend
        ConsoleMgmt["Console Mgmt Service"]:::backend
        SystemMgmt["System Mgmt Service"]:::backend
        Celery["Celery Workers"]:::backend
    end

    %% Messaging & RPC
    subgraph "Messaging & RPC" 
        direction TB
        NATSBroker["NATS Broker"]:::message
        RPCAdapters["RPC Adapters"]:::message
        NATSClient["NATS Client Lib"]:::message
        RabbitMQ["RabbitMQ (Pilot Event Bus)"]:::message
    end

    %% Edge Agents
    subgraph "Edge Agents" 
        direction TB
        Fusion["Fusion-Collector"]:::edge
        Executor["NATS-Executor"]:::edge
        Stargazer["Stargazer"]:::edge
    end

    %% AI & Inference Microservices
    subgraph "AI & Inference Services"
        direction TB
        Metis["Metis (Anomaly/RAG)"]:::ai
        PilotSvc["Pilot (Conversational)"]:::ai
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Postgres["PostgreSQL Clusters"]:::db
        Neo4j["Neo4j (CMDB Graph)"]:::db
        VMetrics["VictoriaMetrics"]:::db
        Elasticsearch["Elasticsearch"]:::db
        RedisDB["Redis (Cache/Broker)"]:::db
        MinIO["MinIO/S3"]:::db
    end

    %% Infrastructure & Orchestration
    subgraph "Infra & Orchestration"
        direction TB
        Compose["Docker Compose Manifests"]:::infra
        KubeBootstrapInfra["K8s Infra Bootstrap"]:::infra
        KubeBootstrapPilot["K8s OpsPilot Bootstrap"]:::infra
    end

    %% Connections
    Browser -->|HTTP/HTTPS| Traefik
    Traefik -->|Routes| NextJS
    Traefik -->|Ingress| CMDB
    Traefik -->|Ingress| Monitor
    Traefik -->|Ingress| NodeMgmt
    Traefik -->|Ingress| OpsPilot
    Traefik -->|Ingress| ConsoleMgmt
    Traefik -->|Ingress| SystemMgmt

    Browser -->|Auth Flow| Keycloak
    Keycloak -->|JWT/OAuth| Traefik

    NextJS -->|REST/GraphQL| CMDB
    NextJS -->|REST/GraphQL| Monitor
    NextJS -->|REST/GraphQL| NodeMgmt
    NextJS -->|REST/GraphQL| OpsPilot
    NextJS -->|REST/GraphQL| ConsoleMgmt
    NextJS -->|REST/GraphQL| SystemMgmt

    CMDB -->|RPC⇄| NATSBroker
    Monitor -->|RPC⇄| NATSBroker
    NodeMgmt -->|RPC⇄| NATSBroker
    OpsPilot -->|RPC⇄| NATSBroker
    ConsoleMgmt -->|RPC⇄| NATSBroker
    SystemMgmt -->|RPC⇄| NATSBroker

    CMDB -->|Async Tasks| Celery
    Monitor -->|Async Tasks| Celery
    NodeMgmt -->|Async Tasks| Celery
    OpsPilot -->|Async Tasks| Celery
    ConsoleMgmt -->|Async Tasks| Celery
    SystemMgmt -->|Async Tasks| Celery

    Celery -->|Broker| RedisDB
    Celery -->|Broker| RabbitMQ

    NATSBroker -->|Pub/Sub| Fusion
    NATSBroker -->|RPC Commands| Executor
    NATSBroker -->|Metrics Push| Fusion

    Fusion -->|Metrics| VMetrics
    Fusion -->|Push via Telegraf| VMetrics

    Metis -->|HTTP/gRPC| NextJS
    Metis -->|Reads/Writes| Elasticsearch
    Metis -->|Reads/Writes| MinIO

    PilotSvc -->|Connects Channels| NextJS
    PilotSvc -->|Event Bus| RabbitMQ
    PilotSvc -->|LLM Calls| Metis

    CMDB -->|Data| Postgres
    Monitor -->|Data| Postgres
    NodeMgmt -->|Data| Postgres
    OpsPilot -->|Data| Postgres
    ConsoleMgmt -->|Data| Postgres
    SystemMgmt -->|Data| Postgres

    CMDB -->|Graph| Neo4j

    %% Infra manifests
    Compose -->|Dev Env| Traefik
    Compose -->|Dev Env| NATSBroker
    Compose -->|Dev Env| RedisDB
    Compose -->|Dev Env| VMetrics

    KubeBootstrapInfra -->|Provision| Traefik
    KubeBootstrapInfra -->|Provision| Keycloak
    KubeBootstrapInfra -->|Provision| Postgres
    KubeBootstrapInfra -->|Provision| NATSBroker
    KubeBootstrapInfra -->|Provision| RedisDB
    KubeBootstrapInfra -->|Provision| MinIO
    KubeBootstrapInfra -->|Provision| Elasticsearch

    KubeBootstrapPilot -->|Provision| RabbitMQ
    KubeBootstrapPilot -->|Provision| OpsPilot

    %% Click Events
    click NextJS "https://github.com/weops-lab/bk-lite/tree/master/web/"
    click Traefik "https://github.com/weops-lab/bk-lite/blob/master/deploy/docker-compose/conf/traefik/dynamic.yml"
    click Keycloak "https://github.com/weops-lab/bk-lite/tree/master/deploy/kubernetes/infra/keycloak/"
    click CMDB "https://github.com/weops-lab/bk-lite/tree/master/server/apps/cmdb/"
    click Monitor "https://github.com/weops-lab/bk-lite/tree/master/server/apps/monitor/"
    click NodeMgmt "https://github.com/weops-lab/bk-lite/tree/master/server/apps/node_mgmt/"
    click OpsPilot "https://github.com/weops-lab/bk-lite/tree/master/server/apps/opspilot/"
    click ConsoleMgmt "https://github.com/weops-lab/bk-lite/tree/master/server/apps/console_mgmt/"
    click SystemMgmt "https://github.com/weops-lab/bk-lite/tree/master/server/apps/system_mgmt/"
    click Celery "https://github.com/weops-lab/bk-lite/blob/master/server/apps/core/celery.py"
    click NATSClient "https://github.com/weops-lab/bk-lite/tree/master/server/nats_client/"
    click RPCAdapters "https://github.com/weops-lab/bk-lite/tree/master/server/apps/rpc/"
    click Fusion "https://github.com/weops-lab/bk-lite/tree/master/agents/fusion-collector/"
    click Executor "https://github.com/weops-lab/bk-lite/tree/master/agents/nats-executor/"
    click Stargazer "https://github.com/weops-lab/bk-lite/tree/master/agents/stargazer/"
    click Metis "https://github.com/weops-lab/bk-lite/tree/master/metis/"
    click PilotSvc "https://github.com/weops-lab/bk-lite/tree/master/pilot/"
    click Postgres "https://github.com/weops-lab/bk-lite/tree/master/deploy/kubernetes/infra/postgres/"
    click RabbitMQ "https://github.com/weops-lab/bk-lite/tree/master/deploy/kubernetes/ops-pilot/rabbitmq/"
    click Elasticsearch "https://github.com/weops-lab/bk-lite/tree/master/infra/support-files/images/elasticsearch/"
    click VMetrics "https://github.com/weops-lab/bk-lite/blob/master/deploy/docker-compose/conf/telegraf/telegraf.conf"
    click Compose "https://github.com/weops-lab/bk-lite/tree/master/deploy/docker-compose/"
    click KubeBootstrapInfra "https://github.com/weops-lab/bk-lite/blob/master/deploy/kubernetes/infra/bootstrap.sh"
    click KubeBootstrapPilot "https://github.com/weops-lab/bk-lite/blob/master/deploy/kubernetes/ops-pilot/bootstrap.sh"

    %% Styles
    classDef external fill:#D6E9FE,stroke:#1E70BF,color:#1E70BF;
    classDef frontend fill:#E3F2FD,stroke:#64B5F6,color:#0D47A1;
    classDef gateway fill:#BBDEFB,stroke:#42A5F5,color:#1565C0;
    classDef backend fill:#E8F5E9,stroke:#66BB6A,color:#1B5E20;
    classDef message fill:#FFF3E0,stroke:#FFA726,color:#E65100;
    classDef edge fill:#EEEEEE,stroke:#BDBDBD,color:#424242;
    classDef ai fill:#FFF9C4,stroke:#FFEE58,color:#F57F17;
    classDef db fill:#F3E5F5,stroke:#BA68C8,color:#4A148C;
    classDef infra fill:#ECEFF1,stroke:#90A4AE,color:#37474F;
```
