# lf-edge/ekuiper architecture

> Architecture diagram of the GitHub repository lf-edge/ekuiper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lf-edge/ekuiper
- Repository: https://github.com/lf-edge/ekuiper

## Overview

##### Main Components
- Data Sources – These are implemented in multiple “source” packages (see directories like etc/sources, internal/io/…, extensions/sources, and SDK examples). They ingest data from MQTT, HTTP, file systems, simulators, and even EdgeX or other external systems.
- Rule Engine & Processing Pipeline – The heart of the application. It parses SQL or graph rules (using internal/xsql, internal/processor, and internal/topo) and creates a processing stream with operators including filtering, aggregation, join, windowing, etc. Include components related to the binder, function evaluation, and operators as seen in internal/binder and internal/xsql.
- Sinks – These output processed data to external targets. There are dedicated sink implementations (in etc/sinks, internal/io/sinks, extensions/sinks) that cover file, MQTT, redis, and more.
- Plugin/Extension System – eKuiper is highly extensible with support for custom Sources, Sinks, and UDFs (extensions, internal/plugin, sdk directories). This layer allows both Go- and Python-based extensions.
- Management & API Layer – The project includes a CLI, REST API endpoints, and even a web-based management dashboard (as referenced in the README and seen in various “manager” components in internal/server and internal/service). These components enable rule management, configuration, and integration with container orchestration tools.
- Configuration, Deployment & Infrastructure – There are multiple deployment configuration directories (e.g., deploy/chart for Helm, Dockerfiles in .github/dockerfile and deploy/docker) indicating containerization, cross-compilation, and Kubernetes support.

##### Relationships & Interactions
- Data flows from the ingestion (Source) layer to the rule engine where it is processed using a streaming pipeline. Intermediate components (such as the rule evaluator, binder, and operators) transform, filter, and perform aggregation before passing data along.
- After processing, results are sent to Sinks, which may be external systems (e.g., file system, MQTT broker, database, or analytics dashboard).
- The plugin system allows dynamic extension at key integration points—each extension can register new functionality into the core pipeline.
- An API layer (exposed via REST/CLI) facilitates configuration changes (rules management, stream creation, etc.) and integrates with external management dashboards.
- There is additional infrastructure for logging, monitoring, and even metrics which are spread across internal packages (for example, in internal/conf, metrics, and tracing modules).

##### Architectural Patterns & Design Principles
- Modular Pipeline Architecture: Clearly separate the ingestion, processing (rule engine), and output (sink) stages.
- Microkernel/Plugin Pattern: Emphasize the extensibility of source, sink, and function modules that can be dynamically added.
- Configuration-Driven Deployment: Different build variants and deployment scripts allow tailoring the engine per environment (edge or cloud).
- Separation of Concerns: Distinct modules for configuration, logging, API management, and rule processing ensure maintainability and scalability.

##### Relevant Technologies, Frameworks & Libraries
- Language: The core is built in Go (as seen from cmd/, internal/, and go.mod), with supporting Python SDKs and examples.
- Containerization: Dockerfiles and Helm charts for Kubernetes deployment.
- Messaging and Integration: Support for MQTT (integration with EMQX and NanoMQ) and ZeroMQ are native, along with integration options like EdgeX.
- Extensibility and Plugin Management: Use of shared libraries and registries, especially under internal/plugin and extensions directories.

## Diagram (Mermaid)

```mermaid
graph TD

    subgraph "Ingestion Layer"
        DS1["Data Sources"]:::ingestion
        DS2["MQTT Source"]:::ingestion
        DS3["HTTP Source"]:::ingestion
        DS4["File Source"]:::ingestion
    end

    subgraph "Processing Layer"
        P1["SQL/Graph Parser"]:::processing
        P2["Stream Processor"]:::processing
        P3["Operator Topology"]:::processing
        P4["Binder & Functions"]:::processing
    end

    subgraph "Output Layer"
        S1["Data Sinks"]:::sinks
    end

    subgraph "Plugin/Extension System"
        PL1["Extensions Manager"]:::plugin
        PL2["Extensions Module"]:::plugin
        PL3["Internal Plugins"]:::plugin
        PL4["SDKs"]:::plugin
    end

    subgraph "Management & API Layer"
        M1["CLI (kuiper)"]:::management
        M2["CLI (kuiperd)"]:::management
        M3["API Server"]:::management
        M4["Management Service"]:::management
    end

    subgraph "Deployment & Infrastructure"
        D1["Helm Charts"]:::deployment
        D2["Docker Deployments"]:::deployment
        D3["GitHub Dockerfile"]:::deployment
    end

    %% Data flow connections
    DS1 -->|"ingests data"| P1
    DS2 -->|"ingests data"| P1
    DS3 -->|"ingests data"| P1
    DS4 -->|"ingests data"| P1

    P1 -->|"parses rules"| P2
    P2 -->|"processes stream"| P3
    P3 -->|"organizes operators"| P4
    P4 -->|"outputs results"| S1

    %% Management & API interactions (bidirectional)
    M3 <-->|"configures"| P2
    M1 -->|"triggers"| M3
    M2 -->|"triggers"| M3
    M4 -->|"manages"| M3

    %% Plugin system integration (dotted lines)
    PL1 ---|"extends"| P2
    PL2 ---|"extends"| P2
    PL3 ---|"extends"| P2
    PL4 ---|"extends"| P2

    %% Deployment integrations (dotted lines)
    M3 ---|"deploys via"| D1
    M3 ---|"deploys via"| D2
    M3 ---|"deploys via"| D3

    %% Click Events for Ingestion Layer
    click DS1 "https://github.com/lf-edge/ekuiper/tree/master/etc/sources"
    click DS2 "https://github.com/lf-edge/ekuiper/tree/master/internal/io/mqtt"
    click DS3 "https://github.com/lf-edge/ekuiper/tree/master/internal/io/http"
    click DS4 "https://github.com/lf-edge/ekuiper/tree/master/internal/io/file"

    %% Click Events for Processing Layer
    click P1 "https://github.com/lf-edge/ekuiper/tree/master/internal/xsql"
    click P2 "https://github.com/lf-edge/ekuiper/tree/master/internal/processor"
    click P3 "https://github.com/lf-edge/ekuiper/tree/master/internal/topo"
    click P4 "https://github.com/lf-edge/ekuiper/tree/master/internal/binder"

    %% Click Event for Sinks
    click S1 "https://github.com/lf-edge/ekuiper/tree/master/etc/sinks"

    %% Click Events for Plugin/Extension System
    click PL2 "https://github.com/lf-edge/ekuiper/tree/master/extensions"
    click PL3 "https://github.com/lf-edge/ekuiper/tree/master/internal/plugin"
    click PL4 "https://github.com/lf-edge/ekuiper/tree/master/sdk"

    %% Click Events for Management & API Layer
    click M1 "https://github.com/lf-edge/ekuiper/tree/master/cmd/kuiper"
    click M2 "https://github.com/lf-edge/ekuiper/tree/master/cmd/kuiperd"
    click M3 "https://github.com/lf-edge/ekuiper/tree/master/internal/server"
    click M4 "https://github.com/lf-edge/ekuiper/tree/master/internal/service"

    %% Click Events for Deployment & Infrastructure
    click D1 "https://github.com/lf-edge/ekuiper/tree/master/deploy/chart"
    click D2 "https://github.com/lf-edge/ekuiper/tree/master/deploy/docker"
    click D3 "https://github.com/lf-edge/ekuiper/tree/master/.github/dockerfile"

    %% Styles
    classDef ingestion fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef processing fill:#A9DFBF,stroke:#1E8449,stroke-width:2px;
    classDef sinks fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef plugin fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef management fill:#FAD7A0,stroke:#CA6F1E,stroke-width:2px;
    classDef deployment fill:#D2B4DE,stroke:#633974,stroke-width:2px;
```
