How lf-edge/ekuiper works
Main Components
• Data Sources – These are implemented in multiple “source” packages (see directories like etc/sources, internal/io/…, extensions/sources, and SDK examples). They ingest data from MQTT, HTTP, file systems, simulators, and even EdgeX or other external systems.
• Rule Engine & Processing Pipeline – The heart of the application. It parses SQL or graph rules (using internal/xsql, internal/processor, and internal/topo) and creates a processing stream with operators including filtering, aggregation, join, windowing, etc. Include components related to the binder, function evaluation, and operators as seen in internal/binder and internal/xsql.
• Sinks – These output processed data to external targets. There are dedicated sink implementations (in etc/sinks, internal/io/sinks, extensions/sinks) that cover file, MQTT, redis, and more.
• Plugin/Extension System – eKuiper is highly extensible with support for custom Sources, Sinks, and UDFs (extensions, internal/plugin, sdk directories). This layer allows both Go- and Python-based extensions.
• Management & API Layer – The project includes a CLI, REST API endpoints, and even a web-based management dashboard (as referenced in the README and seen in various “manager” components in internal/server and internal/service). These components enable rule management, configuration, and integration with container orchestration tools.
• Configuration, Deployment & Infrastructure – There are multiple deployment configuration directories (e.g., deploy/chart for Helm, Dockerfiles in .github/dockerfile and deploy/docker) indicating containerization, cross-compilation, and Kubernetes support.
Relationships & Interactions
• Data flows from the ingestion (Source) layer to the rule engine where it is processed using a streaming pipeline. Intermediate components (such as the rule evaluator, binder, and operators) transform, filter, and perform aggregation before passing data along.
• After processing, results are sent to Sinks, which may be external systems (e.g., file system, MQTT broker, database, or analytics dashboard).
• The plugin system allows dynamic extension at key integration points—each extension can register new functionality into the core pipeline.
• An API layer (exposed via REST/CLI) facilitates configuration changes (rules management, stream creation, etc.) and integrates with external management dashboards.
• There is additional infrastructure for logging, monitoring, and even metrics which are spread across internal packages (for example, in internal/conf, metrics, and tracing modules).
Architectural Patterns & Design Principles
• Modular Pipeline Architecture: Clearly separate the ingestion, processing (rule engine), and output (sink) stages.
• Microkernel/Plugin Pattern: Emphasize the extensibility of source, sink, and function modules that can be dynamically added.
• Configuration-Driven Deployment: Different build variants and deployment scripts allow tailoring the engine per environment (edge or cloud).
• Separation of Concerns: Distinct modules for configuration, logging, API management, and rule processing ensure maintainability and scalability.
Relevant Technologies, Frameworks & Libraries
• Language: The core is built in Go (as seen from cmd/, internal/, and go.mod), with supporting Python SDKs and examples.
• Containerization: Dockerfiles and Helm charts for Kubernetes deployment.
• Messaging and Integration: Support for MQTT (integration with EMQX and NanoMQ) and ZeroMQ are native, along with integration options like EdgeX.
• Extensibility and Plugin Management: Use of shared libraries and registries, especially under internal/plugin and extensions directories.