# axual/ksml architecture

> Architecture diagram of the GitHub repository axual/ksml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axual/ksml
- Repository: https://github.com/axual/ksml

## Overview

##### Project type and purpose
- This is an open-source, multi-module Java application and library (built with Maven) that provides a low-code wrapper around Kafka Streams.
- Users define streaming pipelines in YAML (KSML DSL), then the “ksml-runner” parses the definition, generates a Kafka Streams topology, and runs it. An auxiliary module (“ksml-query”) exposes the running app’s state stores over REST.

##### File structure
- Top-level modules (in pom.xml) map to functional layers:
- ksml-data (core data types & schema) + ksml-data-* extensions (avro, json, csv, protobuf, soap, xml)
- ksml (DSL parser, topology generator, execution engine)
- ksml-kafka-clients (namespaced Kafka producer/consumer wrappers)
- ksml-query (REST server for interactive queries against state stores)
- ksml-runner (CLI app packaging the above, providing backends)
- packaging/helm-charts (Kubernetes deployment)
- Supporting items: Dockerfile, docker-compose.yml, GitHub workflows, docs/, examples/.

##### README for additional insights
- KSML uses GraalVM to run embedded Python user-code snippets.
- Runner targets two backends: KafkaStreamsRunner (full streams) and KafkaProducerRunner (data-generator use cases).
- Metrics are exported to Prometheus.
- Query module is a JAX-RS REST service that uses Kafka Streams interactive queries to expose state.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "DSL Layer"
        YAML["KSML YAML Definitions"]:::external
        click YAML "https://github.com/axual/ksml/tree/main/ksml-runner/src/main/java/io/axual/ksml/runner"
        YAML --> ConfigLoader
        ConfigLoader["Configuration Loader"]:::core
        click ConfigLoader "https://github.com/axual/ksml/tree/main/ksml-runner/src/main/java/io/axual/ksml/runner/config"
        ConfigLoader --> DSLParser
        DSLParser["DSL Parser & Definition Objects"]:::core
        click DSLParser "https://github.com/axual/ksml/tree/main/ksml/src/main/java/io/axual/ksml/parser"
        click DSLParser "https://github.com/axual/ksml/tree/main/ksml/src/main/java/io/axual/ksml/definition"
    end

    subgraph "Runner (ksml-runner)"
        Topo["TopologyGenerator"]:::core
        click Topo "https://github.com/axual/ksml/blob/main/ksml/src/main/java/io/axual/ksml/TopologyGenerator.java"
        Exec["ExecutionContext & ErrorHandler"]:::core
        click Exec "https://github.com/axual/ksml/tree/main/ksml/src/main/java/io/axual/ksml/execution"
        BackendI["Backend Strategy Interface"]:::core
        click BackendI "https://github.com/axual/ksml/tree/main/ksml-runner/src/main/java/io/axual/ksml/runner/backend"
        subgraph Backends
            KR["KafkaStreamsRunner"]:::core
            KP["KafkaProducerRunner"]:::core
        end
        Python["GraalVM Python Bridge"]:::core
        click Python "https://github.com/axual/ksml/tree/main/ksml/src/main/java/io/axual/ksml/python"
        PromExp["Prometheus Exporter"]:::core
        click PromExp "https://github.com/axual/ksml/blob/main/ksml-runner/src/main/java/io/axual/ksml/runner/prometheus/PrometheusExport.java"
    end

    DSLParser --> Topo
    Topo --> Exec
    Exec --> BackendI
    BackendI --> KR
    BackendI --> KP
    Exec --> Python
    Exec --> PromExp

    subgraph "Data & Kafka Clients"
        DataCore["Core Data Library"]:::data
        click DataCore "https://github.com/axual/ksml/tree/main/ksml-data/src/main/java/io/axual/ksml/data"
        subgraph "Data Notations"
            Avro["Avro"]:::data
            click Avro "https://github.com/axual/ksml/tree/main/ksml-data-avro/src/main/java/io/axual/ksml/data/notation/avro"
            JSON["JSON"]:::data
            click JSON "https://github.com/axual/ksml/tree/main/ksml-data-json/src/main/java/io/axual/ksml/data/notation/json"
            CSV["CSV"]:::data
            click CSV "https://github.com/axual/ksml/tree/main/ksml-data-csv/src/main/java/io/axual/ksml/data/notation/csv"
            Protobuf["Protobuf"]:::data
            click Protobuf "https://github.com/axual/ksml/tree/main/ksml-data-protobuf/src/main/java/io/axual/ksml/data/notation/protobuf"
            SOAP["SOAP"]:::data
            click SOAP "https://github.com/axual/ksml/tree/main/ksml-data-soap/src/main/java/io/axual/ksml/data/notation/soap"
            XML["XML"]:::data
            click XML "https://github.com/axual/ksml/tree/main/ksml-data-xml/src/main/java/io/axual/ksml/data/notation/xml"
        end
        Clients["Kafka Clients Wrappers"]:::data
        click Clients "https://github.com/axual/ksml/tree/main/ksml-kafka-clients/src/main/java/io/axual/ksml/client"
    end

    Topo --> DataCore
    Topo --> Avro
    Topo --> JSON
    Topo --> CSV
    Topo --> Protobuf
    Topo --> SOAP
    Topo --> XML
    Topo --> Clients

    subgraph "Infrastructure"
        Kafka["Apache Kafka Cluster"]:::infra
        State["RocksDB State Stores"]:::infra
        Prometheus["Prometheus Server"]:::infra
        Kubernetes["Kubernetes (Helm)"]:::infra
        click Kubernetes "https://github.com/axual/ksml/tree/main/packaging/helm-charts/ksml"
    end

    KR --> Kafka
    KP --> Kafka
    Kafka --> State
    PromExp --> Prometheus

    subgraph "Query Service"
        Q["ksml-query REST Server"]:::service
        click Q "https://github.com/axual/ksml/tree/main/ksml-query/src/main/java/io/axual/ksml/rest"
    end

    State --> Q
    Q -->|"REST API"| ExternalClient["External Clients"]:::external

    Kubernetes --> KR
    Kubernetes --> Q
    Kubernetes --> Prometheus

    classDef core fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef data fill:#90EE90,stroke:#333,stroke-width:1px
    classDef infra fill:#FFA500,stroke:#333,stroke-width:1px
    classDef service fill:#DA70D6,stroke:#333,stroke-width:1px
    classDef external fill:#EEEEEE,stroke:#333,stroke-width:1px
```
