How axual/ksml works
Project type and purpose
• This is an open-source, multi-module Java application and library (built with Maven) that provides a low-code wrapper around Kafka Streams.
• Users define streaming pipelines in YAML (KSML DSL), then the “ksml-runner” parses the definition, generates a Kafka Streams topology, and runs it. An auxiliary module (“ksml-query”) exposes the running app’s state stores over REST.
File structure
• Top-level modules (in pom.xml) map to functional layers:
• ksml-data (core data types & schema) + ksml-data-* extensions (avro, json, csv, protobuf, soap, xml)
• ksml (DSL parser, topology generator, execution engine)
• ksml-kafka-clients (namespaced Kafka producer/consumer wrappers)
• ksml-query (REST server for interactive queries against state stores)
• ksml-runner (CLI app packaging the above, providing backends)
• packaging/helm-charts (Kubernetes deployment)
• Supporting items: Dockerfile, docker-compose.yml, GitHub workflows, docs/, examples/.
README for additional insights
• KSML uses GraalVM to run embedded Python user-code snippets.
• Runner targets two backends: KafkaStreamsRunner (full streams) and KafkaProducerRunner (data-generator use cases).
• Metrics are exported to Prometheus.
• Query module is a JAX-RS REST service that uses Kafka Streams interactive queries to expose state.