# shchelchkov/r-function architecture

> Architecture diagram of the GitHub repository shchelchkov/r-function, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shchelchkov/r-function
- Repository: https://github.com/shchelchkov/r-function

## Overview

r-function is a configurable Kafka message-processing service. Its main workflow consumes Kafka work, dispatches it through a pipeline of configured processors, runs user logic in Wasm or plugins, and publishes results back to Kafka. Git-backed settings describe catalogs, consumers, functions, function values, and streams; the processor also has read/write access to runtime values. The service exposes HTTP and WebSocket routes for settings, values, state, and refresh operations. The README documents the major runtime branches; sampled source confirms pipeline dispatch, API routing, outbound send dispatch, and example function behavior, but not all wiring between subsystems.

Main entry points: [`r-setting/src/git/setting_store.rs`](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/git/setting_store.rs) (Git settings), [`r-runtime/src/runtime/host/mod.rs`](https://github.com/shchelchkov/r-function/blob/main/r-runtime/src/runtime/host/mod.rs) (Runtime host), [`r-router/src/route/routes.rs`](https://github.com/shchelchkov/r-function/blob/main/r-router/src/route/routes.rs) (HTTP API), [`src/pipeline/worker.rs`](https://github.com/shchelchkov/r-function/blob/main/src/pipeline/worker.rs) (Pipeline workers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingress["Message ingress"]
  node_consumer["Kafka consumer<br/>[consumer.rs]"]
  node_pipeline["Pipeline<br/>[pipeline.rs]"]
  node_dispatcher["Work dispatcher<br/>[dispatcher.rs]"]
  node_worker["Pipeline workers<br/>[worker.rs]"]
end

subgraph group_processing["Processing runtime"]
  node_processor["Process engine<br/>[processor.rs]"]
  node_chain["Process chain<br/>[chain.rs]"]
  node_wasm["Wasm runtime<br/>[wasm.rs]"]
  node_runtime_host["Runtime host<br/>[mod.rs]"]
  node_plugin["Plugin runtime<br/>[plugin.rs]"]
  node_plugin_host["Plugin host<br/>[mod.rs]"]
end

subgraph group_configuration["Settings and state"]
  node_git_settings["Git settings<br/>[setting_store.rs]"]
  node_setting_catalogs["Pipeline settings<br/>[catalog.rs]"]
  node_function_settings["Function settings"]
  node_stream_settings["Stream settings<br/>[stream_setting.rs]"]
  node_function_values["Function values<br/>[functions_value.rs]"]
  node_value_db[("Runtime values<br/>[db.rs]")]
end

subgraph group_egress["Message egress"]
  node_producer["Kafka producer<br/>[producer.rs]"]
  node_send_pipeline["Send pipeline<br/>[send_pipeline.rs]"]
end

subgraph group_management["Service management"]
  node_http_api["HTTP API<br/>[routes.rs]"]
  node_app["Service bootstrap<br/>[bootstrap.rs]"]
end

node_kafka(("Kafka"))
node_api_client(("API client"))

node_kafka -->|"delivers messages"| node_consumer
node_consumer -->|"submits work"| node_pipeline
node_pipeline -->|"starts dispatcher"| node_dispatcher
node_dispatcher -->|"dispatches work"| node_worker
node_worker -->|"processes work"| node_processor
node_processor -->|"runs chain"| node_chain
node_chain -->|"executes Wasm"| node_wasm
node_chain -.->|"executes plugin"| node_plugin
node_wasm -->|"calls host"| node_runtime_host
node_plugin -.->|"calls host"| node_plugin_host
node_git_settings -->|"provides settings"| node_setting_catalogs
node_git_settings -->|"provides settings"| node_function_settings
node_git_settings -->|"provides settings"| node_stream_settings
node_git_settings -->|"provides values"| node_function_values
node_runtime_host -->|"reads settings"| node_function_settings
node_runtime_host -->|"reads values"| node_function_values
node_runtime_host -->|"reads and writes"| node_value_db
node_plugin_host -.->|"reads and writes"| node_value_db
node_worker -->|"submits sends"| node_send_pipeline
node_send_pipeline -->|"enqueues messages"| node_producer
node_producer -->|"publishes messages"| node_kafka
node_api_client -->|"requests routes"| node_http_api
node_app -->|"serves routes"| node_http_api
node_http_api -->|"refreshes settings"| node_git_settings
node_http_api -->|"reads and writes"| node_value_db

click node_consumer "https://github.com/shchelchkov/r-function/blob/main/r-consumer/src/kafka/consumer.rs"
click node_pipeline "https://github.com/shchelchkov/r-function/blob/main/src/pipeline.rs"
click node_dispatcher "https://github.com/shchelchkov/r-function/blob/main/src/pipeline/dispatcher.rs"
click node_worker "https://github.com/shchelchkov/r-function/blob/main/src/pipeline/worker.rs"
click node_processor "https://github.com/shchelchkov/r-function/blob/main/r-process/src/process/processor.rs"
click node_chain "https://github.com/shchelchkov/r-function/blob/main/r-process/src/process/chain.rs"
click node_wasm "https://github.com/shchelchkov/r-function/blob/main/r-runtime/src/runtime/wasm.rs"
click node_runtime_host "https://github.com/shchelchkov/r-function/blob/main/r-runtime/src/runtime/host/mod.rs"
click node_plugin "https://github.com/shchelchkov/r-function/blob/main/r-plugin/src/plugin/plugin.rs"
click node_plugin_host "https://github.com/shchelchkov/r-function/blob/main/r-plugin/src/host/mod.rs"
click node_git_settings "https://github.com/shchelchkov/r-function/blob/main/r-setting/src/git/setting_store.rs"
click node_setting_catalogs "https://github.com/shchelchkov/r-function/blob/main/r-setting/src/catalogs/catalog.rs"
click node_function_settings "https://github.com/shchelchkov/r-function/blob/main/r-setting/src/functions/function_setting.rs"
click node_stream_settings "https://github.com/shchelchkov/r-function/blob/main/r-setting/src/streams/stream_setting.rs"
click node_function_values "https://github.com/shchelchkov/r-function/blob/main/r-setting/src/functions/functions_value.rs"
click node_value_db "https://github.com/shchelchkov/r-function/blob/main/r-db/src/db/db.rs"
click node_producer "https://github.com/shchelchkov/r-function/blob/main/r-producer/src/kafka/producer.rs"
click node_send_pipeline "https://github.com/shchelchkov/r-function/blob/main/r-producer/src/host/send_pipeline.rs"
click node_http_api "https://github.com/shchelchkov/r-function/blob/main/r-router/src/route/routes.rs"
click node_app "https://github.com/shchelchkov/r-function/blob/main/src/bootstrap.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_consumer,node_pipeline,node_dispatcher,node_worker,node_api_client toneBlue
class node_processor,node_chain,node_wasm,node_runtime_host,node_plugin,node_plugin_host toneAmber
class node_git_settings,node_setting_catalogs,node_function_settings,node_stream_settings,node_function_values,node_value_db toneMint
class node_producer,node_send_pipeline toneRose
class node_http_api,node_app,node_kafka toneIndigo
```

## Components

### Message ingress

- [Kafka consumer](https://github.com/shchelchkov/r-function/blob/main/r-consumer/src/kafka/consumer.rs): `r-consumer/src/kafka/consumer.rs`
- [Pipeline](https://github.com/shchelchkov/r-function/blob/main/src/pipeline.rs): `src/pipeline.rs`
- [Work dispatcher](https://github.com/shchelchkov/r-function/blob/main/src/pipeline/dispatcher.rs): `src/pipeline/dispatcher.rs`
- [Pipeline workers](https://github.com/shchelchkov/r-function/blob/main/src/pipeline/worker.rs): `src/pipeline/worker.rs`

### Processing runtime

- [Process engine](https://github.com/shchelchkov/r-function/blob/main/r-process/src/process/processor.rs): `r-process/src/process/processor.rs`
- [Process chain](https://github.com/shchelchkov/r-function/blob/main/r-process/src/process/chain.rs): `r-process/src/process/chain.rs`
- [Wasm runtime](https://github.com/shchelchkov/r-function/blob/main/r-runtime/src/runtime/wasm.rs): `r-runtime/src/runtime/wasm.rs`
- [Runtime host](https://github.com/shchelchkov/r-function/blob/main/r-runtime/src/runtime/host/mod.rs): `r-runtime/src/runtime/host/mod.rs`
- [Plugin runtime](https://github.com/shchelchkov/r-function/blob/main/r-plugin/src/plugin/plugin.rs): `r-plugin/src/plugin/plugin.rs`
- [Plugin host](https://github.com/shchelchkov/r-function/blob/main/r-plugin/src/host/mod.rs): `r-plugin/src/host/mod.rs`

### Settings and state

- [Git settings](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/git/setting_store.rs): `r-setting/src/git/setting_store.rs`
- [Pipeline settings](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/catalogs/catalog.rs): `r-setting/src/catalogs/catalog.rs`
- [Function settings](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/functions/function_setting.rs): `r-setting/src/functions/function_setting.rs`
- [Stream settings](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/streams/stream_setting.rs): `r-setting/src/streams/stream_setting.rs`
- [Function values](https://github.com/shchelchkov/r-function/blob/main/r-setting/src/functions/functions_value.rs): `r-setting/src/functions/functions_value.rs`
- [Runtime values](https://github.com/shchelchkov/r-function/blob/main/r-db/src/db/db.rs): `r-db/src/db/db.rs`

### Message egress

- [Kafka producer](https://github.com/shchelchkov/r-function/blob/main/r-producer/src/kafka/producer.rs): `r-producer/src/kafka/producer.rs`
- [Send pipeline](https://github.com/shchelchkov/r-function/blob/main/r-producer/src/host/send_pipeline.rs): `r-producer/src/host/send_pipeline.rs`

### Service management

- [HTTP API](https://github.com/shchelchkov/r-function/blob/main/r-router/src/route/routes.rs): `r-router/src/route/routes.rs`
- [Service bootstrap](https://github.com/shchelchkov/r-function/blob/main/src/bootstrap.rs): `src/bootstrap.rs`

### Other components

- **Kafka**
- **API client**

## Connections

- Kafka → Kafka consumer: delivers messages
- Kafka consumer → Pipeline: submits work
- Pipeline → Work dispatcher: starts dispatcher
- Work dispatcher → Pipeline workers: dispatches work
- Pipeline workers → Process engine: processes work
- Process engine → Process chain: runs chain
- Process chain → Wasm runtime: executes Wasm
- Process chain → Plugin runtime: executes plugin
- Wasm runtime → Runtime host: calls host
- Plugin runtime → Plugin host: calls host
- Git settings → Pipeline settings: provides settings
- Git settings → Function settings: provides settings
- Git settings → Stream settings: provides settings
- Git settings → Function values: provides values
- Runtime host → Function settings: reads settings
- Runtime host → Function values: reads values
- Runtime host → Runtime values: reads and writes
- Plugin host → Runtime values: reads and writes
- Pipeline workers → Send pipeline: submits sends
- Send pipeline → Kafka producer: enqueues messages
- Kafka producer → Kafka: publishes messages
- API client → HTTP API: requests routes
- Service bootstrap → HTTP API: serves routes
- HTTP API → Git settings: refreshes settings
- HTTP API → Runtime values: reads and writes