# rayelg/kafgres architecture

> Architecture diagram of the GitHub repository rayelg/kafgres, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rayelg/kafgres
- Repository: https://github.com/rayelg/kafgres

## Overview

kafgres embeds a Kafka-compatible broker in PostgreSQL. Kafka clients send protocol requests to the broker event loop, which dispatches API handlers that manage topics, produce and fetch records, consumer groups, transactions, and administration against PostgreSQL-backed metadata and log storage. The README also documents SQL-transactional production and a separate logical-decoding CDC path. Source excerpts show broker dispatch and storage boundaries; several feature-specific links below rely on README documentation because their implementation files were not sampled.

Main entry points: [`extension/src/handlers/mod.rs`](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/mod.rs) (API handlers), [`extension/src/storage/mod.rs`](https://github.com/rayelg/kafgres/blob/main/extension/src/storage/mod.rs) (Log storage API), [`extension/src/server/mod.rs`](https://github.com/rayelg/kafgres/blob/main/extension/src/server/mod.rs) (Broker event loop), [`extension/src/cdc.rs`](https://github.com/rayelg/kafgres/blob/main/extension/src/cdc.rs) (Logical-decoding CDC)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_broker["Broker protocol"]
  node_event_loop["Broker event loop<br/>[mod.rs]"]
  node_protocol_handlers["API handlers<br/>[mod.rs]"]
  node_wire_codec["Wire codec<br/>[records.rs]"]
end

subgraph group_domain["Kafka capabilities"]
  node_topic_admin["Topic administration<br/>[topics.rs]"]
  node_produce_fetch["Produce and fetch<br/>[produce.rs]"]
  node_consumer_groups["Consumer groups<br/>[consumer_group.rs]"]
  node_share_groups["Share groups<br/>[share_group.rs]"]
  node_transactions["Transactions<br/>[txn.rs]"]
end

subgraph group_state["State and storage"]
  node_topic_metadata["Topic metadata<br/>[meta.rs]"]
  node_storage_api["Log storage API<br/>[mod.rs]"]
  node_table_engine["Table engine<br/>[table.rs]"]
  node_segment_engine["Segment engine<br/>[segment.rs]"]
  node_retention["Retention<br/>[retention.rs]"]
  node_archive["Segment archiving<br/>[archive.rs]"]
  node_replication["Standby replication<br/>[replication.rs]"]
end

subgraph group_security["Security"]
  node_authentication["Authentication<br/>[sasl.rs]"]
  node_acl["ACL authorization<br/>[acl.rs]"]
end

subgraph group_change["Change delivery"]
  node_sql_produce["SQL production<br/>[produce_sql.rs]"]
  node_cdc["Logical-decoding CDC<br/>[cdc.rs]"]
  node_decoder["Output decoder<br/>[decoding.rs]"]
end

node_clients(("Kafka clients"))
node_postgres[("PostgreSQL")]

node_clients -->|"send requests"| node_event_loop
node_event_loop -->|"dispatches"| node_protocol_handlers
node_protocol_handlers -->|"decodes messages"| node_wire_codec
node_event_loop -->|"checks SASL"| node_authentication
node_protocol_handlers -->|"routes APIs"| node_topic_admin
node_protocol_handlers -->|"routes APIs"| node_produce_fetch
node_protocol_handlers -->|"routes APIs"| node_consumer_groups
node_protocol_handlers -->|"routes APIs"| node_share_groups
node_protocol_handlers -->|"routes APIs"| node_transactions
node_event_loop -->|"authorizes requests"| node_acl
node_topic_admin -->|"manages topics"| node_topic_metadata
node_produce_fetch -->|"reads and writes"| node_storage_api
node_consumer_groups -->|"uses metadata"| node_topic_metadata
node_share_groups -->|"uses log state"| node_storage_api
node_transactions -->|"writes markers"| node_storage_api
node_storage_api -->|"selects engine"| node_table_engine
node_storage_api -->|"selects engine"| node_segment_engine
node_retention -->|"reclaims log"| node_storage_api
node_archive -->|"archives segments"| node_segment_engine
node_replication -.->|"replicates log"| node_segment_engine
node_sql_produce -->|"appends transactionally"| node_storage_api
node_postgres -->|"runs SQL function"| node_sql_produce
node_postgres -->|"supplies changes"| node_cdc
node_cdc -->|"decodes changes"| node_decoder
node_cdc -->|"produces mapped events"| node_sql_produce

click node_event_loop "https://github.com/rayelg/kafgres/blob/main/extension/src/server/mod.rs"
click node_protocol_handlers "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/mod.rs"
click node_wire_codec "https://github.com/rayelg/kafgres/blob/main/codec/src/records.rs"
click node_topic_admin "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/topics.rs"
click node_produce_fetch "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/produce.rs"
click node_consumer_groups "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/consumer_group.rs"
click node_share_groups "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/share_group.rs"
click node_transactions "https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/txn.rs"
click node_topic_metadata "https://github.com/rayelg/kafgres/blob/main/extension/src/meta.rs"
click node_storage_api "https://github.com/rayelg/kafgres/blob/main/extension/src/storage/mod.rs"
click node_table_engine "https://github.com/rayelg/kafgres/blob/main/extension/src/storage/table.rs"
click node_segment_engine "https://github.com/rayelg/kafgres/blob/main/extension/src/storage/segment.rs"
click node_retention "https://github.com/rayelg/kafgres/blob/main/extension/src/retention.rs"
click node_archive "https://github.com/rayelg/kafgres/blob/main/extension/src/archive.rs"
click node_replication "https://github.com/rayelg/kafgres/blob/main/extension/src/replication.rs"
click node_authentication "https://github.com/rayelg/kafgres/blob/main/extension/src/sasl.rs"
click node_acl "https://github.com/rayelg/kafgres/blob/main/extension/src/acl.rs"
click node_sql_produce "https://github.com/rayelg/kafgres/blob/main/extension/src/produce_sql.rs"
click node_cdc "https://github.com/rayelg/kafgres/blob/main/extension/src/cdc.rs"
click node_decoder "https://github.com/rayelg/kafgres/blob/main/extension/src/decoding.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_event_loop,node_protocol_handlers,node_wire_codec,node_clients toneBlue
class node_topic_admin,node_produce_fetch,node_consumer_groups,node_share_groups,node_transactions,node_postgres toneAmber
class node_topic_metadata,node_storage_api,node_table_engine,node_segment_engine,node_retention,node_archive,node_replication toneMint
class node_authentication,node_acl toneRose
class node_sql_produce,node_cdc,node_decoder toneIndigo
```

## Components

### Broker protocol

- [Broker event loop](https://github.com/rayelg/kafgres/blob/main/extension/src/server/mod.rs): `extension/src/server/mod.rs`
- [API handlers](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/mod.rs): `extension/src/handlers/mod.rs`
- [Wire codec](https://github.com/rayelg/kafgres/blob/main/codec/src/records.rs): `codec/src/records.rs`

### Kafka capabilities

- [Topic administration](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/topics.rs): `extension/src/handlers/topics.rs`
- [Produce and fetch](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/produce.rs): `extension/src/handlers/produce.rs`
- [Consumer groups](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/consumer_group.rs): `extension/src/handlers/consumer_group.rs`
- [Share groups](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/share_group.rs): `extension/src/handlers/share_group.rs`
- [Transactions](https://github.com/rayelg/kafgres/blob/main/extension/src/handlers/txn.rs): `extension/src/handlers/txn.rs`

### State and storage

- [Topic metadata](https://github.com/rayelg/kafgres/blob/main/extension/src/meta.rs): `extension/src/meta.rs`
- [Log storage API](https://github.com/rayelg/kafgres/blob/main/extension/src/storage/mod.rs): `extension/src/storage/mod.rs`
- [Table engine](https://github.com/rayelg/kafgres/blob/main/extension/src/storage/table.rs): `extension/src/storage/table.rs`
- [Segment engine](https://github.com/rayelg/kafgres/blob/main/extension/src/storage/segment.rs): `extension/src/storage/segment.rs`
- [Retention](https://github.com/rayelg/kafgres/blob/main/extension/src/retention.rs): `extension/src/retention.rs`
- [Segment archiving](https://github.com/rayelg/kafgres/blob/main/extension/src/archive.rs): `extension/src/archive.rs`
- [Standby replication](https://github.com/rayelg/kafgres/blob/main/extension/src/replication.rs): `extension/src/replication.rs`

### Security

- [Authentication](https://github.com/rayelg/kafgres/blob/main/extension/src/sasl.rs): `extension/src/sasl.rs`
- [ACL authorization](https://github.com/rayelg/kafgres/blob/main/extension/src/acl.rs): `extension/src/acl.rs`

### Change delivery

- [SQL production](https://github.com/rayelg/kafgres/blob/main/extension/src/produce_sql.rs): `extension/src/produce_sql.rs`
- [Logical-decoding CDC](https://github.com/rayelg/kafgres/blob/main/extension/src/cdc.rs): `extension/src/cdc.rs`
- [Output decoder](https://github.com/rayelg/kafgres/blob/main/extension/src/decoding.rs): `extension/src/decoding.rs`

### Other components

- **Kafka clients**
- **PostgreSQL**

## Connections

- Kafka clients → Broker event loop: send requests (evidence: `README.md`)
- Broker event loop → API handlers: dispatches (evidence: `extension/src/server/mod.rs`)
- API handlers → Wire codec: decodes messages (evidence: `extension/src/handlers/mod.rs`)
- Broker event loop → Authentication: checks SASL (evidence: `extension/src/server/mod.rs`)
- API handlers → Topic administration: routes APIs (evidence: `extension/src/handlers/mod.rs`)
- API handlers → Produce and fetch: routes APIs (evidence: `extension/src/handlers/mod.rs`)
- API handlers → Consumer groups: routes APIs (evidence: `extension/src/handlers/mod.rs`)
- API handlers → Share groups: routes APIs (evidence: `extension/src/handlers/mod.rs`)
- API handlers → Transactions: routes APIs (evidence: `extension/src/handlers/mod.rs`)
- Broker event loop → ACL authorization: authorizes requests (evidence: `extension/src/server/mod.rs`)
- Topic administration → Topic metadata: manages topics (evidence: `README.md`)
- Produce and fetch → Log storage API: reads and writes (evidence: `README.md`)
- Consumer groups → Topic metadata: uses metadata (evidence: `extension/src/handlers/consumer_group.rs`)
- Share groups → Log storage API: uses log state (evidence: `extension/src/handlers/share_group.rs`)
- Transactions → Log storage API: writes markers (evidence: `extension/src/handlers/txn.rs`)
- Log storage API → Table engine: selects engine (evidence: `extension/src/storage/mod.rs`)
- Log storage API → Segment engine: selects engine (evidence: `extension/src/storage/mod.rs`)
- Retention → Log storage API: reclaims log (evidence: `extension/src/retention.rs`)
- Segment archiving → Segment engine: archives segments (evidence: `extension/src/archive.rs`)
- Standby replication → Segment engine: replicates log (evidence: `README.md`)
- SQL production → Log storage API: appends transactionally (evidence: `README.md`)
- PostgreSQL → SQL production: runs SQL function (evidence: `README.md`)
- PostgreSQL → Logical-decoding CDC: supplies changes (evidence: `README.md`)
- Logical-decoding CDC → Output decoder: decodes changes (evidence: `README.md`)
- Logical-decoding CDC → SQL production: produces mapped events (evidence: `extension/src/cdc.rs`)
