How rayelg/kafgres works
kafgres embeds a Kafka-compatible broker in PostgreSQL. Kafka clients send protocol requests to the broker event loop, which dispatches API handlers that manage topics, produce and fetch records, consumer groups, transactions, and administration against PostgreSQL-backed metadata and log storage. The README also documents SQL-transactional production and a separate logical-decoding CDC path. Source excerpts show broker dispatch and storage boundaries; several feature-specific links below rely on README documentation because their implementation files were not sampled.
Main entry points: handlers/mod.rs API handlers · storage/mod.rs Log storage API · server/mod.rs Broker event loop · cdc.rs Logical-decoding CDC
Components 22
Broker protocol
- Broker event loop
extension/src/server/mod.rs - API handlers
extension/src/handlers/mod.rs - Wire codec
codec/src/records.rs
Kafka capabilities
- Topic administration
extension/src/handlers/topics.rs - Produce and fetch
extension/src/handlers/produce.rs - Consumer groups
extension/src/handlers/consumer_group.rs - Share groups
extension/src/handlers/share_group.rs - Transactions
extension/src/handlers/txn.rs
State and storage
- Topic metadata
extension/src/meta.rs - Log storage API
extension/src/storage/mod.rs - Table engine
extension/src/storage/table.rs - Segment engine
extension/src/storage/segment.rs - Retention
extension/src/retention.rs - Segment archiving
extension/src/archive.rs - Standby replication
extension/src/replication.rs
Security
- Authentication
extension/src/sasl.rs - ACL authorization
extension/src/acl.rs
Change delivery
- SQL production
extension/src/produce_sql.rs - Logical-decoding CDC
extension/src/cdc.rs - Output decoder
extension/src/decoding.rs
Other components
- Kafka clients
- PostgreSQL
Connections 25
- Kafka clients → Broker event loop: send requests
README.md - Broker event loop → API handlers: dispatches
extension/src/server/mod.rs - API handlers → Wire codec: decodes messages
extension/src/handlers/mod.rs - Broker event loop → Authentication: checks SASL
extension/src/server/mod.rs - API handlers → Topic administration: routes APIs
extension/src/handlers/mod.rs - API handlers → Produce and fetch: routes APIs
extension/src/handlers/mod.rs - API handlers → Consumer groups: routes APIs
extension/src/handlers/mod.rs - API handlers → Share groups: routes APIs
extension/src/handlers/mod.rs - API handlers → Transactions: routes APIs
extension/src/handlers/mod.rs - Broker event loop → ACL authorization: authorizes requests
extension/src/server/mod.rs - Topic administration → Topic metadata: manages topics
README.md - Produce and fetch → Log storage API: reads and writes
README.md - Consumer groups → Topic metadata: uses metadata
extension/src/handlers/consumer_group.rs - Share groups → Log storage API: uses log state
extension/src/handlers/share_group.rs - Transactions → Log storage API: writes markers
extension/src/handlers/txn.rs - Log storage API → Table engine: selects engine
extension/src/storage/mod.rs - Log storage API → Segment engine: selects engine
extension/src/storage/mod.rs - Retention → Log storage API: reclaims log
extension/src/retention.rs - Segment archiving → Segment engine: archives segments
extension/src/archive.rs - Standby replication → Segment engine: replicates log
README.md - SQL production → Log storage API: appends transactionally
README.md - PostgreSQL → SQL production: runs SQL function
README.md - PostgreSQL → Logical-decoding CDC: supplies changes
README.md - Logical-decoding CDC → Output decoder: decodes changes
README.md - Logical-decoding CDC → SQL production: produces mapped events
extension/src/cdc.rs