# danielfelromero/kafka_producer_etl architecture

> Architecture diagram of the GitHub repository danielfelromero/kafka_producer_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danielfelromero/kafka_producer_etl
- Repository: https://github.com/danielfelromero/kafka_producer_etl

## Overview

This repository is a teaching lab for a SQLite transactional-outbox ETL pipeline, with local and classroom modes. The main workflow is: a user inserts a measurement, SQLite atomically creates an outbox event, the polling producer publishes it to Kafka and marks it delivered, and consumers read the topic. The README also documents a classroom startup path that exposes Kafka through Bore. Source excerpts confirm the database and producer behavior; classroom wiring is represented from README statements, not inspected scripts.

Main entry points: [`init_db.py`](https://github.com/danielfelromero/kafka_producer_etl/blob/main/init_db.py) (Outbox trigger), [`start_classroom.sh`](https://github.com/danielfelromero/kafka_producer_etl/blob/main/start_classroom.sh) (TCP tunnel), [`producer.py`](https://github.com/danielfelromero/kafka_producer_etl/blob/main/producer.py) (Kafka publishing), [`insert_record.py`](https://github.com/danielfelromero/kafka_producer_etl/blob/main/insert_record.py) (Record insertion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Record and Outbox"]
  node_insert["Record insertion<br/>[insert_record.py]"]
  node_sqlite[("SQLite database")]
  node_trigger["Outbox trigger<br/>[init_db.py]"]
end

subgraph group_delivery["Kafka Delivery"]
  node_outbox_poll["Outbox polling<br/>[producer.py]"]
  node_publish["Kafka publishing<br/>[producer.py]"]
  node_broker["Kafka broker"]
  node_topic["etl.records topic"]
  node_local_consumer(("Local consumer"))
end

subgraph group_classroom["Classroom Access"]
  node_students(("Student consumers"))
  node_classroom_start["Classroom startup<br/>[start_classroom.sh]"]
  node_kafka_config["Broker configuration<br/>[configure_kafka.py]"]
  node_bore_tunnel["TCP tunnel<br/>[start_classroom.sh]"]
end

node_user(("Record submitter"))
node_bore_service["Bore public service"]

node_user -->|"submits measurement"| node_insert
node_insert -->|"inserts record"| node_sqlite
node_trigger -->|"creates schema and trigger"| node_sqlite
node_sqlite -->|"fires after insert"| node_trigger
node_trigger -->|"writes outbox row"| node_sqlite
node_outbox_poll -->|"reads pending events"| node_sqlite
node_publish -->|"publishes events"| node_broker
node_broker -->|"routes to topic"| node_topic
node_publish -->|"marks delivery status"| node_sqlite
node_local_consumer -->|"consumes events"| node_topic
node_students -->|"consumes events"| node_topic
node_classroom_start -->|"opens tunnel"| node_bore_tunnel
node_bore_tunnel -->|"exposes TCP endpoint"| node_bore_service
node_classroom_start -->|"generates listeners"| node_kafka_config
node_bore_tunnel -->|"forwards external traffic"| node_broker
node_students -->|"connects through"| node_bore_service

click node_insert "https://github.com/danielfelromero/kafka_producer_etl/blob/main/insert_record.py"
click node_trigger "https://github.com/danielfelromero/kafka_producer_etl/blob/main/init_db.py"
click node_outbox_poll "https://github.com/danielfelromero/kafka_producer_etl/blob/main/producer.py"
click node_publish "https://github.com/danielfelromero/kafka_producer_etl/blob/main/producer.py"
click node_classroom_start "https://github.com/danielfelromero/kafka_producer_etl/blob/main/start_classroom.sh"
click node_kafka_config "https://github.com/danielfelromero/kafka_producer_etl/blob/main/configure_kafka.py"
click node_bore_tunnel "https://github.com/danielfelromero/kafka_producer_etl/blob/main/start_classroom.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_insert,node_sqlite,node_trigger toneBlue
class node_outbox_poll,node_publish,node_broker,node_topic,node_local_consumer toneAmber
class node_students,node_classroom_start,node_kafka_config,node_bore_tunnel toneMint
class node_user,node_bore_service toneIndigo
```

## Components

### Record and Outbox

- [Record insertion](https://github.com/danielfelromero/kafka_producer_etl/blob/main/insert_record.py): `insert_record.py`
- **SQLite database**
- [Outbox trigger](https://github.com/danielfelromero/kafka_producer_etl/blob/main/init_db.py): `init_db.py`

### Kafka Delivery

- [Outbox polling](https://github.com/danielfelromero/kafka_producer_etl/blob/main/producer.py): `producer.py`
- [Kafka publishing](https://github.com/danielfelromero/kafka_producer_etl/blob/main/producer.py): `producer.py`
- **Kafka broker**
- **etl.records topic**
- **Local consumer**

### Classroom Access

- **Student consumers**
- [Classroom startup](https://github.com/danielfelromero/kafka_producer_etl/blob/main/start_classroom.sh): `start_classroom.sh`
- [Broker configuration](https://github.com/danielfelromero/kafka_producer_etl/blob/main/configure_kafka.py): `configure_kafka.py`
- [TCP tunnel](https://github.com/danielfelromero/kafka_producer_etl/blob/main/start_classroom.sh): `start_classroom.sh`

### Other components

- **Record submitter**
- **Bore public service**

## Connections

- Record submitter → Record insertion: submits measurement (evidence: `README.md`)
- Record insertion → SQLite database: inserts record (evidence: `insert_record.py`)
- Outbox trigger → SQLite database: creates schema and trigger (evidence: `init_db.py`)
- SQLite database → Outbox trigger: fires after insert (evidence: `init_db.py`)
- Outbox trigger → SQLite database: writes outbox row (evidence: `init_db.py`)
- Outbox polling → SQLite database: reads pending events (evidence: `producer.py`)
- Kafka publishing → Kafka broker: publishes events (evidence: `producer.py`)
- Kafka broker → etl.records topic: routes to topic (evidence: `README.md`)
- Kafka publishing → SQLite database: marks delivery status (evidence: `producer.py`)
- Local consumer → etl.records topic: consumes events (evidence: `README.md`)
- Student consumers → etl.records topic: consumes events (evidence: `README.md`)
- Classroom startup → TCP tunnel: opens tunnel (evidence: `README.md`)
- TCP tunnel → Bore public service: exposes TCP endpoint (evidence: `README.md`)
- Classroom startup → Broker configuration: generates listeners (evidence: `README.md`)
- TCP tunnel → Kafka broker: forwards external traffic (evidence: `README.md`)
- Student consumers → Bore public service: connects through (evidence: `README.md`)