How danielfelromero/kafka_producer_etl works
This repository is a teaching lab for a SQLite transactional-outbox ETL pipeline, with local and classroom modes. The main workflow is: a user inserts a measurement, SQLite atomically creates an outbox event, the polling producer publishes it to Kafka and marks it delivered, and consumers read the topic. The README also documents a classroom startup path that exposes Kafka through Bore. Source excerpts confirm the database and producer behavior; classroom wiring is represented from README statements, not inspected scripts.
Main entry points: init_db.py Outbox trigger · start_classroom.sh TCP tunnel · producer.py Kafka publishing · insert_record.py Record insertion
Components 14
Record and Outbox
- Record insertion
insert_record.py - SQLite database
- Outbox trigger
init_db.py
Kafka Delivery
- Outbox polling
producer.py - Kafka publishing
producer.py - Kafka broker
- etl.records topic
- Local consumer
Classroom Access
- Student consumers
- Classroom startup
start_classroom.sh - Broker configuration
configure_kafka.py - TCP tunnel
start_classroom.sh
Other components
- Record submitter
- Bore public service
Connections 16
- Record submitter → Record insertion: submits measurement
README.md - Record insertion → SQLite database: inserts record
insert_record.py - Outbox trigger → SQLite database: creates schema and trigger
init_db.py - SQLite database → Outbox trigger: fires after insert
init_db.py - Outbox trigger → SQLite database: writes outbox row
init_db.py - Outbox polling → SQLite database: reads pending events
producer.py - Kafka publishing → Kafka broker: publishes events
producer.py - Kafka broker → etl.records topic: routes to topic
README.md - Kafka publishing → SQLite database: marks delivery status
producer.py - Local consumer → etl.records topic: consumes events
README.md - Student consumers → etl.records topic: consumes events
README.md - Classroom startup → TCP tunnel: opens tunnel
README.md - TCP tunnel → Bore public service: exposes TCP endpoint
README.md - Classroom startup → Broker configuration: generates listeners
README.md - TCP tunnel → Kafka broker: forwards external traffic
README.md - Student consumers → Bore public service: connects through
README.md