How julien-duponchelle/python-mysql-replication works
Project Type and Purpose
• This is a pure-Python library (an open-source tool) for Change Data Capture (CDC) from MySQL/MariaDB.
• It implements the MySQL replication protocol on top of PyMySQL to stream binary-log events (INSERT, UPDATE, DELETE, GTID, etc.) and expose them as Python objects.
File Structure Analysis
• Top-level directories:
• pymysqlreplication/ – core library modules (binlogstream, event parsing, constants, utils).
• pymysqlreplication/tests/ – unit and benchmark tests validating protocol support.
• examples/ – sample consumer scripts showing integrations (Kafka, RabbitMQ, Redis, Logstash).
• docs/ – Sphinx documentation sources.
• scripts/ – utility scripts (MySQL installation, linting).
• Key files:
• setup.py / pyproject.toml – packaging & dependencies.
• docker-compose*.yml – local test MySQL brokers.
• .github/workflows/*.yml – CI: lint, pytest, release.
README Insights
• Describes use cases: streaming to NoSQL, search engines, cache invalidation, analytics, audit.
• Mentions supported MySQL versions, server settings (binlog_format=ROW, metadata=FULL).
• Links to examples of integration with external systems (Kinesis, Kafka, Snowflake, etc.).
a. Main Components
1. Client Application – your Python code that instantiates BinLogStreamReader.
2. CDC Library Core
• Connection Layer (PyMySQL adapter)
• BinLogStreamReader (network I/O, fetch raw binlog packets)
• Packet Parser → Event Factory
• Event Objects (WriteRowsEvent, UpdateRowsEvent, DeleteRowsEvent, QueryEvent, GTIDEvent, etc.)
• Utilities & Constants (byte-level decoding, field types)
3. MySQL/MariaDB Server – hosts binary logs with replication settings enabled
4. Optional Sink/Adapters – downstream systems (Kafka topic, RabbitMQ queue, Redis cache, Elasticsearch, Snowflake, custom callback)
b. Relationships and Interactions
• BinLogStreamReader ↔ MySQL Server over TCP (protocol handshake, authentication, binlog stream).
• MySQL Server → raw binlog data stream → Connection Layer.
• Connection Layer → Packet Parser → Event Objects → passed back to Client (iteration or callback).
• Client Application → (optional) Sink/Adapters (push to Kafka, call webhooks, write logs).
c. Architectural Patterns & Principles
• Layered pattern:
– Transport layer (PyMySQL)
– Stream processing layer (BinLogStreamReader)
– DSL layer (Event Objects + constants)
– Application layer (user callbacks/sinks)
• Separation of concerns: connection management, binary decoding, event modeling are decoupled.
• Extensibility: new Event types or sinks can be added without modifying core parser.
• Configuration-driven: connection settings, included schemas/tables, server_id.
d. Key Technologies & Frameworks
• Python 3.x
• PyMySQL (MySQL client driver)
• Sphinx (docs)
• pytest (unit tests)
• Docker Compose (integration test MySQL instances)
• CI: GitHub Actions
• Packaging: setuptools / pyproject.toml