# julien-duponchelle/python-mysql-replication architecture

> Architecture diagram of the GitHub repository julien-duponchelle/python-mysql-replication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/julien-duponchelle/python-mysql-replication
- Repository: https://github.com/julien-duponchelle/python-mysql-replication

## Overview

##### Project Type and Purpose
- This is a pure-Python library (an open-source tool) for Change Data Capture (CDC) from MySQL/MariaDB.
- It implements the MySQL replication protocol on top of PyMySQL to stream binary-log events (INSERT, UPDATE, DELETE, GTID, etc.) and expose them as Python objects.

##### File Structure Analysis
- Top-level directories:
- pymysqlreplication/ – core library modules (binlogstream, event parsing, constants, utils).
- pymysqlreplication/tests/ – unit and benchmark tests validating protocol support.
- examples/ – sample consumer scripts showing integrations (Kafka, RabbitMQ, Redis, Logstash).
- docs/ – Sphinx documentation sources.
- scripts/ – utility scripts (MySQL installation, linting).
- Key files:
- setup.py / pyproject.toml – packaging & dependencies.
- docker-compose*.yml – local test MySQL brokers.
- .github/workflows/*.yml – CI: lint, pytest, release.

##### README Insights
- Describes use cases: streaming to NoSQL, search engines, cache invalidation, analytics, audit.
- Mentions supported MySQL versions, server settings (binlog_format=ROW, metadata=FULL).
- Links to examples of integration with external systems (Kinesis, Kafka, Snowflake, etc.).

a. Main Components
1. Client Application – your Python code that instantiates BinLogStreamReader.
2. CDC Library Core
- Connection Layer (PyMySQL adapter)
- BinLogStreamReader (network I/O, fetch raw binlog packets)
- Packet Parser → Event Factory
- Event Objects (WriteRowsEvent, UpdateRowsEvent, DeleteRowsEvent, QueryEvent, GTIDEvent, etc.)
- Utilities & Constants (byte-level decoding, field types)
3. MySQL/MariaDB Server – hosts binary logs with replication settings enabled
4. Optional Sink/Adapters – downstream systems (Kafka topic, RabbitMQ queue, Redis cache, Elasticsearch, Snowflake, custom callback)

b. Relationships and Interactions
- BinLogStreamReader ↔ MySQL Server over TCP (protocol handshake, authentication, binlog stream).
- MySQL Server → raw binlog data stream → Connection Layer.
- Connection Layer → Packet Parser → Event Objects → passed back to Client (iteration or callback).
- Client Application → (optional) Sink/Adapters (push to Kafka, call webhooks, write logs).

c. Architectural Patterns & Principles
- Layered pattern:
– Transport layer (PyMySQL)
– Stream processing layer (BinLogStreamReader)
– DSL layer (Event Objects + constants)
– Application layer (user callbacks/sinks)
- Separation of concerns: connection management, binary decoding, event modeling are decoupled.
- Extensibility: new Event types or sinks can be added without modifying core parser.
- Configuration-driven: connection settings, included schemas/tables, server_id.

d. Key Technologies & Frameworks
- Python 3.x
- PyMySQL (MySQL client driver)
- Sphinx (docs)
- pytest (unit tests)
- Docker Compose (integration test MySQL instances)
- CI: GitHub Actions
- Packaging: setuptools / pyproject.toml

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Application Layer
    subgraph "Application Layer"
        direction TB
        ClientApp["Client Application<br/>(examples/)"]:::external
    end

    %% CDC Library Core
    subgraph "CDC Library Core" 
        direction TB
        Stream["BinLogStreamReader"]:::core
        Conn["Connection Layer & Packet Reader"]:::core
        Parser["Packet Parser & Event Factory"]:::core
        Events["Event Objects"]:::core
        GTID["GTID Handling"]:::core
        JSON["JSON/Complex Types"]:::core
        Bitmap["Bitmap/Column Mapping"]:::core
        Table["Table Metadata"]:::core
        Excp["Exceptions"]:::core
        Utils["Utilities<br/>(byte‐level decoding)"]:::core
        Const["Protocol Constants"]:::core
    end

    %% External Systems / Sinks
    subgraph "Downstream Systems"
        direction TB
        Kafka["Kafka Topic"]:::sink
        Rabbit["RabbitMQ Queue"]:::sink
        Redis["Redis Cache"]:::sink
        Elastic["Elasticsearch"]:::sink
        Snowflake["Snowflake"]:::sink
        Custom["Custom Callback"]:::sink
    end

    %% Infrastructure
    subgraph "Infrastructure & Support"
        direction TB
        MySQL["MySQL/MariaDB Server"]:::infra
        Tests["Unit & Integration Tests<br/>(pymysqlreplication/tests/)"]:::test
        DockerCI["docker-compose-test.yml"]:::test
        Docs["Documentation (docs/)"]:::infra
        Packaging["Packaging & CI<br/>(setup.py, pyproject.toml,<br/>.github/workflows/)"]:::infra
        Scripts["Utility Scripts<br/>(scripts/)"]:::infra
    end

    %% Data Flows
    ClientApp -->|"start(), stop() calls"| Stream
    Stream <-->|"TCP Protocol"| MySQL
    MySQL -->|"binlog bytes"| Conn
    Conn -->|decode| Parser
    Parser -->|instantiate| Events
    Events -->|yield events| ClientApp
    ClientApp -->|push events| Kafka
    ClientApp -->|push events| Rabbit
    ClientApp -->|push events| Redis
    ClientApp -->|push events| Elastic
    ClientApp -->|push events| Snowflake
    ClientApp -->|callback| Custom

    %% Support Flows
    Const --> Parser
    Utils --> Conn
    Bitmap --> Parser
    JSON --> Parser
    Table --> Parser
    GTID --> Parser
    Excp --> Parser
    Tests --> DockerCI
    DockerCI --> MySQL

    %% Click Events
    click ClientApp "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/examples/"
    click Stream "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py"
    click Conn "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/packet.py"
    click Parser "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/event.py"
    click Parser "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/row_event.py"
    click GTID "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/gtid.py"
    click JSON "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/json_binary.py"
    click Bitmap "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/bitmap.py"
    click Bitmap "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/column.py"
    click Table "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/table.py"
    click Excp "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/exceptions.py"
    click Utils "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/util/bytes.py"
    click Const "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/pymysqlreplication/constants/"
    click Tests "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/pymysqlreplication/tests/"
    click DockerCI "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/docker-compose-test.yml"
    click Docs "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/docs/"
    click Packaging "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/setup.py"
    click Packaging "https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pyproject.toml"
    click Packaging "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/.github/workflows/"
    click Scripts "https://github.com/julien-duponchelle/python-mysql-replication/tree/main/scripts/"

    %% Styles
    classDef core fill:#D0E7FF,stroke:#0000FF,color:#0000A0
    classDef external fill:#E0FFE0,stroke:#00AA00,color:#007700
    classDef sink fill:#E0FFE0,stroke:#00AA00,color:#007700
    classDef infra fill:#FFE4B5,stroke:#FF8C00,color:#A25C00
    classDef test fill:#F0F0F0,stroke:#888888,color:#555555
```
