# apecloud/mysql-binlog-connector-rust architecture

> Architecture diagram of the GitHub repository apecloud/mysql-binlog-connector-rust, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apecloud/mysql-binlog-connector-rust
- Repository: https://github.com/apecloud/mysql-binlog-connector-rust

## Overview

Project Overview:
This project is a Rust library designed to dump MySQL binary logs and parse Row Based Replication Events using asynchronous I/O. It is not a full-stack web application but rather a low-level tool that focuses on reliably processing MySQL binlogs. The library supports multiple MySQL versions (5.6, 5.7, and 8.0) and replication types (file-position-based and GTID-based), and provides detailed support for numerous event types.

Project Structure Insights:
- The top-level files include Cargo.toml, license files, and readme files. There are separate directories for examples and tests.
- The source code is organized in a modular fashion under the "src" directory. Key submodules are clearly defined:
– binlog_client.rs: Likely implements the client that initiates a connection with the MySQL server and begins binlog dumping.
– binlog_parser.rs and binlog_stream.rs: Responsible for reading and parsing incoming binary log events.
– The "event" directory defines various event types, such as WriteRows, UpdateRows, DeleteRows, and others.
– The "command" directory organizes various command types (authentication commands, dumping commands, and query commands).
– The "column" directory (and its "json" subfolder) handles the conversion between MySQL column types and Rust types, especially for binary and JSON data representations.
– The "network" directory defines packet types and the overall communication protocol (greeting, error, result set rows, etc.).
– The "ext" directory likely contains utility extensions that help with buffer processing and cursor movement.
- The "tests" directory provides comprehensive testing for data types, DDL/DML operations, file parsing, and a full integration test.
- The "example" directory shows a minimal use case of the library, demonstrating connection setup, event streaming, and asynchronous operations.

##### Client Component
– BinlogClient (located in binlog_client.rs): Initiates connection based on configuration (such as server ID, binlog file name, and position). It is the entry point for the library user.
– Example code in the "example" directory that demonstrates how to use the client in an asynchronous context.

##### Network Communication Layer
– Components in the "src/network" directory handle various packet types (authentication, greeting, error, and result set packets).
– PacketChannel and related utilities manage the low-level TCP communication with the MySQL server.

##### Parser Component
– BinlogParser (src/binlog_parser.rs): Processes the byte stream and converts it into events.
– Utilizes submodules (columns, events, and ext) to interpret packet data fully.
– Supports both methods: directly reading from a live stream (via network) and parsing from binlog files.

##### Event Handling Layer
– A collection of modules within "src/event" which process each event type: FormatDescriptionEvent, RotateEvent, GTID events, Query events, Transaction events, etc.
– Each event type is encapsulated in its own module, ensuring separation of concerns and modularity.

##### Command Layer
– The "src/command" directory handles MySQL commands required during the connection and during binlog dumping.
– Contains specific implementations of authentication commands (native password, SHA2 variants, etc.), as well as commands for dumping and GTID handling.

##### Data Type and Column Mapping Layer
– The "src/column" directory bridges MySQL column types to Rust types.
– It includes a specialized submodule for handling JSON columns, such as JsonBinary for parsing JSON data.

##### Testing and Examples
– Tests are organized to cover data types, DDL/DML operations, integration, and file-based testing.
– These folders illustrate the detailed expected functionality and usage scenarios of the library.

- The BinlogClient initiates a connection and uses the Network Communication Layer to talk with the MySQL server.
- Data received via the Network Layer is forwarded to the BinlogParser, which reads the raw binary data.
- The parser leverages the Data Type and Column Mapping Layer to interpret the byte streams into strongly typed events.
- Parsed events are then dispatched to the Event Handling Layer, which processes various different replication events.
- The Command Layer is used both during initial handshake (authentication) and during continuous pulling of binlog events.
- The Tests simulate interactions with a live MySQL instance, confirming that events are properly parsed and processed. The Example directory demonstrates a real-world usage scenario.

- Labels and notes to indicate relationships such as:
– The dependency between the Parser and the Data Mapping layer.
– The interactions between the Command layer and the Network Communication layer.
- For the binlog file parsing versus live stream processing, illustrate a bifurcation where the parser can work with an active TCP stream or with a file input.
– Each module (binlog_client, parser, command, network, event, column, ext) is designed to perform a single responsibility, adhering to modular design principles.

- That the separation between the network communication and event parsing is clearly depicted. This is decisive as the asynchronous I/O nature of Rust is central to the design.
- Supporting functionalities such as authentication and command execution since they are critical to how the client connects to MySQL.
- Provide annotation for how data flows from the MySQL server (or a binlog file) through the parsing pipeline and onto the event processing logic.
- Clearly denote integration points where external modules (like Docker-based MySQL instances for testing) interact with the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Core Library"
        A["BinlogClient"]:::core
        B["Network Communication"]:::core
        C["BinlogParser"]:::core
        D["BinlogStream"]:::core
        E["Event Handling"]:::core
        F["Command"]:::core
        G["Data Mapping"]:::core
        H["JSON Mapping"]:::core
    end

    subgraph "Usage"
        I["ExampleClient"]:::example
        J["Tests"]:::test
    end

    A -->|"initiates"| B
    A -->|"uses"| F
    B -->|"sendsDataTo"| C
    B -->|"sendsDataTo"| D
    C -->|"leverages"| G
    G -->|"convertsUsing"| H
    C -->|"dispatchesTo"| E
    F -->|"executesOn"| B
    I -->|"invokes"| A
    J -->|"tests"| A
    J -->|"tests"| E

    click A "https://github.com/apecloud/mysql-binlog-connector-rust/blob/main/src/binlog_client.rs"
    click I "https://github.com/apecloud/mysql-binlog-connector-rust/blob/main/example/src/main.rs"
    click B "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/src/network"
    click C "https://github.com/apecloud/mysql-binlog-connector-rust/blob/main/src/binlog_parser.rs"
    click D "https://github.com/apecloud/mysql-binlog-connector-rust/blob/main/src/binlog_stream.rs"
    click E "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/src/event"
    click F "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/src/command"
    click G "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/src/column"
    click H "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/src/column/json"
    click J "https://github.com/apecloud/mysql-binlog-connector-rust/tree/main/tests"

    classDef core fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef example fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef test fill:#90EE90,stroke:#333,stroke-width:2px;
```
