# moiot/gravity architecture

> Architecture diagram of the GitHub repository moiot/gravity, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moiot/gravity
- Repository: https://github.com/moiot/gravity

## Overview

##### Project type and purpose
- Gravity is an open‐source, Go‑based data‑replication tool (Change Data Capture + batch).
- It captures from various sources (MySQL binlog/scan, MongoDB oplog/scan, TiDB, PostgreSQL WAL) and writes to various sinks (MySQL/TiDB, Kafka, Elasticsearch).
- It can run in a Kubernetes cluster, uses etcd for coordination and Prometheus/Grafana for observability.

##### File structure
- cmd/: CLI entrypoints (gravity server, dcp coordinator, padder, prometheus_etcd_sd, verifier).
- pkg/: core libraries organized by domain – inputs, outputs, filters, matchers, scheduler, emitter, protocol, registry, metrics, utils, etc.
- dcp/: distributed‑coordination primitives (barrier, checker, collector) for multi‑instance synchronization.
- deploy/: Kubernetes/Docker Compose manifests, Grafana dashboards.
- integration_test/, mock/: end‑to‑end and unit mocks.
- protocol/: Protobuf definitions for interprocess communication.

##### README for additional insights
- The “Architecture” doc (docs/2.0/00‑arch‑en.md) describes a modular pipeline: Input → Filter → Matcher → Scheduler → Emitter → Output.
- Mentions distributed cluster capabilities and customization via plugins.

- Source Systems (MySQL, MongoDB, TiDB, PostgreSQL)
- Gravity Input Plugins (binlog tailer, oplog tailer, batch scan)
- Gravity Core Pipeline
– Filters & Matchers
– Scheduler
– Router
– Emitter
- Gravity Output Plugins (MySQL writer, Kafka producer, Elasticsearch client)
- DCP Cluster Services (Barrier, Collector, Checker)
- Padder Service (job processor for late events)
- Verifier Service (consistency check)
- Service Discovery (etcd + prometheus_etcd_sd)
- Observability (Prometheus metrics, Grafana dashboards)
- Registry (plugin registry for dynamic filters/outputs)

1. Source Systems → Input Plugins (connect via MySQL/Mongo wire protocols or SDK)
2. Input Plugins → Core Pipeline (hand off event stream)
3. Core → Filters/Matchers → Scheduler (event ordering) → Emitter
4. Emitter → Output Plugins → Sink Systems (DBs, Kafka, ES)
5. DCP Collector ↔ Gravity instances (gRPC) for offset collection
6. DCP Barrier & Checker synchronize instance states via etcd
7. Padder jobs pull missing transactions and re‑emit into core
8. Verifier reads from sink and source to detect drift
9. Prometheus scraping endpoints on each service; Grafana dashboards render metrics

c. Architectural patterns & design principles
- Plugin architecture: inputs, filters, outputs loaded via registry.
- Modular pipeline: decoupled stages, pluggable at each stage.
- Distributed coordination: DCP using barrier/collector/checker patterns.
- Service discovery: etcd + Prometheus SD for auto‑scaling and cluster management.
- Containerized microservices: each cmd/ service runs in its own container.

d. Key technologies & frameworks
- Go modules (pkg/*)
- gRPC (dcp collector, filters/grpc sidecar)
- Protobuf (protocol/message.proto)
- etcd (coordination & config v3)
- Prometheus & Grafana (metrics)
- Docker/Kubernetes (deployment)
- Makefile & Dockerfiles (build pipelines)

- Configurations loaded from TOML via pkg/config.
- Dynamic plugin loading via registry.go.
- Metrics endpoints on each service and Prometheus SD via prometheus_etcd_sd.
- Container boundaries and Kubernetes objects if targeting K8s.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Systems
    subgraph "Source Systems"
        MySQL[(MySQL)]:::source
        MongoDB[(MongoDB)]:::source
        TiDB_Source[(TiDB)]:::source
        PostgreSQL_Source[(PostgreSQL)]:::source
    end

    %% Input Plugins
    subgraph "Gravity Input Plugins"
        MySQLBinlog["MySQL Binlog Input"]:::plugin
        click MySQLBinlog "https://github.com/moiot/gravity/tree/master/pkg/inputs/mysqlstream/"
        MySQLBatch["MySQL Batch Input"]:::plugin
        click MySQLBatch "https://github.com/moiot/gravity/tree/master/pkg/inputs/mysqlbatch/"
        MongoOplog["MongoDB Oplog Input"]:::plugin
        click MongoOplog "https://github.com/moiot/gravity/tree/master/pkg/inputs/mongostream/"
        MongoBatch["MongoDB Batch Input"]:::plugin
        click MongoBatch "https://github.com/moiot/gravity/tree/master/pkg/inputs/mongobatch/"
        TiDBKafka["TiDB Kafka Input"]:::plugin
        click TiDBKafka "https://github.com/moiot/gravity/tree/master/pkg/inputs/tidb_kafka/"
        DumpInput["Dump Input"]:::plugin
        click DumpInput "https://github.com/moiot/gravity/blob/master/pkg/inputs/dump_input.go"
    end

    %% Core Pipeline
    subgraph "Gravity Core Pipeline"
        InputHandoff["Input Handoff"]:::core
        click InputHandoff "https://github.com/moiot/gravity/blob/master/pkg/core/input.go"
        FilterStage["Filter Stage"]:::core
        click FilterStage "https://github.com/moiot/gravity/blob/master/pkg/core/filter.go"
        MatcherStage["Matcher Stage"]:::core
        click MatcherStage "https://github.com/moiot/gravity/blob/master/pkg/core/matcher.go"
        Scheduler["Scheduler"]:::core
        click Scheduler "https://github.com/moiot/gravity/blob/master/pkg/core/scheduler.go"
        Router["Router"]:::core
        click Router "https://github.com/moiot/gravity/blob/master/pkg/core/router.go"
        Emitter["Emitter"]:::core
        click Emitter "https://github.com/moiot/gravity/blob/master/pkg/core/emitter.go"
        OutputHandoff["Output Handoff"]:::core
        click OutputHandoff "https://github.com/moiot/gravity/blob/master/pkg/core/output.go"
    end

    %% Output Plugins
    subgraph "Gravity Output Plugins"
        MySQLOut["MySQL/TiDB Writer"]:::plugin
        click MySQLOut "https://github.com/moiot/gravity/blob/master/pkg/outputs/mysql/mysql.go"
        KafkaOut["Kafka Producer"]:::plugin
        click KafkaOut "https://github.com/moiot/gravity/blob/master/pkg/outputs/async_kafka/async_kafka.go"
        ESOut["Elasticsearch Client"]:::plugin
        click ESOut "https://github.com/moiot/gravity/blob/master/pkg/outputs/elasticsearch/elasticsearch.go"
        ESModel["ES Model"]:::plugin
        click ESModel "https://github.com/moiot/gravity/blob/master/pkg/outputs/esmodel/esmodel.go"
        StdoutOut["Stdout Sink"]:::plugin
        click StdoutOut "https://github.com/moiot/gravity/blob/master/pkg/outputs/stdout/stdout.go"
    end

    %% Sink Systems
    subgraph "Sink Systems"
        MySQL_Sink[(MySQL/TiDB)]:::source
        Kafka_Sink[(Kafka)]:::source
        ES_Sink[(Elasticsearch)]:::source
    end

    %% DCP Cluster Services
    subgraph "DCP Cluster Services"
        Barrier["DCP Barrier"]:::dcp
        click Barrier "https://github.com/moiot/gravity/blob/master/dcp/barrier/barrier.go"
        Collector["DCP Collector"]:::dcp
        click Collector "https://github.com/moiot/gravity/blob/master/dcp/collector/collector.go"
        Checker["DCP Checker"]:::dcp
        click Checker "https://github.com/moiot/gravity/blob/master/dcp/checker/checker.go"
    end

    %% Auxiliary Services
    subgraph "Auxiliary Services"
        GravityMain["Gravity Main Service"]:::core
        click GravityMain "https://github.com/moiot/gravity/blob/master/cmd/gravity/main.go"
        DCP_Coordinator["DCP Coordinator Service"]:::dcp
        click DCP_Coordinator "https://github.com/moiot/gravity/blob/master/cmd/dcp/main.go"
        PadderService["Padder Service"]:::aux
        click PadderService "https://github.com/moiot/gravity/blob/master/cmd/padder/main.go"
        VerifierService["Verifier Service"]:::aux
        click VerifierService "https://github.com/moiot/gravity/blob/master/cmd/verifier/main.go"
        PromSD["Prometheus Etcd SD"]:::infra
        click PromSD "https://github.com/moiot/gravity/blob/master/cmd/prometheus_etcd_sd/main.go"
        ETCDUtil["etcd Utils"]:::infra
        click ETCDUtil "https://github.com/moiot/gravity/blob/master/pkg/utils/etcd.go"
        Registry["Plugin Registry"]:::infra
        click Registry "https://github.com/moiot/gravity/blob/master/pkg/registry/registry.go"
    end

    %% Observability
    subgraph "Observability"
        Prometheus[(Prometheus)]:::infra
        Grafana[(Grafana Dashboards)]:::infra
        click Grafana "https://github.com/moiot/gravity/blob/master/deploy/grafana/pipeline-overview.json"
    end

    %% Protocol Definitions
    subgraph "Protocol Definitions"
        ProtocolDCP["DCP Message Proto"]:::infra
        click ProtocolDCP "https://github.com/moiot/gravity/blob/master/protocol/dcp/message.proto"
        ProtocolMsg["MsgPB Proto"]:::infra
        click ProtocolMsg "https://github.com/moiot/gravity/blob/master/protocol/msgpb/message.proto"
        ProtocolTiDB["TiDB Proto"]:::infra
        click ProtocolTiDB "https://github.com/moiot/gravity/blob/master/protocol/tidb/tidb.proto"
    end

    %% Data Flows
    MySQL -->|binlog/scan| MySQLBinlog
    MySQL -->|batch scan| MySQLBatch
    MongoDB -->|oplog tail| MongoOplog
    MongoDB -->|batch scan| MongoBatch
    TiDB_Source -->|Kafka read| TiDBKafka
    PostgreSQL_Source -->|WAL stream| DumpInput

    MySQLBinlog --> InputHandoff
    MySQLBatch --> InputHandoff
    MongoOplog --> InputHandoff
    MongoBatch --> InputHandoff
    TiDBKafka --> InputHandoff
    DumpInput --> InputHandoff

    InputHandoff --> FilterStage --> MatcherStage --> Scheduler --> Router --> Emitter --> OutputHandoff

    OutputHandoff --> MySQLOut --> MySQL_Sink
    OutputHandoff --> KafkaOut --> Kafka_Sink
    OutputHandoff --> ESOut --> ES_Sink
    ESOut --> ESModel
    OutputHandoff --> StdoutOut

    %% DCP interactions
    Collector <--> GravityMain
    Barrier -->|sync via etcd| Collector
    Checker -->|validate sync| Barrier
    DCP_Coordinator --> Barrier
    DCP_Coordinator --> Collector
    DCP_Coordinator --> Checker

    %% Auxiliary flows
    PadderService -->|replay late events| InputHandoff
    VerifierService -->|compare data| MySQL_Sink
    VerifierService -->|compare data| MySQL

    GravityMain -->|load plugins| Registry
    Registry --> FilterStage
    Registry --> MySQLBinlog
    Registry --> MySQLOut

    %% Observability flows
    GravityMain -->|metrics| Prometheus
    DCP_Coordinator -->|metrics| Prometheus
    PadderService -->|metrics| Prometheus
    VerifierService -->|metrics| Prometheus
    PromSD -->|scrape etcd| etcd[(etcd)]:::infra
    PromSD --> Prometheus
    Prometheus --> Grafana

    %% Styles
    classDef source fill:#AED6F1,stroke:#21618C,color:#1B4F72
    classDef plugin fill:#ABEBC6,stroke:#1D8348,color:#145A32
    classDef core fill:#A9DFBF,stroke:#196F3D,color:#145A32
    classDef dcp fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef aux fill:#F5CBA7,stroke:#A04000,color:#641E16
    classDef infra fill:#D5D8DC,stroke:#566573,color:#2C3E50
```
