How moiot/gravity works
Project type and purpose
• Gravity is an open‐source, Go‑based data‑replication tool (Change Data Capture + batch).
• It captures from various sources (MySQL binlog/scan, MongoDB oplog/scan, TiDB, PostgreSQL WAL) and writes to various sinks (MySQL/TiDB, Kafka, Elasticsearch).
• It can run in a Kubernetes cluster, uses etcd for coordination and Prometheus/Grafana for observability.
File structure
• cmd/: CLI entrypoints (gravity server, dcp coordinator, padder, prometheus_etcd_sd, verifier).
• pkg/: core libraries organized by domain – inputs, outputs, filters, matchers, scheduler, emitter, protocol, registry, metrics, utils, etc.
• dcp/: distributed‑coordination primitives (barrier, checker, collector) for multi‑instance synchronization.
• deploy/: Kubernetes/Docker Compose manifests, Grafana dashboards.
• integration_test/, mock/: end‑to‑end and unit mocks.
• protocol/: Protobuf definitions for interprocess communication.
README for additional insights
• The “Architecture” doc (docs/2.0/00‑arch‑en.md) describes a modular pipeline: Input → Filter → Matcher → Scheduler → Emitter → Output.
• Mentions distributed cluster capabilities and customization via plugins.
• Source Systems (MySQL, MongoDB, TiDB, PostgreSQL)
• Gravity Input Plugins (binlog tailer, oplog tailer, batch scan)
• Gravity Core Pipeline
– Filters & Matchers
– Scheduler
– Router
– Emitter
• Gravity Output Plugins (MySQL writer, Kafka producer, Elasticsearch client)
• DCP Cluster Services (Barrier, Collector, Checker)
• Padder Service (job processor for late events)
• Verifier Service (consistency check)
• Service Discovery (etcd + prometheus_etcd_sd)
• Observability (Prometheus metrics, Grafana dashboards)
• Registry (plugin registry for dynamic filters/outputs)
1. Source Systems → Input Plugins (connect via MySQL/Mongo wire protocols or SDK)
2. Input Plugins → Core Pipeline (hand off event stream)
3. Core → Filters/Matchers → Scheduler (event ordering) → Emitter
4. Emitter → Output Plugins → Sink Systems (DBs, Kafka, ES)
5. DCP Collector ↔ Gravity instances (gRPC) for offset collection
6. DCP Barrier & Checker synchronize instance states via etcd
7. Padder jobs pull missing transactions and re‑emit into core
8. Verifier reads from sink and source to detect drift
9. Prometheus scraping endpoints on each service; Grafana dashboards render metrics
c. Architectural patterns & design principles
• Plugin architecture: inputs, filters, outputs loaded via registry.
• Modular pipeline: decoupled stages, pluggable at each stage.
• Distributed coordination: DCP using barrier/collector/checker patterns.
• Service discovery: etcd + Prometheus SD for auto‑scaling and cluster management.
• Containerized microservices: each cmd/ service runs in its own container.
d. Key technologies & frameworks
• Go modules (pkg/*)
• gRPC (dcp collector, filters/grpc sidecar)
• Protobuf (protocol/message.proto)
• etcd (coordination & config v3)
• Prometheus & Grafana (metrics)
• Docker/Kubernetes (deployment)
• Makefile & Dockerfiles (build pipelines)
• Configurations loaded from TOML via pkg/config.
• Dynamic plugin loading via registry.go.
• Metrics endpoints on each service and Prometheus SD via prometheus_etcd_sd.
• Container boundaries and Kubernetes objects if targeting K8s.