How bersler/openlogreplicator works
1. What this repository is
- OpenLogReplicator is a C++ Oracle CDC engine. Its core job is to take Oracle redo-log changes, reconstruct transactions, and emit them as structured change events.
- The main architectural boundary is: Oracle redo logs in / formatted change events out.
2. Top-level runtime
- The executable entrypoint is src/main.cpp, which drives OpenLogReplicator and the Replicator layer.
- Build and packaging are handled with CMake; runtime configuration is JSON-based, with example configs in scripts/.
3. Input acquisition
- The input side is split between a generic Reader abstraction and concrete implementations like ReaderFilesystem.
- The project supports both offline/batch log reading and online replication from an Oracle instance, which is reflected in ReplicatorBatch and ReplicatorOnline.
4. Oracle integration boundary
- The replicator and Database* classes are the Oracle-facing control plane: they manage database connectivity, environment setup, and statements needed for metadata or replication startup.
- This is the part that bridges from the external Oracle system into the log-processing pipeline.
5. Redo-log parsing pipeline
- The parser subsystem is the low-level binary decoder for Oracle redo records. The opcode-specific headers show that parsing is organized by redo record type.
- Parsed records become transactions via Transaction and TransactionBuffer, which is where individual redo operations are grouped and ordered into commit-aware units.
6. Metadata and schema reconstruction
- The metadata subsystem tracks checkpoints, schema state, and serialization of that state.
- Classes like Schema, Checkpoint, and SerializerJson indicate that the replicator maintains an internal view of tables, objects, and replication progress so it can interpret redo data correctly over time.
7. Type system, tables, and Oracle-specific data handling
- src/common contains the shared domain model: SCN/XID/ROWID/Lob identifiers, redo-log record abstractions, table metadata models, and generic data wrappers.
- The table and types subfolders show that Oracle object metadata and physical identifiers are first-class concepts, not just parsed payloads.
8. Character sets and LOB support
- The locales and DbLob/Lob* classes indicate explicit handling for Oracle character sets and large objects.
- This is important architecturally because redo-log data is not just bytes; it must be decoded into correct text and LOB payloads before formatting.
9. Event building and output formatting
- The builder subsystem converts reconstructed database changes into output representations.
- There are two concrete serialization paths: JSON and Protobuf. The generated OraProtoBuf types and BuilderProtobuf show a structured binary schema option alongside the JSON path.
10. Targets and transport
- The writer layer is the sink abstraction. Concrete writers include file output, discard/testing, Kafka, and streaming variants.
- The stream subsystem provides transport implementations for network delivery, including plain TCP/IP and ZeroMQ. So the sink boundary is not just “write to disk” but “deliver to external consumers over multiple transports.”
11. Checkpointing, state, and recovery
- state persists runtime progress, while metadata/Checkpoint and StateDisk show that restartability is built into the design.
- This is the mechanism that lets the replicator resume from a known SCN/log position rather than replaying from scratch.
12. Metrics, logging, and operational visibility
- The metrics subsystem, including Prometheus support, provides runtime observability.
- The documentation tree also highlights detailed log-message categories and troubleshooting guidance, suggesting the system is designed for operational debugging of parsing, configuration, and redo-log issues.
13. Primary data flow
- The normal pipeline is: Oracle redo logs or Oracle-connected source → Reader/Replicator → Parser/Transaction buffering → Metadata/schema interpretation → Builder serialization → Writer/Stream/Kafka/File target.
- Checkpoint/state updates run alongside that flow so the next process start can continue from the last durable position.
14. Architectural constraints
- This is a correctness-sensitive, low-level systems application rather than a general web service.
- The key boundaries are between: Oracle input acquisition, binary redo parsing, semantic reconstruction of transactions, serialization format choice, and delivery transport/target.
Main entry points: Metadata.cpp Metadata · Parser.cpp Parser · Replicator.cpp Replicator · ReplicatorBatch.cpp Batch/Online
Components 22
Runtime
Process entrypoint and replication orchestration
- Main · Program startup and CLI/config bootstrap
src/main.cpp - Core Replicator · Top-level replication control and lifecycle
src/OpenLogReplicator.cpp - Replicator · Coordinates batch and online replication modes
src/replicator/Replicator.cpp - Batch/Online · Offline log replay and live Oracle-backed replication
src/replicator/ReplicatorBatch.cpp
Ingest
Reading Oracle redo input and connecting to Oracle
- Reader · Generic source interface for redo input
src/reader/Reader.cpp - File Reader · Reads redo logs from local files
src/reader/ReaderFilesystem.cpp - Oracle Access · Database connectivity, environment, and statements
src/replicator/DatabaseConnection.cpp
Parse
Redo decoding and transaction reconstruction
- Parser · Binary redo record decoding and opcode dispatch
src/parser/Parser.cpp - OpCodes · Opcode-specific redo record formats
src/parser/OpCode.h - Transactions · Groups redo operations into transaction units
src/parser/Transaction.cpp - Tx Buffer · Buffers and orders transaction fragments
src/parser/TransactionBuffer.cpp
Metadata
Schema, checkpoint, and runtime state
- Metadata · Schema, checkpoints, and redo metadata model
src/metadata/Metadata.cpp - Schema · Table and object reconstruction state
src/metadata/Schema.cpp - Checkpoint · Durable replication position and progress tracking
src/metadata/Checkpoint.cpp
Emit
Event building and delivery targets
- Builder · Converts reconstructed changes into output events
src/builder/Builder.cpp - JSON/Proto · JSON and Protobuf output formats
src/builder/BuilderJson.cpp - Writer · Output sink interface for delivered events
src/writer/Writer.cpp - Targets · File, Kafka, discard, and stream writers
src/writer/WriterFile.cpp - Streams · TCP/IP and ZeroMQ delivery transports
src/stream/Stream.cpp
Support
Shared domain types, locales, observability, and docs
- Common Model · SCN/XID/ROWID, LOBs, table metadata, and shared utilities
src/common/types/Types.h - Charsets · Oracle character set decoding and locale handling
src/locales/Locales.cpp - Metrics · Metrics collection and Prometheus export
src/common/metrics/Metrics.cpp
Connections 23
- Main → Core Replicator: starts · Program entry initializes the core replicator
- Core Replicator → Replicator: drives · Top-level runtime delegates work to replication control
- Replicator → Batch/Online: selects mode · Chooses offline batch or online Oracle replication
- Batch/Online → Reader: uses · Batch mode consumes redo via the reader abstraction
- Reader → File Reader: implements · Filesystem-backed concrete reader
- Batch/Online → Oracle Access: uses · Online mode talks to Oracle control-plane APIs
- Reader → Parser: feeds · Redo input is decoded by the parser
- Oracle Access → Metadata: seeds · Oracle metadata and startup state populate internal schema model
- Parser → OpCodes: dispatches · Opcode-specific decoding handles redo record variants
- Parser → Transactions: builds · Parsed operations are assembled into transactions
- Transactions → Tx Buffer: buffers · Transaction fragments are ordered and held for commit awareness
- Tx Buffer → Metadata: updates · Transaction progress interacts with schema/checkpoint state
- Metadata → Schema: contains · Schema reconstruction is part of metadata state
- Metadata → Checkpoint: persists · Checkpoint state is tracked for restartability
- Transactions → Builder: passes changes · Committed transactions flow into event construction
- Builder → JSON/Proto: serializes · Builder emits either JSON or Protobuf payloads
- JSON/Proto → Writer: hands off · Serialized events are delivered to sink layer
- Writer → Targets: routes to · Concrete sinks include file, Kafka, discard, and stream writers
- Targets → Streams: may use · Streaming sinks depend on network transports
- Builder → Common Model: uses · Event formatting relies on shared Oracle domain types
- Parser → Charsets: decodes · Redo payloads require character set handling
- Replicator → Metrics: reports · Runtime operations feed observability metrics
- Metadata → Common Model: shares model · Metadata uses the same core identifiers and table abstractions