# bersler/openlogreplicator architecture

> Architecture diagram of the GitHub repository bersler/openlogreplicator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bersler/openlogreplicator
- Repository: https://github.com/bersler/openlogreplicator

## Overview

1. **What this repository is**
   - OpenLogReplicator is a C++ Oracle CDC engine. Its core job is to take Oracle redo-log changes, reconstruct transactions, and emit them as structured change events.
   - The main architectural boundary is: **Oracle redo logs in / formatted change events out**.

2. **Top-level runtime**
   - The executable entrypoint is `src/main.cpp`, which drives `OpenLogReplicator` and the `Replicator` layer.
   - Build and packaging are handled with CMake; runtime configuration is JSON-based, with example configs in `scripts/`.

3. **Input acquisition**
   - The input side is split between a generic `Reader` abstraction and concrete implementations like `ReaderFilesystem`.
   - The project supports both **offline/batch log reading** and **online replication** from an Oracle instance, which is reflected in `ReplicatorBatch` and `ReplicatorOnline`.

4. **Oracle integration boundary**
   - The `replicator` and `Database*` classes are the Oracle-facing control plane: they manage database connectivity, environment setup, and statements needed for metadata or replication startup.
   - This is the part that bridges from the external Oracle system into the log-processing pipeline.

5. **Redo-log parsing pipeline**
   - The `parser` subsystem is the low-level binary decoder for Oracle redo records. The opcode-specific headers show that parsing is organized by redo record type.
   - Parsed records become transactions via `Transaction` and `TransactionBuffer`, which is where individual redo operations are grouped and ordered into commit-aware units.

6. **Metadata and schema reconstruction**
   - The `metadata` subsystem tracks checkpoints, schema state, and serialization of that state.
   - Classes like `Schema`, `Checkpoint`, and `SerializerJson` indicate that the replicator maintains an internal view of tables, objects, and replication progress so it can interpret redo data correctly over time.

7. **Type system, tables, and Oracle-specific data handling**
   - `src/common` contains the shared domain model: SCN/XID/ROWID/Lob identifiers, redo-log record abstractions, table metadata models, and generic data wrappers.
   - The `table` and `types` subfolders show that Oracle object metadata and physical identifiers are first-class concepts, not just parsed payloads.

8. **Character sets and LOB support**
   - The `locales` and `DbLob`/`Lob*` classes indicate explicit handling for Oracle character sets and large objects.
   - This is important architecturally because redo-log data is not just bytes; it must be decoded into correct text and LOB payloads before formatting.

9. **Event building and output formatting**
   - The `builder` subsystem converts reconstructed database changes into output representations.
   - There are two concrete serialization paths: **JSON** and **Protobuf**. The generated `OraProtoBuf` types and `BuilderProtobuf` show a structured binary schema option alongside the JSON path.

10. **Targets and transport**
   - The `writer` layer is the sink abstraction. Concrete writers include file output, discard/testing, Kafka, and streaming variants.
   - The `stream` subsystem provides transport implementations for network delivery, including plain TCP/IP and ZeroMQ. So the sink boundary is not just “write to disk” but “deliver to external consumers over multiple transports.”

11. **Checkpointing, state, and recovery**
   - `state` persists runtime progress, while `metadata/Checkpoint` and `StateDisk` show that restartability is built into the design.
   - This is the mechanism that lets the replicator resume from a known SCN/log position rather than replaying from scratch.

12. **Metrics, logging, and operational visibility**
   - The `metrics` subsystem, including Prometheus support, provides runtime observability.
   - The documentation tree also highlights detailed log-message categories and troubleshooting guidance, suggesting the system is designed for operational debugging of parsing, configuration, and redo-log issues.

13. **Primary data flow**
   - The normal pipeline is: **Oracle redo logs or Oracle-connected source → Reader/Replicator → Parser/Transaction buffering → Metadata/schema interpretation → Builder serialization → Writer/Stream/Kafka/File target**.
   - Checkpoint/state updates run alongside that flow so the next process start can continue from the last durable position.

14. **Architectural constraints**
   - This is a correctness-sensitive, low-level systems application rather than a general web service.
   - The key boundaries are between: Oracle input acquisition, binary redo parsing, semantic reconstruction of transactions, serialization format choice, and delivery transport/target.

Main entry points: [`src/metadata/Metadata.cpp`](https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Metadata.cpp) (Metadata), [`src/parser/Parser.cpp`](https://github.com/bersler/openlogreplicator/blob/master/src/parser/Parser.cpp) (Parser), [`src/replicator/Replicator.cpp`](https://github.com/bersler/openlogreplicator/blob/master/src/replicator/Replicator.cpp) (Replicator), [`src/replicator/ReplicatorBatch.cpp`](https://github.com/bersler/openlogreplicator/blob/master/src/replicator/ReplicatorBatch.cpp) (Batch/Online)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_runtime["Runtime"]
  node_n_main["Main<br/>entrypoint<br/>[main.cpp]"]
  node_n_orl["Core Replicator<br/>orchestrator"]
  node_n_repl["Replicator<br/>coordination<br/>[Replicator.cpp]"]
  node_n_batch_online["Batch/Online<br/>modes"]
end

subgraph group_grp_ingest["Ingest"]
  node_n_reader["Reader<br/>input abstraction<br/>[Reader.cpp]"]
  node_n_fs_reader["File Reader<br/>filesystem input"]
  node_n_db["Oracle Access<br/>db control plane"]
end

subgraph group_grp_parse["Parse"]
  node_n_parser["Parser<br/>redo decoder<br/>[Parser.cpp]"]
  node_n_opcodes["OpCodes<br/>record handlers<br/>[OpCode.h]"]
  node_n_tx["Transactions<br/>commit assembly<br/>[Transaction.cpp]"]
  node_n_txbuf["Tx Buffer<br/>ordering buffer"]
end

subgraph group_grp_meta["Metadata"]
  node_n_meta["Metadata<br/>schema state<br/>[Metadata.cpp]"]
  node_n_schema["Schema<br/>catalog model<br/>[Schema.cpp]"]
  node_n_ckpt["Checkpoint<br/>progress state<br/>[Checkpoint.cpp]"]
end

subgraph group_grp_emit["Emit"]
  node_n_builder["Builder<br/>event formatter<br/>[Builder.cpp]"]
  node_n_json_pb["JSON/Proto<br/>serialization<br/>[BuilderJson.cpp]"]
  node_n_writer["Writer<br/>sink abstraction<br/>[Writer.cpp]"]
  node_n_targets["Targets<br/>delivery sinks<br/>[WriterFile.cpp]"]
  node_n_streams["Streams<br/>transports<br/>[Stream.cpp]"]
end

subgraph group_grp_support["Support"]
  node_n_common["Common Model<br/>domain types<br/>[Types.h]"]
  node_n_locales["Charsets<br/>encoding support<br/>[Locales.cpp]"]
  node_n_metrics["Metrics<br/>observability<br/>[Metrics.cpp]"]
end

node_n_main -->|"starts"| node_n_orl
node_n_orl -->|"drives"| node_n_repl
node_n_repl -->|"selects mode"| node_n_batch_online
node_n_batch_online -->|"uses"| node_n_reader
node_n_reader -->|"implements"| node_n_fs_reader
node_n_batch_online -->|"uses"| node_n_db
node_n_reader -->|"feeds"| node_n_parser
node_n_db -->|"seeds"| node_n_meta
node_n_parser -->|"dispatches"| node_n_opcodes
node_n_parser -->|"builds"| node_n_tx
node_n_tx -->|"buffers"| node_n_txbuf
node_n_txbuf -->|"updates"| node_n_meta
node_n_meta -->|"contains"| node_n_schema
node_n_meta -->|"persists"| node_n_ckpt
node_n_tx -->|"passes changes"| node_n_builder
node_n_builder -->|"serializes"| node_n_json_pb
node_n_json_pb -->|"hands off"| node_n_writer
node_n_writer -->|"routes to"| node_n_targets
node_n_targets -->|"may use"| node_n_streams
node_n_builder -->|"uses"| node_n_common
node_n_parser -->|"decodes"| node_n_locales
node_n_repl -->|"reports"| node_n_metrics
node_n_meta -->|"shares model"| node_n_common

click node_n_main "https://github.com/bersler/openlogreplicator/blob/master/src/main.cpp"
click node_n_orl "https://github.com/bersler/openlogreplicator/blob/master/src/OpenLogReplicator.cpp"
click node_n_repl "https://github.com/bersler/openlogreplicator/blob/master/src/replicator/Replicator.cpp"
click node_n_batch_online "https://github.com/bersler/openlogreplicator/blob/master/src/replicator/ReplicatorBatch.cpp"
click node_n_reader "https://github.com/bersler/openlogreplicator/blob/master/src/reader/Reader.cpp"
click node_n_fs_reader "https://github.com/bersler/openlogreplicator/blob/master/src/reader/ReaderFilesystem.cpp"
click node_n_db "https://github.com/bersler/openlogreplicator/blob/master/src/replicator/DatabaseConnection.cpp"
click node_n_parser "https://github.com/bersler/openlogreplicator/blob/master/src/parser/Parser.cpp"
click node_n_opcodes "https://github.com/bersler/openlogreplicator/blob/master/src/parser/OpCode.h"
click node_n_tx "https://github.com/bersler/openlogreplicator/blob/master/src/parser/Transaction.cpp"
click node_n_txbuf "https://github.com/bersler/openlogreplicator/blob/master/src/parser/TransactionBuffer.cpp"
click node_n_meta "https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Metadata.cpp"
click node_n_schema "https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Schema.cpp"
click node_n_ckpt "https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Checkpoint.cpp"
click node_n_builder "https://github.com/bersler/openlogreplicator/blob/master/src/builder/Builder.cpp"
click node_n_json_pb "https://github.com/bersler/openlogreplicator/blob/master/src/builder/BuilderJson.cpp"
click node_n_writer "https://github.com/bersler/openlogreplicator/blob/master/src/writer/Writer.cpp"
click node_n_targets "https://github.com/bersler/openlogreplicator/blob/master/src/writer/WriterFile.cpp"
click node_n_streams "https://github.com/bersler/openlogreplicator/blob/master/src/stream/Stream.cpp"
click node_n_common "https://github.com/bersler/openlogreplicator/blob/master/src/common/types/Types.h"
click node_n_locales "https://github.com/bersler/openlogreplicator/blob/master/src/locales/Locales.cpp"
click node_n_metrics "https://github.com/bersler/openlogreplicator/blob/master/src/common/metrics/Metrics.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_main,node_n_orl,node_n_repl,node_n_batch_online toneBlue
class node_n_reader,node_n_fs_reader,node_n_db toneAmber
class node_n_parser,node_n_opcodes,node_n_tx,node_n_txbuf toneMint
class node_n_meta,node_n_schema,node_n_ckpt toneRose
class node_n_builder,node_n_json_pb,node_n_writer,node_n_targets,node_n_streams toneIndigo
class node_n_common,node_n_locales,node_n_metrics toneTeal
```

## Components

### Runtime

Process entrypoint and replication orchestration

- [Main](https://github.com/bersler/openlogreplicator/blob/master/src/main.cpp): entrypoint · Program startup and CLI/config bootstrap · `src/main.cpp`
- [Core Replicator](https://github.com/bersler/openlogreplicator/blob/master/src/OpenLogReplicator.cpp): orchestrator · Top-level replication control and lifecycle · `src/OpenLogReplicator.cpp`
- [Replicator](https://github.com/bersler/openlogreplicator/blob/master/src/replicator/Replicator.cpp): coordination · Coordinates batch and online replication modes · `src/replicator/Replicator.cpp`
- [Batch/Online](https://github.com/bersler/openlogreplicator/blob/master/src/replicator/ReplicatorBatch.cpp): modes · Offline log replay and live Oracle-backed replication · `src/replicator/ReplicatorBatch.cpp`

### Ingest

Reading Oracle redo input and connecting to Oracle

- [Reader](https://github.com/bersler/openlogreplicator/blob/master/src/reader/Reader.cpp): input abstraction · Generic source interface for redo input · `src/reader/Reader.cpp`
- [File Reader](https://github.com/bersler/openlogreplicator/blob/master/src/reader/ReaderFilesystem.cpp): filesystem input · Reads redo logs from local files · `src/reader/ReaderFilesystem.cpp`
- [Oracle Access](https://github.com/bersler/openlogreplicator/blob/master/src/replicator/DatabaseConnection.cpp): db control plane · Database connectivity, environment, and statements · `src/replicator/DatabaseConnection.cpp`

### Parse

Redo decoding and transaction reconstruction

- [Parser](https://github.com/bersler/openlogreplicator/blob/master/src/parser/Parser.cpp): redo decoder · Binary redo record decoding and opcode dispatch · `src/parser/Parser.cpp`
- [OpCodes](https://github.com/bersler/openlogreplicator/blob/master/src/parser/OpCode.h): record handlers · Opcode-specific redo record formats · `src/parser/OpCode.h`
- [Transactions](https://github.com/bersler/openlogreplicator/blob/master/src/parser/Transaction.cpp): commit assembly · Groups redo operations into transaction units · `src/parser/Transaction.cpp`
- [Tx Buffer](https://github.com/bersler/openlogreplicator/blob/master/src/parser/TransactionBuffer.cpp): ordering buffer · Buffers and orders transaction fragments · `src/parser/TransactionBuffer.cpp`

### Metadata

Schema, checkpoint, and runtime state

- [Metadata](https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Metadata.cpp): schema state · Schema, checkpoints, and redo metadata model · `src/metadata/Metadata.cpp`
- [Schema](https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Schema.cpp): catalog model · Table and object reconstruction state · `src/metadata/Schema.cpp`
- [Checkpoint](https://github.com/bersler/openlogreplicator/blob/master/src/metadata/Checkpoint.cpp): progress state · Durable replication position and progress tracking · `src/metadata/Checkpoint.cpp`

### Emit

Event building and delivery targets

- [Builder](https://github.com/bersler/openlogreplicator/blob/master/src/builder/Builder.cpp): event formatter · Converts reconstructed changes into output events · `src/builder/Builder.cpp`
- [JSON/Proto](https://github.com/bersler/openlogreplicator/blob/master/src/builder/BuilderJson.cpp): serialization · JSON and Protobuf output formats · `src/builder/BuilderJson.cpp`
- [Writer](https://github.com/bersler/openlogreplicator/blob/master/src/writer/Writer.cpp): sink abstraction · Output sink interface for delivered events · `src/writer/Writer.cpp`
- [Targets](https://github.com/bersler/openlogreplicator/blob/master/src/writer/WriterFile.cpp): delivery sinks · File, Kafka, discard, and stream writers · `src/writer/WriterFile.cpp`
- [Streams](https://github.com/bersler/openlogreplicator/blob/master/src/stream/Stream.cpp): transports · TCP/IP and ZeroMQ delivery transports · `src/stream/Stream.cpp`

### Support

Shared domain types, locales, observability, and docs

- [Common Model](https://github.com/bersler/openlogreplicator/blob/master/src/common/types/Types.h): domain types · SCN/XID/ROWID, LOBs, table metadata, and shared utilities · `src/common/types/Types.h`
- [Charsets](https://github.com/bersler/openlogreplicator/blob/master/src/locales/Locales.cpp): encoding support · Oracle character set decoding and locale handling · `src/locales/Locales.cpp`
- [Metrics](https://github.com/bersler/openlogreplicator/blob/master/src/common/metrics/Metrics.cpp): observability · Metrics collection and Prometheus export · `src/common/metrics/Metrics.cpp`

## Connections

- Main → Core Replicator: starts · Program entry initializes the core replicator
- Core Replicator → Replicator: drives · Top-level runtime delegates work to replication control
- Replicator → Batch/Online: selects mode · Chooses offline batch or online Oracle replication
- Batch/Online → Reader: uses · Batch mode consumes redo via the reader abstraction
- Reader → File Reader: implements · Filesystem-backed concrete reader
- Batch/Online → Oracle Access: uses · Online mode talks to Oracle control-plane APIs
- Reader → Parser: feeds · Redo input is decoded by the parser
- Oracle Access → Metadata: seeds · Oracle metadata and startup state populate internal schema model
- Parser → OpCodes: dispatches · Opcode-specific decoding handles redo record variants
- Parser → Transactions: builds · Parsed operations are assembled into transactions
- Transactions → Tx Buffer: buffers · Transaction fragments are ordered and held for commit awareness
- Tx Buffer → Metadata: updates · Transaction progress interacts with schema/checkpoint state
- Metadata → Schema: contains · Schema reconstruction is part of metadata state
- Metadata → Checkpoint: persists · Checkpoint state is tracked for restartability
- Transactions → Builder: passes changes · Committed transactions flow into event construction
- Builder → JSON/Proto: serializes · Builder emits either JSON or Protobuf payloads
- JSON/Proto → Writer: hands off · Serialized events are delivered to sink layer
- Writer → Targets: routes to · Concrete sinks include file, Kafka, discard, and stream writers
- Targets → Streams: may use · Streaming sinks depend on network transports
- Builder → Common Model: uses · Event formatting relies on shared Oracle domain types
- Parser → Charsets: decodes · Redo payloads require character set handling
- Replicator → Metrics: reports · Runtime operations feed observability metrics
- Metadata → Common Model: shares model · Metadata uses the same core identifiers and table abstractions
