swathipalla123/cdc-synchronization-engine

Purpose and Runtime Entry

This is a Java 21/Spring Boot CDC synchronization service. It captures PostgreSQL changes through Debezium/Kafka Connect, consumes Kafka topic events, and projects synchronization state into Elasticsearch, Redis, and PostgreSQL audit/status records. The application entry point is src/main/java/com/example/cdc/CdcSynchronizationEngineApplication.java; runtime wiring and deployment are defined by src/main/resources/application.properties, Dockerfile, and docker-compose.yml.

CDC Ingestion Boundary

Debezium is the source-database-to-event boundary: PostgreSQL changes are captured and emitted into Kafka topics such as orders, products, customers, and inventory. Connector integration is represented by src/main/java/com/example/cdc/config/DebeziumConfig.java and src/main/java/com/example/cdc/kafka/debezium/DebeziumConnector.java. Kafka Connect/Debezium runs externally rather than as ordinary request processing in the application.

Kafka Event Processing

Topic-specific consumers form the main asynchronous processing stage:

src/main/java/com/example/cdc/kafka/consumer/OrderConsumer.java,

src/main/java/com/example/cdc/kafka/consumer/ProductConsumer.java, and

src/main/java/com/example/cdc/kafka/consumer/CustomerConsumer.java.

They receive CDC messages with manual acknowledgement and delegate synchronization work to the service layer. src/main/java/com/example/cdc/kafka/producer/KafkaProducerService.java supports publishing, including the exposed Kafka test/trigger endpoint.

Synchronization and Projection

src/main/java/com/example/cdc/service/SyncService.java is the central orchestration boundary for applying consumed changes. It coordinates persistence of event/status state and downstream projections. src/main/java/com/example/cdc/service/ElasticsearchService.java writes searchable synchronized data, while src/main/java/com/example/cdc/service/RedisService.java provides cache integration. Their clients/configuration are isolated in src/main/java/com/example/cdc/config/ElasticsearchConfig.java and src/main/java/com/example/cdc/config/RedisConfig.java.

Reliability, Failure, and Audit Path

Failed processing flows through retry and DLQ handling: src/main/java/com/example/cdc/kafka/retry/ExponentialRetryHandler.java, src/main/java/com/example/cdc/service/RetryService.java, and src/main/java/com/example/cdc/kafka/dlq/DeadLetterConsumer.java. Failed events are persisted via src/main/java/com/example/cdc/entity/FailedEvent.java and exposed for operations. Successful and failed work is tracked through ProcessStatus, SyncEvent, and audit logs, with auditing centralized in src/main/java/com/example/cdc/service/AuditService.java.

API, Security, and Operational Access

The synchronous API is an operational/control plane, not the CDC data path. Authentication begins at src/main/java/com/example/cdc/controller/AuthController.java; JWT and role-based authorization are enforced by src/main/java/com/example/cdc/config/SecurityConfig.java and src/main/java/com/example/cdc/security/JwtFilter.java. Status and failure inspection are served by src/main/java/com/example/cdc/controller/SyncStatusController.java and src/main/java/com/example/cdc/controller/FailedEventController.java.

Observability and Infrastructure

A correlation ID is attached at the HTTP boundary in src/main/java/com/example/cdc/tracing/CorrelationIdFilter.java. Metrics/Prometheus are configured in src/main/java/com/example/cdc/config/MicrometerConfig.java and src/main/java/com/example/cdc/config/PrometheusConfig.java; Swagger config exposes API documentation. The composed environment includes PostgreSQL, Kafka/Zookeeper, Debezium, Redis, Elasticsearch/Kibana, and the Spring Boot application.