# ehsanfaridi92/idempotencyusingkafkaandmongo architecture

> Architecture diagram of the GitHub repository ehsanfaridi92/idempotencyusingkafkaandmongo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ehsanfaridi92/idempotencyusingkafkaandmongo
- Repository: https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo

## Overview

##### Main Components
- Producer Component:
- Represented by the KafkaProducer project. Highlight key classes such as KafkaProducerWorker and KafkaProducerConfiguration.
- How it sends serialized JSON messages (with unique EventIds) to Kafka.
- Consumer Component:
- Represented by the KafkaConsumer project including classes like KafkaConsumerWorker, KafkaIdempotentConsumer, and the IDuplicateMessageHandler interface implemented by MongoDuplicateMessageHandler.
- How this component reads messages from Kafka and processes them after checking for duplicates in MongoDB.
- External Services:
- Kafka: Used as the messaging system to transport messages between the producer and consumer.
- MongoDB: Used as the datastore to record the processing state and enforce idempotency.

##### Relationships and Interactions
- That the check is done using the DuplicateMessageHandler logic to prevent reprocessing.

##### Architectural Patterns and Design Principles
- Use of the Idempotency pattern which is crucial in event-driven scenarios.
- Separation of concerns between message production and consumption.
- Mention that configuration settings (for Kafka and MongoDB) are externally provided via appsettings.json files and specialized configuration classes.
- If desired, note that the project demonstrates a clear microservice-ish pattern where each component (producer and consumer) works independently yet integrates through Kafka.

##### Relevant Technologies
- .NET 8.0 (for running the C# applications)
- Confluent.Kafka (for Kafka client integrations)
- MongoDB.Driver (for interfacing with MongoDB)
- Additional libraries such as Newtonsoft.Json for message serialization
- Optionally, include configuration management which is handled through the appsettings and dedicated configuration classes

##### Additional Details
- Under “Producer”, include a side note that messages contain a unique identifier (EventId) added to Kafka message headers.
- Under “Consumer”, note that processed messages are validated through IDuplicateMessageHandler and persisted in MongoDB to prevent duplicate processing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Producer Component
    subgraph "Producer Component"
        direction TB
        prodWorker["KafkaProducerWorker"]:::producer
        prodConfig["KafkaProducerConfiguration"]:::producer
        prodService["KafkaProducer Service & IKafkaProducer"]:::producer
        prodSettings["Producer appsettings.json"]:::config
        prodWorker -->|"uses"| prodConfig
        prodService -->|"invokes"| prodWorker
        prodConfig -->|"configured via"| prodSettings
    end

    %% External Kafka
    subgraph "Kafka"
        direction TB
        kafka["Kafka Broker"]:::broker
    end

    %% Consumer Component
    subgraph "Consumer Component"
        direction TB
        consWorker["KafkaConsumerWorker"]:::consumer
        consIdempotent["KafkaIdempotentConsumer"]:::consumer
        dupInterface["IDuplicateMessageHandler"]:::consumer
        mongoDuplicate["MongoDuplicateMessageHandler"]:::consumer
        subgraph "Consumer Configurations"
            direction TB
            consKafkaConfig["KafkaConsumerConfiguration"]:::config
            consMongoConfig["MongoDbConfiguration"]:::config
            consSettings["Consumer appsettings.json"]:::config
        end
        consWorker -->|"processes & invokes"| consIdempotent
        consIdempotent -->|"performs idempotency-check via"| dupInterface
        dupInterface -->|"implemented-by"| mongoDuplicate
        consWorker -->|"loads config from"| consKafkaConfig
        consWorker -->|"loads config from"| consMongoConfig
        consKafkaConfig -->|"loaded from"| consSettings
        consMongoConfig -->|"loaded from"| consSettings
    end

    %% External MongoDB
    subgraph "MongoDB"
        direction TB
        mongo["MongoDB Database"]:::database
    end

    %% Data Flow Between Components
    prodWorker -->|"sends JSON with EventId"| kafka
    kafka -->|"delivers message"| consWorker
    consIdempotent -->|"queries/writes state"| mongo
    dupInterface -->|"checks duplicates in"| mongo

    %% Styles
    classDef producer fill:#CDEFFF,stroke:#333,stroke-width:2px;
    classDef consumer fill:#FFE6CC,stroke:#333,stroke-width:2px;
    classDef config fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef broker fill:#FDEBD0,stroke:#333,stroke-width:2px;
    classDef database fill:#F9EBEA,stroke:#333,stroke-width:2px;

    %% Click Events for Producer Components
    click prodWorker "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaProducer/KafkaProducerWorker.cs"
    click prodConfig "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaProducer/ConfigurationModels/KafkaProducerConfiguration.cs"
    click prodService "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaProducer/Services/KafkaProducer.cs"
    click prodService "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaProducer/Services/IKafkaProducer.cs"
    click prodSettings "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaProducer/appsettings.json"

    %% Click Events for Consumer Components
    click consWorker "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/KafkaConsumerWorker.cs"
    click consIdempotent "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/Services/KafkaIdempotentConsumer.cs"
    click dupInterface "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/IdempotencyServices/IDuplicateMessageHandler.cs"
    click mongoDuplicate "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/IdempotencyServices/MongoDuplicateMessageHandler.cs"
    click consKafkaConfig "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/ConfigurationModels/KafkaConsumerConfiguration.cs"
    click consMongoConfig "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/ConfigurationModels/MongoDbConfiguration.cs"
    click consSettings "https://github.com/ehsanfaridi92/idempotencyusingkafkaandmongo/blob/master/Code/src/KafkaConsumer/appsettings.json"
```
