# yadavan88/data-plumber architecture

> Architecture diagram of the GitHub repository yadavan88/data-plumber, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yadavan88/data-plumber
- Repository: https://github.com/yadavan88/data-plumber

## Overview

##### Main Components
- Data Sources: There are dedicated source implementations (e.g., CsvSource.scala, MongoSource.scala, PostgresSource.scala) split across different integration patterns (simple, effectful, offsetable, and streaming in some cases). Represent these as separate “Source” modules.
- Data Sinks: Similarly, there are matching sink implementations (e.g., CSVSink.scala, MongoSink.scala, and their offsetable counterparts) corresponding to each integration type.
- Transformation/Integration Logic: At the heart of every pattern is a “DataPlumber” (or its variants) that orchestrates the flow—pulling data from a source, optionally transforming it, and then passing it to the sink.
- External Dependencies: The README and the docker-compose.yml indicate external services (MongoDB, PostgreSQL, Redis) are required. Represent each of these as external systems that interact with the application.
- Utility/Support Components: Notice additional utilities such as the RedisClient (in the offsetable package), which supports the offset tracking mechanism.

##### Establish Relationships and Data Flow
- For “Simple” integration, show a straightforward, synchronous data flow (source → transform → sink) with basic error handling.
- For “Effectful” integration, illustrate that the process is wrapped in an effect (using cats-effect IO). This pattern will include resource management and enhanced error handling.
- For “Streaming” integration, if applicable, indicate a continuous data flow with real-time source/sink modules.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The design cleanly segregates sources, sinks, and transformation logic.
- Type Safety: Emphasize that the architecture leverages Scala’s type system (e.g., via generic implementations in DataPlumber).
- Extensibility: Showcase how a developer can extend source/sink modules to integrate with additional data stores.
- Effect Handling: Differentiate between synchronous and effectful (asynchronous/resource-safe) data flows.

##### Relevant Technologies
- Scala is the core language.
- cats-effect IO for the “effectful” integration.
- External systems: MongoDB, PostgreSQL, and Redis, which are managed in Docker using docker-compose.
- Optionally, note any build tools and formatters (build.sbt, scalafmt).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Simple Integration Pipeline"
        S_DS["Simple Data Source"]:::simple
        S_PL["Simple Data Plumber"]:::simple
        S_DSink["Simple Data Sink"]:::simple
        S_DS -->|"flow"| S_PL
        S_PL -->|"flow"| S_DSink
    end

    subgraph "Effectful Integration Pipeline"
        EF_DS["Effectful Data Source"]:::effectful
        EF_PL["IODataPlumber"]:::effectful
        EF_DSink["Effectful Data Sink"]:::effectful
        EF_DS -->|"flow"| EF_PL
        EF_PL -->|"flow"| EF_DSink
    end

    subgraph "Offsetable Integration Pipeline"
        OF_DS["Offsetable Data Source"]:::offsetable
        OF_PL["Offsetable Data Plumber"]:::offsetable
        OF_DSink["Offsetable Data Sink"]:::offsetable
        OF_RC["Redis Client"]:::offsetable
        OF_DS -->|"flow"| OF_PL
        OF_PL -->|"flow"| OF_DSink
        OF_PL -->|"offset_tracking"| OF_RC
    end

    subgraph "Streaming Integration Pipeline"
        ST_DS["Streaming Data Source"]:::streaming
        ST_PL["Streaming Data Plumber"]:::streaming
        ST_DSink["Streaming Data Sink"]:::streaming
        ST_DS -->|"flow"| ST_PL
        ST_PL -->|"flow"| ST_DSink
    end

    subgraph "External Systems"
        MONGO["MongoDB"]:::external
        PSQL["PostgreSQL"]:::external
        REDIS["Redis DB"]:::external
    end

    EX["Example Implementations"]

    %% External Dotted Connections
    S_DS -.->|"uses"| MONGO
    S_DS -.->|"uses"| PSQL
    EF_DS -.->|"uses"| MONGO
    EF_DS -.->|"uses"| PSQL
    OF_DS -.->|"uses"| MONGO
    OF_DS -.->|"uses"| PSQL
    ST_DSink -.->|"outputs_to"| MONGO
    ST_DS -.->|"ingests_from"| PSQL
    OF_RC -->|"tracks_offset"| REDIS

    %% Click Events for Component Mapping
    click S_DS "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/simple/source/"
    click S_DSink "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/simple/sink/"
    click S_PL "https://github.com/yadavan88/data-plumber/blob/main/src/main/scala/com/yadavan88/dataplumber/simple/DataPlumber.scala"

    click EF_DS "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/effectful/source/"
    click EF_DSink "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/effectful/sink/"
    click EF_PL "https://github.com/yadavan88/data-plumber/blob/main/src/main/scala/com/yadavan88/dataplumber/effectful/IODataPlumber.scala"

    click OF_DS "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/offsetable/source/"
    click OF_DSink "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/offsetable/sink/"
    click OF_PL "https://github.com/yadavan88/data-plumber/blob/main/src/main/scala/com/yadavan88/dataplumber/offsetable/OffsetableDataPlumber.scala"
    click OF_RC "https://github.com/yadavan88/data-plumber/blob/main/src/main/scala/com/yadavan88/dataplumber/offsetable/RedisClient.scala"

    click ST_DS "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/streaming/source/"
    click ST_DSink "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/com/yadavan88/dataplumber/streaming/sink/"
    click ST_PL "https://github.com/yadavan88/data-plumber/blob/main/src/main/scala/com/yadavan88/dataplumber/streaming/StreamingDataPlumber.scala"

    click EX "https://github.com/yadavan88/data-plumber/tree/main/src/main/scala/examples/"

    %% Styles
    classDef simple fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef effectful fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef offsetable fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef streaming fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef external fill:#F8F8F8,stroke:#000,stroke-width:2px,stroke-dasharray: 5,5;
```
