# data-integrations/elasticsearch-plugins architecture

> Architecture diagram of the GitHub repository data-integrations/elasticsearch-plugins, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/data-integrations/elasticsearch-plugins
- Repository: https://github.com/data-integrations/elasticsearch-plugins

## Overview

##### Main Components
- CDAP Pipeline Environment – This is the runtime orchestration system where all plugins are loaded and executed.
- Elasticsearch Plugins – Break these into distinct plugin types:
- Elasticsearch Batch Source (src/main/java/io/cdap/plugin/elastic/source/ElasticsearchSource and its config)
- Elasticsearch Batch Sink (src/main/java/io/cdap/plugin/elastic/sink/BatchElasticsearchSink and its config)
- (Optional) Elasticsearch Realtime Sink – Mentioned in the README, even though the code detail might not be in this tree. It is still part of the overall offering.
- Plugin Core and Shared Libraries – For example, the base configuration and helper classes (such as BaseElasticsearchConfig and RecordWritableConverter) and supporting integration with the Elasticsearch Hadoop library (packages under org/elasticsearch/hadoop).
- External Elasticsearch Cluster – The target data store for both reading data (source) and writing data (sink).
- Deployment Artifacts – Highlight the build and deployment configuration (e.g., pom.xml, widget JSON files for plugin configuration) which will be used to load these plugins via the CDAP CLI.

##### Relationships and Interactions
- The CDAP Pipeline environment loads the various Elasticsearch plugins, which then serve as either entry points for ingesting data from Elasticsearch or as exit points for writing data.
- The plugins use a common configuration (handled in shared classes) and depend on the Elasticsearch and Hadoop integrations (note the org/elasticsearch/hadoop package).
- The Elasticsearch Batch Source plugin component when data is being ingested.
- The Elasticsearch Batch Sink (and Realtime Sink, if applicable) when data is sent to an Elasticsearch cluster.

##### Architectural Patterns and Design Principles
- Plugin-based Architecture – The system is modular: each plugin (source, sink) is a standalone component that can be loaded into the CDAP platform.
- Configuration Sharing and Abstraction – A common configuration (e.g., BaseElasticsearchConfig) and helper utilities (e.g., RecordWritableConverter) are utilized across plugins.
- Integration with External Systems – Emphasize the coupling between the CDAP environment and the external Elasticsearch service through the use of Elasticsearch Hadoop libraries.

##### Technologies, Frameworks, and Libraries
- CDAP (Cask Data Application Platform) – The host pipeline system where the plugins operate.
- Elasticsearch – The target external data store.
- Maven – Indicated by the pom.xml; used for building the project.
- Java – The primary programming language (visible from src/main/java structure).
- Elasticsearch-Hadoop – Integration library for communication between Hadoop (and by extension, CDAP’s batch operations) and Elasticsearch.

## Diagram (Mermaid)

```mermaid
flowchart TD
    CP("CDAP Pipeline Environment"):::cdap

    subgraph "Plugins"
        subgraph "Elasticsearch Batch Source Plugin"
            SourceMain("ElasticsearchSource"):::plugin
            SourceConfig("ElasticsearchSourceConfig"):::plugin
        end
        subgraph "Elasticsearch Batch Sink Plugin"
            SinkMain("BatchElasticsearchSink"):::plugin
            SinkConfig("ElasticsearchSinkConfig"):::plugin
        end
        Realtime("Elasticsearch Realtime Sink Plugin"):::plugin
    end

    subgraph "Shared Config & Libraries"
        BaseConfig("BaseElasticsearchConfig"):::sharedDetail
        RecordConverter("RecordWritableConverter"):::sharedDetail
    end

    subgraph "Elasticsearch-Hadoop Integration"
        EsHadoopMain("org/elasticsearch/hadoop"):::esHadoop
        EsMR("MR: EsInputFormat/EsOutputFormat"):::esHadoop
    end

    External("External Elasticsearch Cluster"):::external

    subgraph "Deployment Artifacts"
        pom("pom.xml"):::deploy
        json1("Elasticsearch-batchsink.json"):::deploy
        json2("Elasticsearch-batchsource.json"):::deploy
    end

    subgraph "Documentation"
        DocSink("Elasticsearch-batchsink.md"):::doc
        DocSource("Elasticsearch-batchsource.md"):::doc
    end

    %% Relationships
    CP -->|"loads"| SourceMain
    CP -->|"loads"| SinkMain
    CP -->|"loads"| Realtime

    SourceMain -->|"uses"| BaseConfig
    SinkMain -->|"uses"| BaseConfig
    Realtime  -->|"uses"| BaseConfig

    BaseConfig -->|"integrates"| EsHadoopMain

    SourceMain -->|"reads"| External
    SinkMain -->|"writes"| External
    Realtime  -->|"writes"| External

    pom -->|"deploys"| CP
    DocSink -->|"docs"| pom
    DocSource -->|"docs"| pom

    %% Click Events
    click CP "https://github.com/data-integrations/elasticsearch-plugins/tree/develop/src/main/java/io/cdap/plugin"
    click SourceMain "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/source/ElasticsearchSource.java"
    click SourceConfig "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/source/ElasticsearchSourceConfig.java"
    click SinkMain "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/sink/BatchElasticsearchSink.java"
    click SinkConfig "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/sink/ElasticsearchSinkConfig.java"
    click BaseConfig "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/BaseElasticsearchConfig.java"
    click RecordConverter "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/src/main/java/io/cdap/plugin/elastic/RecordWritableConverter.java"
    click EsHadoopMain "https://github.com/data-integrations/elasticsearch-plugins/tree/develop/src/main/java/org/elasticsearch/hadoop"
    click EsMR "https://github.com/data-integrations/elasticsearch-plugins/tree/develop/src/main/java/org/elasticsearch/hadoop/mr"
    click pom "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/pom.xml"
    click json1 "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/widgets/Elasticsearch-batchsink.json"
    click json2 "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/widgets/Elasticsearch-batchsource.json"
    click DocSink "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/docs/Elasticsearch-batchsink.md"
    click DocSource "https://github.com/data-integrations/elasticsearch-plugins/blob/develop/docs/Elasticsearch-batchsource.md"

    %% Styles
    classDef cdap fill:#aec6ff,stroke:#000,stroke-width:1px;
    classDef plugin fill:#77dd77,stroke:#000,stroke-width:1px;
    classDef shared fill:#fff5ba,stroke:#000,stroke-width:1px;
    classDef sharedDetail fill:#fff68f,stroke:#000,stroke-width:1px;
    classDef esHadoop fill:#d291bc,stroke:#000,stroke-width:1px;
    classDef external fill:#ff6961,stroke:#000,stroke-width:1px;
    classDef deploy fill:#ffb347,stroke:#000,stroke-width:1px;
    classDef doc fill:#aec6cf,stroke:#000,stroke-width:1px;
```
