How data-integrations/elasticsearch-plugins works
Main Components
• CDAP Pipeline Environment – This is the runtime orchestration system where all plugins are loaded and executed.
• Elasticsearch Plugins – Break these into distinct plugin types:
• Elasticsearch Batch Source (src/main/java/io/cdap/plugin/elastic/source/ElasticsearchSource and its config)
• Elasticsearch Batch Sink (src/main/java/io/cdap/plugin/elastic/sink/BatchElasticsearchSink and its config)
• (Optional) Elasticsearch Realtime Sink – Mentioned in the README, even though the code detail might not be in this tree. It is still part of the overall offering.
• Plugin Core and Shared Libraries – For example, the base configuration and helper classes (such as BaseElasticsearchConfig and RecordWritableConverter) and supporting integration with the Elasticsearch Hadoop library (packages under org/elasticsearch/hadoop).
• External Elasticsearch Cluster – The target data store for both reading data (source) and writing data (sink).
• Deployment Artifacts – Highlight the build and deployment configuration (e.g., pom.xml, widget JSON files for plugin configuration) which will be used to load these plugins via the CDAP CLI.
Relationships and Interactions
• The CDAP Pipeline environment loads the various Elasticsearch plugins, which then serve as either entry points for ingesting data from Elasticsearch or as exit points for writing data.
• The plugins use a common configuration (handled in shared classes) and depend on the Elasticsearch and Hadoop integrations (note the org/elasticsearch/hadoop package).
• The Elasticsearch Batch Source plugin component when data is being ingested.
• The Elasticsearch Batch Sink (and Realtime Sink, if applicable) when data is sent to an Elasticsearch cluster.
Architectural Patterns and Design Principles
• Plugin-based Architecture – The system is modular: each plugin (source, sink) is a standalone component that can be loaded into the CDAP platform.
• Configuration Sharing and Abstraction – A common configuration (e.g., BaseElasticsearchConfig) and helper utilities (e.g., RecordWritableConverter) are utilized across plugins.
• Integration with External Systems – Emphasize the coupling between the CDAP environment and the external Elasticsearch service through the use of Elasticsearch Hadoop libraries.
Technologies, Frameworks, and Libraries
• CDAP (Cask Data Application Platform) – The host pipeline system where the plugins operate.
• Elasticsearch – The target external data store.
• Maven – Indicated by the pom.xml; used for building the project.
• Java – The primary programming language (visible from src/main/java structure).
• Elasticsearch-Hadoop – Integration library for communication between Hadoop (and by extension, CDAP’s batch operations) and Elasticsearch.