How data-integrations/hierarchy-plugins works
Project type and purpose
• This is an open-source plugin library (“hierarchy-plugins”) for the CDAP (Cask Data Application Platform) data-pipeline framework. It provides two transform plugins—FlattenRecord and CreateRecordTransform—that users can drop into CDAP ETL pipelines to manipulate record hierarchies.
File structure
• Top-level build and config files:
• pom.xml – Maven build, dependencies on CDAP SDK
• checkstyle.xml, suppressions.xml – code style rules
• Documentation:
• docs/…-transform.md – detailed behavior of each plugin
• examples/…pipeline.json – sample pipeline config using the plugins
• Plugin descriptors/widgets:
• widgets/*.json – UI form definitions for plugin configuration in the CDAP UI
• Source code:
• src/main/java/io/cdap/plugin/transform/CreateRecordTransform.java
• src/main/java/io/cdap/plugin/FlattenRecord.java
• Automated tests:
• src/test/java/…CreateRecordTransformTest.java
• src/test/java/…FlattenRecordTest.java
Components & relationships
a. Main components
• Developer and CI (Maven build server)
• Plugin code modules: FlattenRecord and CreateRecordTransform
• Build system (Maven) producing a plugin JAR
• CDAP Platform: Plugin Registry & ETL Pipeline runtime
• Pipeline components: Source → FlattenRecord → CreateRecordTransform → Sink
• Configuration widgets (JSON) used by the CDAP UI
• Data schemas and underlying datasets (CDAP Datasets or Streams)
b. Interactions
1. Developer checks out code, runs mvn package → JAR
2. JAR is uploaded/deployed to CDAP’s plugin registry
3. In CDAP UI, user drags “FlattenRecord” or “CreateRecord” stage onto the canvas; widget JSON drives the configuration form
4. At pipeline start: data flows from Source → FlattenRecord (flattens hierarchical fields) → CreateRecordTransform (builds new record structure) → Sink (writes to a Dataset)
5. Tests validate correct behavior before deployment
c. Architectural patterns & principles
• Plugin-based architecture: each transform implements the CDAP Transform interface
• Separation of concerns: transform logic vs. UI configuration vs. testing
• Reusable, stateless stages suitable for parallel execution in the CDAP pipeline engine
d. Key technologies
• Java 8+, Maven, CDAP SDK
• JSON for pipeline definitions and widget metadata
• JUnit for unit tests