# data-integrations/hierarchy-plugins architecture

> Architecture diagram of the GitHub repository data-integrations/hierarchy-plugins, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/data-integrations/hierarchy-plugins
- Repository: https://github.com/data-integrations/hierarchy-plugins

## Overview

##### Project type and purpose
- This is an open-source plugin library (“hierarchy-plugins”) for the CDAP (Cask Data Application Platform) data-pipeline framework. It provides two transform plugins—FlattenRecord and CreateRecordTransform—that users can drop into CDAP ETL pipelines to manipulate record hierarchies.

##### File structure
- Top-level build and config files:
- pom.xml – Maven build, dependencies on CDAP SDK
- checkstyle.xml, suppressions.xml – code style rules
- Documentation:
- docs/…-transform.md – detailed behavior of each plugin
- examples/…pipeline.json – sample pipeline config using the plugins
- Plugin descriptors/widgets:
- widgets/*.json – UI form definitions for plugin configuration in the CDAP UI
- Source code:
- src/main/java/io/cdap/plugin/transform/CreateRecordTransform.java
- src/main/java/io/cdap/plugin/FlattenRecord.java
- Automated tests:
- src/test/java/…CreateRecordTransformTest.java
- src/test/java/…FlattenRecordTest.java

##### Components & relationships
a. Main components
- Developer and CI (Maven build server)
- Plugin code modules: FlattenRecord and CreateRecordTransform
- Build system (Maven) producing a plugin JAR
- CDAP Platform: Plugin Registry & ETL Pipeline runtime
- Pipeline components: Source → FlattenRecord → CreateRecordTransform → Sink
- Configuration widgets (JSON) used by the CDAP UI
- Data schemas and underlying datasets (CDAP Datasets or Streams)
b. Interactions
1. Developer checks out code, runs mvn package → JAR
2. JAR is uploaded/deployed to CDAP’s plugin registry
3. In CDAP UI, user drags “FlattenRecord” or “CreateRecord” stage onto the canvas; widget JSON drives the configuration form
4. At pipeline start: data flows from Source → FlattenRecord (flattens hierarchical fields) → CreateRecordTransform (builds new record structure) → Sink (writes to a Dataset)
5. Tests validate correct behavior before deployment
c. Architectural patterns & principles
- Plugin-based architecture: each transform implements the CDAP Transform interface
- Separation of concerns: transform logic vs. UI configuration vs. testing
- Reusable, stateless stages suitable for parallel execution in the CDAP pipeline engine
d. Key technologies
- Java 8+, Maven, CDAP SDK
- JSON for pipeline definitions and widget metadata
- JUnit for unit tests

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Build & Deploy Lane
    subgraph "Build & Deploy" 
        direction TB
        DevCI["Developer / CI"]:::actor
        Maven["Maven Build System"]:::build
        CodeStyle["Code Style Config"]:::config
        SrcFlatten["FlattenRecord Plugin"]:::code
        SrcCreate["CreateRecordTransform Plugin"]:::code
        TestFlatten["FlattenRecord Unit Tests"]:::test
        TestCreate["CreateRecordTransform Unit Tests"]:::test
        JAR["Plugin JAR Artifact"]:::artifact
        Registry["CDAP Plugin Registry"]:::platform

        DevCI -->|checks out code| SrcFlatten
        DevCI -->|checks out code| SrcCreate
        DevCI -->|runs mvn package| Maven
        Maven -->|applies| CodeStyle
        Maven -->|compiles & packages| JAR
        SrcFlatten -->|tested by| TestFlatten
        SrcCreate -->|tested by| TestCreate
        JAR -->|deployed to| Registry
    end

    %% CDAP UI Lane
    subgraph "CDAP UI" 
        direction TB
        Widgets["UI Configuration Widgets"]:::ui
        UIForm["Plugin Configuration Form"]:::ui
        Docs["Plugin Documentation"]:::docs
        Examples["Example Pipeline Definitions"]:::docs

        Widgets --> UIForm
        Docs --> UIForm
        Examples --> UIForm
        Registry --> UIForm
    end

    %% Runtime Pipeline Lane
    subgraph "Runtime Pipeline" 
        direction TB
        Source["Source (CDAP Dataset/Stream)"]:::datastore
        FlatStage["FlattenRecord Transform"]:::stage
        CreateStage["CreateRecordTransform"]:::stage
        Sink["Sink (CDAP Dataset)"]:::datastore

        Source -->|JSON/Avro| FlatStage
        FlatStage -->|transformed record| CreateStage
        CreateStage -->|output record| Sink
    end

    %% Styles
    classDef actor fill:#E3F2FD,stroke:#90CAF9;
    classDef build fill:#E8EAF6,stroke:#5C6BC0;
    classDef config fill:#FFF3E0,stroke:#FB8C00;
    classDef code fill:#E8F5E9,stroke:#43A047;
    classDef test fill:#F3E5F5,stroke:#AB47BC;
    classDef artifact fill:#FFEBEE,stroke:#E53935;
    classDef platform fill:#ECEFF1,stroke:#78909C,stroke-dasharray: 5 5;
    classDef ui fill:#FCE4EC,stroke:#D81B60;
    classDef docs fill:#FFFDE7,stroke:#FDD835;
    classDef stage fill:#E0F7FA,stroke:#00838F;
    classDef datastore fill:#F1F8E9,stroke:#689F38;

    %% Click Events
    click Maven "https://github.com/data-integrations/hierarchy-plugins/blob/develop/pom.xml"
    click CodeStyle "https://github.com/data-integrations/hierarchy-plugins/blob/develop/checkstyle.xml"
    click CodeStyle "https://github.com/data-integrations/hierarchy-plugins/blob/develop/suppressions.xml"
    click SrcFlatten "https://github.com/data-integrations/hierarchy-plugins/blob/develop/src/main/java/io/cdap/plugin/FlattenRecord.java"
    click SrcCreate "https://github.com/data-integrations/hierarchy-plugins/blob/develop/src/main/java/io/cdap/plugin/transform/CreateRecordTransform.java"
    click TestFlatten "https://github.com/data-integrations/hierarchy-plugins/blob/develop/src/test/java/io/cdap/plugin/FlattenRecordTest.java"
    click TestCreate "https://github.com/data-integrations/hierarchy-plugins/blob/develop/src/test/java/io/cdap/plugin/transform/CreateRecordTransformTest.java"
    click Widgets "https://github.com/data-integrations/hierarchy-plugins/blob/develop/widgets/FlattenRecord-transform.json"
    click Widgets "https://github.com/data-integrations/hierarchy-plugins/blob/develop/widgets/CreateRecord-transform.json"
    click Docs "https://github.com/data-integrations/hierarchy-plugins/blob/develop/docs/FlattenRecord-transform.md"
    click Docs "https://github.com/data-integrations/hierarchy-plugins/blob/develop/docs/CreateRecord-transform.md"
    click Examples "https://github.com/data-integrations/hierarchy-plugins/blob/develop/examples/example-create-record-tranform-pipeline.json"
    click Docs "https://github.com/data-integrations/hierarchy-plugins/tree/develop/icons/"
```
