# nerdammer/spark-hbase-connector architecture

> Architecture diagram of the GitHub repository nerdammer/spark-hbase-connector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nerdammer/spark-hbase-connector
- Repository: https://github.com/nerdammer/spark-hbase-connector

## Overview

- Connector Core Library – This is the heart of the project, where the actual functionality is implemented. It resides in the “src/main/scala/it/nerdammer/spark/hbase” package (and its subpackages). It includes key classes such as HBaseReaderBuilder, HBaseWriterBuilder, HBaseSparkContext, HBaseUtils, and other utilities.
- Conversion Layer – Identify components responsible for mapping between HBase data formats and native Scala types. This includes the “conversion” subpackage (FieldMapper, FieldReader, FieldWriter, etc.) and the support for custom mapping via case classes.
- Spark API Integration – Represent the integration points (implicit conversions) that let an RDD be “converted” to and from an HBase table. This layer uses extensions like “toHBaseTable” on RDDs.
- Configuration & Deployment – Include how the library is configured (via hbase-site.xml, spark-submit parameters, JVM arguments, or explicitly in code). Highlight that configuration files (in “src/main/resources”) are used to set the HBase host.
- Testing & Examples – Although these are not part of the runtime architecture, include an “Examples” component and “Integration Tests” (in “src/test/scala” and “examples”) to show how the connector is exercised in sample Spark applications.
- Build & CI Infrastructure – Represent the build toolchain (sbt, build.sbt at the root and in the examples) and the CI scripts (inside “scripts/ci”). Also include Travis CI integration from the README badge.

##### Relationships and interactions
- Data Flow in a Spark Application that uses the Connector:
- The Spark Application (Spark Context/Driver) creates an RDD.
- The RDD is enriched with implicit conversion methods (import it.nerdammer.spark.hbase._) to call “toHBaseTable()” for writing, or “hbaseTable()” for reading.
- The conversion layer is used within these APIs to map native Scala tuples or custom case classes into HBase row/column structure.
- Configuration components (from hbase-site.xml or runtime parameters) provide HBase host information to the connector.
- The Connector’s role is therefore to bridge Spark’s RDDs and DataFrames (as shown with conversions into Spark SQL Row objects) with HBase operations.

##### Architectural patterns or design principles
- Abstraction & Implicit Conversions – Emphasize how the library encapsulates HBase complexity using implicit conversion methods.
- Separation of Concerns – The project isolates high-level business logic (reading/writing operations) from low-level HBase interactions, achieved through clean module boundaries (conversion, configuration, API extensions).
- Configuration Flexibility – Multiple ways to set the HBase host (e.g., command line, XML file, JVM parameter, code-level configuration) indicate an extensible design.
- Test-Driven and Example-Driven – The clear structure of test and example directories suggests that the library is designed with hands-on validation in mind.

##### Relevant technologies, frameworks, or libraries
- Apache Spark (notably “spark-core” in version 1.6.0 as a provided dependency).
- Apache HBase (common, client, server libraries).
- Scala and sbt for the build.
- Travis CI for continuous integration.
- Optional SparkSQL/Dataframe integration segments which highlight conversion from HBaseRDD to DataFrames.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Application
    subgraph "User Application"
        A["Spark Application"]:::external
    end

    %% Connector Library Modules
    subgraph "Connector Library"
        B["Spark API Integration"]:::api
        C["Conversion Layer"]:::conversion
        subgraph "Connector Core Library"
            X["Core Library"]:::core
            D["HBaseReaderBuilder"]:::core
            E["HBaseWriterBuilder"]:::core
            F["HBaseSparkContext"]:::core
            G["HBaseUtils"]:::core
        end
    end

    %% Configuration & Deployment
    subgraph "Configuration & Deployment"
        H["Main Config"]:::config
        I["Test Config"]:::config
        J["Example Config"]:::config
    end

    %% Testing & Examples
    subgraph "Testing & Examples"
        K["Test Suite"]:::test
        L["Example Application"]:::test
    end

    %% Build & CI Infrastructure
    subgraph "Build & CI Infrastructure"
        M["Build Config (Root)"]:::build
        N["Build Config (Example)"]:::build
        O["CI Scripts"]:::build
    end

    %% External HBase cluster
    P["HBase Cluster"]:::external

    %% Data Flow Arrows
    A -->|"calls"| B
    B -->|"invokes"| X
    C -->|"supports"| X
    H -->|"configures"| X
    I -->|"configures"| X
    J -->|"configures"| X
    X -->|"includes"| D
    X -->|"includes"| E
    X -->|"includes"| F
    X -->|"includes"| G
    X -->|"reads/writes"| P
    L -->|"demonstrates"| A

    %% Click Events
    click X "https://github.com/nerdammer/spark-hbase-connector/tree/master/src/main/scala/it/nerdammer/spark/hbase"
    click D "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/main/scala/it/nerdammer/spark/hbase/HBaseReaderBuilder.scala"
    click E "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/main/scala/it/nerdammer/spark/hbase/HBaseWriterBuilder.scala"
    click F "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/main/scala/it/nerdammer/spark/hbase/HBaseSparkContext.scala"
    click G "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/main/scala/it/nerdammer/spark/hbase/HBaseUtils.scala"
    click B "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/main/scala/it/nerdammer/spark/hbase/package.scala"
    click C "https://github.com/nerdammer/spark-hbase-connector/tree/master/src/main/scala/it/nerdammer/spark/hbase/conversion"
    click H "https://github.com/nerdammer/spark-hbase-connector/tree/master/src/main/resources"
    click I "https://github.com/nerdammer/spark-hbase-connector/blob/master/src/test/resources/hbase-site.xml"
    click J "https://github.com/nerdammer/spark-hbase-connector/tree/master/examples/spark-2-0-scala-2-11-sample/src/main/resources"
    click K "https://github.com/nerdammer/spark-hbase-connector/tree/master/src/test/scala"
    click L "https://github.com/nerdammer/spark-hbase-connector/tree/master/examples/spark-2-0-scala-2-11-sample"
    click M "https://github.com/nerdammer/spark-hbase-connector/blob/master/build.sbt"
    click N "https://github.com/nerdammer/spark-hbase-connector/blob/master/examples/spark-2-0-scala-2-11-sample/build.sbt"
    click O "https://github.com/nerdammer/spark-hbase-connector/tree/master/scripts/ci"

    %% Styles
    classDef external fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef api fill:#C2FFC2,stroke:#333,stroke-width:2px;
    classDef conversion fill:#FFFFA0,stroke:#333,stroke-width:2px;
    classDef core fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef config fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef test fill:#DDA0DD,stroke:#333,stroke-width:2px;
    classDef build fill:#FFB6C1,stroke:#333,stroke-width:2px;
```
