How nerdammer/spark-hbase-connector works
• Connector Core Library – This is the heart of the project, where the actual functionality is implemented. It resides in the “src/main/scala/it/nerdammer/spark/hbase” package (and its subpackages). It includes key classes such as HBaseReaderBuilder, HBaseWriterBuilder, HBaseSparkContext, HBaseUtils, and other utilities.
• Conversion Layer – Identify components responsible for mapping between HBase data formats and native Scala types. This includes the “conversion” subpackage (FieldMapper, FieldReader, FieldWriter, etc.) and the support for custom mapping via case classes.
• Spark API Integration – Represent the integration points (implicit conversions) that let an RDD be “converted” to and from an HBase table. This layer uses extensions like “toHBaseTable” on RDDs.
• Configuration & Deployment – Include how the library is configured (via hbase-site.xml, spark-submit parameters, JVM arguments, or explicitly in code). Highlight that configuration files (in “src/main/resources”) are used to set the HBase host.
• Testing & Examples – Although these are not part of the runtime architecture, include an “Examples” component and “Integration Tests” (in “src/test/scala” and “examples”) to show how the connector is exercised in sample Spark applications.
• Build & CI Infrastructure – Represent the build toolchain (sbt, build.sbt at the root and in the examples) and the CI scripts (inside “scripts/ci”). Also include Travis CI integration from the README badge.
Relationships and interactions
• Data Flow in a Spark Application that uses the Connector:
• The Spark Application (Spark Context/Driver) creates an RDD.
• The RDD is enriched with implicit conversion methods (import it.nerdammer.spark.hbase._) to call “toHBaseTable()” for writing, or “hbaseTable()” for reading.
• The conversion layer is used within these APIs to map native Scala tuples or custom case classes into HBase row/column structure.
• Configuration components (from hbase-site.xml or runtime parameters) provide HBase host information to the connector.
• The Connector’s role is therefore to bridge Spark’s RDDs and DataFrames (as shown with conversions into Spark SQL Row objects) with HBase operations.
Architectural patterns or design principles
• Abstraction & Implicit Conversions – Emphasize how the library encapsulates HBase complexity using implicit conversion methods.
• Separation of Concerns – The project isolates high-level business logic (reading/writing operations) from low-level HBase interactions, achieved through clean module boundaries (conversion, configuration, API extensions).
• Configuration Flexibility – Multiple ways to set the HBase host (e.g., command line, XML file, JVM parameter, code-level configuration) indicate an extensible design.
• Test-Driven and Example-Driven – The clear structure of test and example directories suggests that the library is designed with hands-on validation in mind.
Relevant technologies, frameworks, or libraries
• Apache Spark (notably “spark-core” in version 1.6.0 as a provided dependency).
• Apache HBase (common, client, server libraries).
• Scala and sbt for the build.
• Travis CI for continuous integration.
• Optional SparkSQL/Dataframe integration segments which highlight conversion from HBaseRDD to DataFrames.