# lucidworks/hbase-indexer architecture

> Architecture diagram of the GitHub repository lucidworks/hbase-indexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucidworks/hbase-indexer
- Repository: https://github.com/lucidworks/hbase-indexer

## Overview

##### Project type and purpose
- This is an open-source, Java-based backend system (“HBase Indexer”) that hooks into HBase replication (via the SEP library) to stream or bulk‐index HBase row updates into Solr (or Fusion) in real time or via MapReduce.
- Key use cases: real-time indexing of HBase table changes, bulk indexing via Hadoop MR jobs, CLI management of indexers, and monitoring replication.

##### High-level file-structure analysis
- Top‐level aggregator POM and submodules:
- hbase‐sep (API, impl, tools, demos) – SEP library for HBase replication.
- hbase‐indexer‐common – shared config, utilities, metrics.
- hbase‐indexer‐engine – core mapping & Solr writing logic.
- hbase‐indexer‐server – daemon process (Master, Supervisor, REST API).
- hbase‐indexer‐cli – command-line tooling to add/update/list indexers.
- hbase‐indexer‐mr – bulk MapReduce indexing job.
- hbase‐indexer‐model – data model for indexer definitions and state.
- hbase‐indexer‐dist – assembly and packaging scripts.
- hbase‐indexer‐demo, hbase‐sep‐demo – example apps.

##### Core README insights
- The Indexer runs as an HBase replication sink daemon.
- Configuration: XML files under conf/ define ZK, HBase, Solr endpoints and mapping from HBase columns → Solr fields.
- CLI uses a Java Main that registers indexer definitions via REST to the running daemon.
- Optional Kerberos security and Fusion pipeline output are supported.

##### Main components
a. External systems:
- HBase Cluster (RegionServers + ZK quorum)
- ZooKeeper ensemble (shared by HBase and Solr)
- Solr (or Fusion) Cluster
- Hadoop/YARN – for MapReduce jobs (bulk indexing)
b. SEP Replication Layer:
- hbase-sep-api + hbase-sep-impl on each HBase node
c. Indexer Daemon Cluster:
- hbase-indexer-server processes
- REST API endpoints for indexer management
- IndexerMaster and IndexerSupervisor modules
- BatchStateUpdater
d. Core Engine Modules:
- hbase-indexer-engine: parse events, build SolrInputDocuments, sharding, writer factories
- hbase-indexer-common: configuration loading, metrics, utilities
- hbase-indexer-model: in-memory/persistent model of indexer definitions
- hbase-indexer-cli: CLI commands (add, update, delete, list indexers)
e. Bulk-indexing Path:
- hbase-indexer-mr job that reads HBase tables → uses engine writer classes → writes to Solr
f. Packaging & Deployment:
- hbase-indexer-dist: assembly of scripts, config, and JARs
- bin/ and conf/ directory structure

##### Relationships and interactions
- Within daemon: events flow into Engine modules → map & transform → Solr writers → index documents
- Bulk MR jobs → standalone MapReduce cluster → use same Engine and Common libraries → write to Solr
- All components register metrics via MetricsUtil (common)
- ZK used by HBase, SEP, and Solr for coordination

##### Architectural patterns & design principles
- Modular multi‐project Maven structure for clear separation of concerns
- Plugin/factory pattern in Engine (IndexerComponentFactory) for pluggable parsers and writers
- Model‐View‐Controller style layering: CLI/REST (view), Master/Supervisor (controller), Engine/Model (model)
- Reusable common library for config, metrics, JSON etc.
- Assembly layer (dist) decouples packaging from code

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems"
        HBaseCluster["HBase Cluster"]:::external
        ZooKeeper["ZooKeeper Ensemble"]:::external
        SolrCluster["Solr/Fusion Cluster"]:::external
        YARN["Hadoop/YARN"]:::external
    end

    %% SEP Replication Layer
    subgraph "SEP Replication Layer"
        sepAPI["hbase-sep-api"]:::sep
        sepImpl["hbase-sep-impl"]:::sep
        sepTools["SEP Monitoring Tools"]:::sep
        sepDemo["SEP Demo Application"]:::sep
    end

    %% Core Libraries
    subgraph "Common Library"
        commonUtil["hbase-indexer-common"]:::common
    end

    subgraph "Data Model"
        modelAPI["Model API Interfaces"]:::model
        modelImpl["Model Implementation"]:::model
    end

    %% Core Engine Modules
    subgraph "Core Engine"
        confFactory["Configuration & Factory"]:::engine
        indexingLogic["Indexing Logic"]:::engine
        parsingMapping["Parsing & Mapping"]:::engine
    end

    %% Indexer Daemon Cluster
    subgraph "Indexer Daemon"
        Main["Main Entry Point"]:::server
        Master["IndexerMaster & BatchStateUpdater"]:::server
        Supervisor["IndexerSupervisor & Registry"]:::server
        REST["REST API Endpoints"]:::server
    end

    %% CLI Tools
    subgraph "CLI Tools"
        AddCli["AddIndexerCli"]:::cli
        ListCli["ListIndexersCli"]:::cli
        UpdateCli["UpdateIndexerCli"]:::cli
        BaseCli["BaseCli Utilities"]:::cli
    end

    %% Bulk MapReduce Job
    subgraph "Bulk MapReduce"
        MRTool["HBaseMapReduceIndexerTool"]:::mr
        MRMapper["HBaseIndexerMapper"]:::mr
        MRWriter["Writers & Counters"]:::mr
    end

    %% Packaging & Distribution
    subgraph "Packaging & Distribution"
        assembly["Assembly Descriptor"]:::dist
        scripts["Runtime Scripts"]:::dist
    end

    %% Demo Applications
    subgraph "Demo Applications"
        indexerDemo["Indexer Demo"]:::demo
    end

    %% Data/Event Flows
    HBaseCluster -->|"HBase Replication RPC"| sepAPI
    sepAPI --> sepImpl
    sepImpl -->|"push events"| Main
    Main --> Master
    Master --> Supervisor
    Supervisor --> confFactory
    confFactory --> indexingLogic
    indexingLogic --> parsingMapping
    parsingMapping -->|"SolrInputDocument"| SolrCluster

    CLI["CLI"] -.-> AddCli
    AddCli -->|"HTTP/REST"| REST
    ListCli --> REST
    UpdateCli --> REST
    BaseCli --> AddCli

    REST --> Master
    MRTool --> indexingLogic
    MRMapper --> MRWriter
    MRWriter --> SolrCluster

    %% Common & Model Dependencies
    commonUtil --> confFactory
    commonUtil --> indexingLogic
    commonUtil --> parsingMapping
    commonUtil --> Master
    modelAPI --> Supervisor
    modelImpl --> Supervisor

    %% ZK Coordination
    ZooKeeper -.-> HBaseCluster
    ZooKeeper -.-> sepImpl
    ZooKeeper -.-> SolrCluster
    ZooKeeper -.-> Main

    %% Packaging & Demo
    assembly --> scripts
    scripts --> Main
    indexerDemo --> confFactory

    %% Click Events
    click sepAPI "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-sep/hbase-sep-api/src/main/java/com/ngdata/sep"
    click sepImpl "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-sep/hbase-sep-impl/src/main/java/com/ngdata/sep/impl"
    click sepTools "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-sep/hbase-sep-tools/src/main/java/com/ngdata/sep/tools/monitoring"
    click sepDemo "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-sep/hbase-sep-demo/src/main/java/com/ngdata/sep/demo"
    click commonUtil "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-indexer-common/src/main/java/com/ngdata/hbaseindexer"
    click confFactory "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-engine/src/main/java/com/ngdata/hbaseindexer/conf/DefaultIndexerComponentFactory.java"
    click indexingLogic "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-engine/src/main/java/com/ngdata/hbaseindexer/indexer/Indexer.java"
    click parsingMapping "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-engine/src/main/java/com/ngdata/hbaseindexer/parse/ResultToSolrMapper.java"
    click modelAPI "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-model/src/main/java/com/ngdata/hbaseindexer/model/api/IndexerDefinition.java"
    click modelImpl "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-model/src/main/java/com/ngdata/hbaseindexer/model/impl/IndexerModelImpl.java"
    click Main "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-server/src/main/java/com/ngdata/hbaseindexer/Main.java"
    click Master "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-server/src/main/java/com/ngdata/hbaseindexer/master/IndexerMaster.java"
    click Supervisor "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-server/src/main/java/com/ngdata/hbaseindexer/supervisor/IndexerSupervisor.java"
    click REST "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-server/src/main/java/com/ngdata/hbaseindexer/rest/IndexerResource.java"
    click AddCli "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-cli/src/main/java/com/ngdata/hbaseindexer/cli/AddIndexerCli.java"
    click ListCli "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-cli/src/main/java/com/ngdata/hbaseindexer/cli/ListIndexersCli.java"
    click UpdateCli "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-cli/src/main/java/com/ngdata/hbaseindexer/cli/UpdateIndexerCli.java"
    click BaseCli "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-cli/src/main/java/com/ngdata/hbaseindexer/cli/BaseCli.java"
    click MRTool "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-mr/src/main/java/com/ngdata/hbaseindexer/mr/HBaseMapReduceIndexerTool.java"
    click MRMapper "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-mr/src/main/java/com/ngdata/hbaseindexer/mr/HBaseIndexerMapper.java"
    click MRWriter "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-mr/src/main/java/com/ngdata/hbaseindexer/mr/BufferedSolrInputDocumentWriter.java"
    click assembly "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-dist/src/assembly/assembly.xml"
    click scripts "https://github.com/lucidworks/hbase-indexer/tree/master/hbase-indexer-dist/src/main/resources/scripts/hbase-indexer"
    click indexerDemo "https://github.com/lucidworks/hbase-indexer/blob/master/hbase-indexer-demo/src/main/java/com/ngdata/hbaseindexer/demo/DemoMessageIngester.java"

    %% Styles
    classDef external fill:#a0c4ff,stroke:#034f84,stroke-width:2px
    classDef sep fill:#ffe066,stroke:#ffba08,stroke-width:2px
    classDef common fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    classDef model fill:#d1c4e9,stroke:#512da8,stroke-width:2px
    classDef engine fill:#fff3b0,stroke:#f9a825,stroke-width:2px
    classDef server fill:#ffd5cd,stroke:#d84315,stroke-width:2px
    classDef cli fill:#eeeeee,stroke:#616161,stroke-width:2px
    classDef mr fill:#e0f7fa,stroke:#006064,stroke-width:2px
    classDef dist fill:#f0f4c3,stroke:#827717,stroke-width:2px
    classDef demo fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px
```
