How lucidworks/hbase-indexer works
Project type and purpose
• This is an open-source, Java-based backend system (“HBase Indexer”) that hooks into HBase replication (via the SEP library) to stream or bulk‐index HBase row updates into Solr (or Fusion) in real time or via MapReduce.
• Key use cases: real-time indexing of HBase table changes, bulk indexing via Hadoop MR jobs, CLI management of indexers, and monitoring replication.
High-level file-structure analysis
• Top‐level aggregator POM and submodules:
• hbase‐sep (API, impl, tools, demos) – SEP library for HBase replication.
• hbase‐indexer‐common – shared config, utilities, metrics.
• hbase‐indexer‐engine – core mapping & Solr writing logic.
• hbase‐indexer‐server – daemon process (Master, Supervisor, REST API).
• hbase‐indexer‐cli – command-line tooling to add/update/list indexers.
• hbase‐indexer‐mr – bulk MapReduce indexing job.
• hbase‐indexer‐model – data model for indexer definitions and state.
• hbase‐indexer‐dist – assembly and packaging scripts.
• hbase‐indexer‐demo, hbase‐sep‐demo – example apps.
Core README insights
• The Indexer runs as an HBase replication sink daemon.
• Configuration: XML files under conf/ define ZK, HBase, Solr endpoints and mapping from HBase columns → Solr fields.
• CLI uses a Java Main that registers indexer definitions via REST to the running daemon.
• Optional Kerberos security and Fusion pipeline output are supported.
Main components
a. External systems:
• HBase Cluster (RegionServers + ZK quorum)
• ZooKeeper ensemble (shared by HBase and Solr)
• Solr (or Fusion) Cluster
• Hadoop/YARN – for MapReduce jobs (bulk indexing)
b. SEP Replication Layer:
• hbase-sep-api + hbase-sep-impl on each HBase node
c. Indexer Daemon Cluster:
• hbase-indexer-server processes
• REST API endpoints for indexer management
• IndexerMaster and IndexerSupervisor modules
• BatchStateUpdater
d. Core Engine Modules:
• hbase-indexer-engine: parse events, build SolrInputDocuments, sharding, writer factories
• hbase-indexer-common: configuration loading, metrics, utilities
• hbase-indexer-model: in-memory/persistent model of indexer definitions
• hbase-indexer-cli: CLI commands (add, update, delete, list indexers)
e. Bulk-indexing Path:
• hbase-indexer-mr job that reads HBase tables → uses engine writer classes → writes to Solr
f. Packaging & Deployment:
• hbase-indexer-dist: assembly of scripts, config, and JARs
• bin/ and conf/ directory structure
Relationships and interactions
• Within daemon: events flow into Engine modules → map & transform → Solr writers → index documents
• Bulk MR jobs → standalone MapReduce cluster → use same Engine and Common libraries → write to Solr
• All components register metrics via MetricsUtil (common)
• ZK used by HBase, SEP, and Solr for coordination
Architectural patterns & design principles
• Modular multi‐project Maven structure for clear separation of concerns
• Plugin/factory pattern in Engine (IndexerComponentFactory) for pluggable parsers and writers
• Model‐View‐Controller style layering: CLI/REST (view), Master/Supervisor (controller), Engine/Model (model)
• Reusable common library for config, metrics, JSON etc.
• Assembly layer (dist) decouples packaging from code