How machtermassi06/hbase-mapreduce-learning-project works
This is an educational set of HBase, Hadoop MapReduce, and Spark usage cases, not a single service. The main workflow generates or loads website user/visit data, stores it in HBase, and runs alternative aggregation and enrichment jobs; Spark adds connector and thin-bulk-load paths plus table reading. The graph retains documented workflows whose implementation was not sampled. Edges are limited to README statements and shown source references/calls; uninspected job wiring is not inferred.
Main entry points: bulk_load.py Thin bulk load · bulk_load MapReduce bulk load · classic_load.py Connector write · read_Hbase_table.py HBase table reader
Components 24
Data preparation
- CSV generators
wsite_visits_gen - Users and visits CSV
data - HDFS
HBase storage
- Table setup
hbase/create_tables.sh - HBase tables
MapReduce jobs
- MapReduce bulk load
mapreduce/src/main/java/mapreduce/bulk_load - Users CSV mapper
mapreduce/src/main/java/mapreduce/bulk_load/UsersBulkLoadMapper.java - Visits CSV mapper
mapreduce/src/main/java/mapreduce/bulk_load/VisitsBulkLoadMapper.java - Classic MR aggregation
mapreduce/src/main/java/mapreduce/nb_visits_by_column_ClassicMR - HBase table aggregation
mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR - Table aggregation driver
mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java - HBase output reducer
mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/JobReducerHbase.java - Visits-user join
mapreduce/src/main/java/mapreduce/visits_users_join - Join mapper
mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java - User profile
mapreduce/src/main/java/mapreduce/visits_users_join/UserProfile.java - Table utilities
mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java
Spark workflows
- Spark session setup
spark/spark_utils/session.py - Connector write
spark/classic_load.py - Thin bulk load
spark/bulk_load.py - Thin-load wrapper
mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java - Thin row mapper
mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadMapper.java - HBase table reader
spark/read_Hbase_table.py - HBase catalogs
spark/catalogs
Other components
- Project operator
Connections 37
- Project operator → CSV generators: runs
README.md - CSV generators → Users and visits CSV: generates
README.md - Project operator → Table setup: creates tables
README.md - Table setup → HBase tables: creates
README.md - Project operator → Users and visits CSV: loads data
README.md - Users and visits CSV → HDFS: transfers to
README.md - Project operator → MapReduce bulk load: runs
README.md - MapReduce bulk load → Users CSV mapper: dispatches
mapreduce/src/main/java/mapreduce/bulk_load/BulkLoadDriver.java - MapReduce bulk load → Visits CSV mapper: dispatches
mapreduce/src/main/java/mapreduce/bulk_load/BulkLoadDriver.java - MapReduce bulk load → HDFS: writes HFiles
README.md - Project operator → HBase tables: loads HFiles
README.md - Project operator → Classic MR aggregation: runs
README.md - Classic MR aggregation → HDFS: reads and writes
README.md - Project operator → HBase table aggregation: runs
README.md - Table aggregation driver → HBase output reducer: configures output
mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java - HBase table aggregation → HBase tables: scans visits
README.md - HBase table aggregation → HDFS: writes results
README.md - HBase output reducer → HBase tables: writes statistics
README.md - Project operator → Visits-user join: runs
README.md - Visits-user join → HBase tables: reads users and visits
mapreduce/src/main/java/mapreduce/visits_users_join/JoinDriver.java - Join mapper → User profile: constructs
mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java - Join mapper → HBase tables: reads users; writes join
mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java - Table utilities → HBase tables: creates tables
mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java - Project operator → Connector write: runs
README.md - Connector write → Spark session setup: creates session
spark/classic_load.py - Connector write → HBase catalogs: reads catalog
spark/classic_load.py - Connector write → HBase tables: writes rows
spark/classic_load.py - Project operator → Thin bulk load: runs
README.md - Thin bulk load → Spark session setup: creates session
spark/bulk_load.py - Thin bulk load → HBase catalogs: reads catalog
spark/bulk_load.py - Thin bulk load → Thin-load wrapper: invokes
spark/bulk_load.py - Thin-load wrapper → Thin row mapper: maps rows
mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java - Thin bulk load → HBase tables: loads HFiles
spark/bulk_load.py - Project operator → HBase table reader: runs
README.md - HBase table reader → Spark session setup: creates session
spark/read_Hbase_table.py - HBase table reader → HBase catalogs: reads catalog
spark/read_Hbase_table.py - HBase table reader → HBase tables: reads table
spark/read_Hbase_table.py