# machtermassi06/hbase-mapreduce-learning-project architecture

> Architecture diagram of the GitHub repository machtermassi06/hbase-mapreduce-learning-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/machtermassi06/hbase-mapreduce-learning-project
- Repository: https://github.com/machtermassi06/hbase-mapreduce-learning-project

## Overview

This is an educational set of HBase, Hadoop MapReduce, and Spark usage cases, not a single service. The main workflow generates or loads website user/visit data, stores it in HBase, and runs alternative aggregation and enrichment jobs; Spark adds connector and thin-bulk-load paths plus table reading. The graph retains documented workflows whose implementation was not sampled. Edges are limited to README statements and shown source references/calls; uninspected job wiring is not inferred.

Main entry points: [`spark/bulk_load.py`](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/bulk_load.py) (Thin bulk load), [`mapreduce/src/main/java/mapreduce/bulk_load`](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/bulk_load) (MapReduce bulk load), [`spark/classic_load.py`](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/classic_load.py) (Connector write), [`spark/read_Hbase_table.py`](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/read_Hbase_table.py) (HBase table reader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_generators["CSV generators"]
  node_csvdata["Users and visits CSV"]
  node_hdfs[("HDFS")]
end

subgraph group_hbase["HBase storage"]
  node_table_setup["Table setup<br/>[create_tables.sh]"]
  node_hbase_store[("HBase tables")]
end

subgraph group_mr["MapReduce jobs"]
  node_mr_bulk["MapReduce bulk load"]
  node_users_mapper["Users CSV mapper"]
  node_visits_mapper["Visits CSV mapper"]
  node_classic_agg["Classic MR aggregation"]
  node_table_agg["HBase table aggregation"]
  node_table_driver["Table aggregation driver<br/>[Driver.java]"]
  node_hbase_reducer["HBase output reducer"]
  node_visit_join["Visits-user join"]
  node_join_mapper["Join mapper<br/>[JoinMapper.java]"]
  node_user_profile["User profile<br/>[UserProfile.java]"]
  node_table_utils["Table utilities"]
end

subgraph group_spark["Spark workflows"]
  node_spark_session["Spark session setup<br/>[session.py]"]
  node_spark_classic["Connector write<br/>[classic_load.py]"]
  node_spark_thin["Thin bulk load<br/>[bulk_load.py]"]
  node_thin_wrapper["Thin-load wrapper"]
  node_thin_mapper["Thin row mapper"]
  node_spark_reader["HBase table reader"]
  node_catalogs["HBase catalogs"]
end

node_operator(("Project operator"))

node_operator -->|"runs"| node_generators
node_generators -->|"generates"| node_csvdata
node_operator -->|"creates tables"| node_table_setup
node_table_setup -->|"creates"| node_hbase_store
node_operator -->|"loads data"| node_csvdata
node_csvdata -->|"transfers to"| node_hdfs
node_operator -->|"runs"| node_mr_bulk
node_mr_bulk -->|"dispatches"| node_users_mapper
node_mr_bulk -->|"dispatches"| node_visits_mapper
node_mr_bulk -->|"writes HFiles"| node_hdfs
node_operator -->|"loads HFiles"| node_hbase_store
node_operator -->|"runs"| node_classic_agg
node_classic_agg -->|"reads and writes"| node_hdfs
node_operator -->|"runs"| node_table_agg
node_table_driver -->|"configures output"| node_hbase_reducer
node_table_agg -->|"scans visits"| node_hbase_store
node_table_agg -->|"writes results"| node_hdfs
node_hbase_reducer -->|"writes statistics"| node_hbase_store
node_operator -->|"runs"| node_visit_join
node_visit_join -->|"reads users and visits"| node_hbase_store
node_join_mapper -->|"constructs"| node_user_profile
node_join_mapper -->|"reads users; writes join"| node_hbase_store
node_table_utils -->|"creates tables"| node_hbase_store
node_operator -->|"runs"| node_spark_classic
node_spark_classic -->|"creates session"| node_spark_session
node_spark_classic -->|"reads catalog"| node_catalogs
node_spark_classic -->|"writes rows"| node_hbase_store
node_operator -->|"runs"| node_spark_thin
node_spark_thin -->|"creates session"| node_spark_session
node_spark_thin -->|"reads catalog"| node_catalogs
node_spark_thin -->|"invokes"| node_thin_wrapper
node_thin_wrapper -->|"maps rows"| node_thin_mapper
node_spark_thin -->|"loads HFiles"| node_hbase_store
node_operator -->|"runs"| node_spark_reader
node_spark_reader -->|"creates session"| node_spark_session
node_spark_reader -->|"reads catalog"| node_catalogs
node_spark_reader -->|"reads table"| node_hbase_store

click node_generators "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/wsite_visits_gen"
click node_csvdata "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/data"
click node_table_setup "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/hbase/create_tables.sh"
click node_mr_bulk "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/bulk_load"
click node_users_mapper "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/bulk_load/UsersBulkLoadMapper.java"
click node_visits_mapper "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/bulk_load/VisitsBulkLoadMapper.java"
click node_classic_agg "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_ClassicMR"
click node_table_agg "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR"
click node_table_driver "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java"
click node_hbase_reducer "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/JobReducerHbase.java"
click node_visit_join "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/visits_users_join"
click node_join_mapper "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java"
click node_user_profile "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/visits_users_join/UserProfile.java"
click node_table_utils "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java"
click node_spark_session "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/spark_utils/session.py"
click node_spark_classic "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/classic_load.py"
click node_spark_thin "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/bulk_load.py"
click node_thin_wrapper "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java"
click node_thin_mapper "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadMapper.java"
click node_spark_reader "https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/read_Hbase_table.py"
click node_catalogs "https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/spark/catalogs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generators,node_csvdata,node_hdfs toneBlue
class node_table_setup,node_hbase_store toneAmber
class node_mr_bulk,node_users_mapper,node_visits_mapper,node_classic_agg,node_table_agg,node_table_driver,node_hbase_reducer,node_visit_join,node_join_mapper,node_user_profile,node_table_utils toneMint
class node_spark_session,node_spark_classic,node_spark_thin,node_thin_wrapper,node_thin_mapper,node_spark_reader,node_catalogs toneRose
class node_operator toneIndigo
```

## Components

### Data preparation

- [CSV generators](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/wsite_visits_gen): `wsite_visits_gen`
- [Users and visits CSV](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/data): `data`
- **HDFS**

### HBase storage

- [Table setup](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/hbase/create_tables.sh): `hbase/create_tables.sh`
- **HBase tables**

### MapReduce jobs

- [MapReduce bulk load](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/bulk_load): `mapreduce/src/main/java/mapreduce/bulk_load`
- [Users CSV mapper](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/bulk_load/UsersBulkLoadMapper.java): `mapreduce/src/main/java/mapreduce/bulk_load/UsersBulkLoadMapper.java`
- [Visits CSV mapper](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/bulk_load/VisitsBulkLoadMapper.java): `mapreduce/src/main/java/mapreduce/bulk_load/VisitsBulkLoadMapper.java`
- [Classic MR aggregation](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_ClassicMR): `mapreduce/src/main/java/mapreduce/nb_visits_by_column_ClassicMR`
- [HBase table aggregation](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR): `mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR`
- [Table aggregation driver](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java): `mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java`
- [HBase output reducer](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/JobReducerHbase.java): `mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/JobReducerHbase.java`
- [Visits-user join](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/mapreduce/src/main/java/mapreduce/visits_users_join): `mapreduce/src/main/java/mapreduce/visits_users_join`
- [Join mapper](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java): `mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java`
- [User profile](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/visits_users_join/UserProfile.java): `mapreduce/src/main/java/mapreduce/visits_users_join/UserProfile.java`
- [Table utilities](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java): `mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java`

### Spark workflows

- [Spark session setup](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/spark_utils/session.py): `spark/spark_utils/session.py`
- [Connector write](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/classic_load.py): `spark/classic_load.py`
- [Thin bulk load](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/bulk_load.py): `spark/bulk_load.py`
- [Thin-load wrapper](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java): `mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java`
- [Thin row mapper](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadMapper.java): `mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadMapper.java`
- [HBase table reader](https://github.com/machtermassi06/hbase-mapreduce-learning-project/blob/main/spark/read_Hbase_table.py): `spark/read_Hbase_table.py`
- [HBase catalogs](https://github.com/machtermassi06/hbase-mapreduce-learning-project/tree/main/spark/catalogs): `spark/catalogs`

### Other components

- **Project operator**

## Connections

- Project operator → CSV generators: runs (evidence: `README.md`)
- CSV generators → Users and visits CSV: generates (evidence: `README.md`)
- Project operator → Table setup: creates tables (evidence: `README.md`)
- Table setup → HBase tables: creates (evidence: `README.md`)
- Project operator → Users and visits CSV: loads data (evidence: `README.md`)
- Users and visits CSV → HDFS: transfers to (evidence: `README.md`)
- Project operator → MapReduce bulk load: runs (evidence: `README.md`)
- MapReduce bulk load → Users CSV mapper: dispatches (evidence: `mapreduce/src/main/java/mapreduce/bulk_load/BulkLoadDriver.java`)
- MapReduce bulk load → Visits CSV mapper: dispatches (evidence: `mapreduce/src/main/java/mapreduce/bulk_load/BulkLoadDriver.java`)
- MapReduce bulk load → HDFS: writes HFiles (evidence: `README.md`)
- Project operator → HBase tables: loads HFiles (evidence: `README.md`)
- Project operator → Classic MR aggregation: runs (evidence: `README.md`)
- Classic MR aggregation → HDFS: reads and writes (evidence: `README.md`)
- Project operator → HBase table aggregation: runs (evidence: `README.md`)
- Table aggregation driver → HBase output reducer: configures output (evidence: `mapreduce/src/main/java/mapreduce/nb_visits_by_column_TableMR/Driver.java`)
- HBase table aggregation → HBase tables: scans visits (evidence: `README.md`)
- HBase table aggregation → HDFS: writes results (evidence: `README.md`)
- HBase output reducer → HBase tables: writes statistics (evidence: `README.md`)
- Project operator → Visits-user join: runs (evidence: `README.md`)
- Visits-user join → HBase tables: reads users and visits (evidence: `mapreduce/src/main/java/mapreduce/visits_users_join/JoinDriver.java`)
- Join mapper → User profile: constructs (evidence: `mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java`)
- Join mapper → HBase tables: reads users; writes join (evidence: `mapreduce/src/main/java/mapreduce/visits_users_join/JoinMapper.java`)
- Table utilities → HBase tables: creates tables (evidence: `mapreduce/src/main/java/mapreduce/hbase_table_utils/HBaseTableUtils.java`)
- Project operator → Connector write: runs (evidence: `README.md`)
- Connector write → Spark session setup: creates session (evidence: `spark/classic_load.py`)
- Connector write → HBase catalogs: reads catalog (evidence: `spark/classic_load.py`)
- Connector write → HBase tables: writes rows (evidence: `spark/classic_load.py`)
- Project operator → Thin bulk load: runs (evidence: `README.md`)
- Thin bulk load → Spark session setup: creates session (evidence: `spark/bulk_load.py`)
- Thin bulk load → HBase catalogs: reads catalog (evidence: `spark/bulk_load.py`)
- Thin bulk load → Thin-load wrapper: invokes (evidence: `spark/bulk_load.py`)
- Thin-load wrapper → Thin row mapper: maps rows (evidence: `mapreduce/src/main/java/mapreduce/hbase/ThinBulkLoadWrapper.java`)
- Thin bulk load → HBase tables: loads HFiles (evidence: `spark/bulk_load.py`)
- Project operator → HBase table reader: runs (evidence: `README.md`)
- HBase table reader → Spark session setup: creates session (evidence: `spark/read_Hbase_table.py`)
- HBase table reader → HBase catalogs: reads catalog (evidence: `spark/read_Hbase_table.py`)
- HBase table reader → HBase tables: reads table (evidence: `spark/read_Hbase_table.py`)
