# igormoraestech/big-data-ecommerce-grupo6 architecture

> Architecture diagram of the GitHub repository igormoraestech/big-data-ecommerce-grupo6, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/igormoraestech/big-data-ecommerce-grupo6
- Repository: https://github.com/igormoraestech/big-data-ecommerce-grupo6

## Overview

This repository implements real-time e-commerce trend detection alongside daily historical product analytics. In the streaming path, generated JSON events are ingested and split between raw HDFS storage and a file-based feed for Flink; Flink produces windowed alerts stored in HBase. In the batch path, Spark reads raw HDFS events, aggregates product metrics, writes curated Parquet data, and registers and queries it through Hive. The Flink excerpt is partial, so its documented processing parameters are represented without asserting unsampled implementation details.

Main entry points: [`flume/flume.conf`](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flume/flume.conf) (Flume Ingestion), [`spark/etl_historico.py`](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/spark/etl_historico.py) (Spark SQL Metrics), [`scripts/promote_stream_files.py`](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/scripts/promote_stream_files.py) (File Promoter), [`flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java`](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java) (Ready File Source)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation_ingest["Event Ingestion"]
  node_generator["Event Generator<br/>[gerador.py]"]
  node_flume["Flume Ingestion<br/>[flume.conf]"]
  node_file_roll["FileRoll Sink<br/>[flume.conf]"]
  node_promoter["File Promoter"]
end

subgraph group_realtime["Real-time Trends"]
  node_flink_source["Ready File Source"]
  node_flink_event["Event Parsing"]
  node_flink_window["Trend Windows"]
  node_flink_sink["HBase REST Sink"]
end

subgraph group_batch["Batch Analytics"]
  node_spark_rdd["RDD Aggregation<br/>[etl_historico.py]"]
  node_spark_sql["Spark SQL Metrics<br/>[etl_historico.py]"]
end

subgraph group_storage_query["Storage and Query"]
  node_hdfs_raw[("Raw HDFS")]
  node_hbase[("Trend Alerts<br/>[README.md]")]
  node_hdfs_curated[("Curated Parquet")]
  node_hive[("Hive Metrics Table<br/>[hive-site.xml]")]
end

node_generator -->|"streams events"| node_flume
node_flume -->|"writes raw events"| node_hdfs_raw
node_flume -->|"routes events"| node_file_roll
node_file_roll -->|"places files"| node_promoter
node_promoter -->|"moves finalized files"| node_flink_source
node_flink_source -->|"reads lines"| node_flink_event
node_flink_event -->|"filters and groups"| node_flink_window
node_flink_window -->|"sends alerts"| node_flink_sink
node_flink_sink -->|"writes via REST"| node_hbase
node_hdfs_raw -->|"reads history"| node_spark_rdd
node_spark_rdd -->|"creates metrics"| node_spark_sql
node_spark_sql -->|"writes Parquet"| node_hdfs_curated
node_spark_sql -->|"registers and queries"| node_hive
node_hive -->|"reads table data"| node_hdfs_curated

click node_generator "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/generator/gerador.py"
click node_flume "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flume/flume.conf"
click node_file_roll "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flume/flume.conf"
click node_promoter "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/scripts/promote_stream_files.py"
click node_flink_source "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java"
click node_flink_event "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java"
click node_flink_window "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java"
click node_flink_sink "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java"
click node_hbase "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/hbase/README.md"
click node_spark_rdd "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/spark/etl_historico.py"
click node_spark_sql "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/spark/etl_historico.py"
click node_hive "https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/hive/hive-site.xml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_flume,node_file_roll,node_promoter toneBlue
class node_flink_source,node_flink_event,node_flink_window,node_flink_sink toneAmber
class node_spark_rdd,node_spark_sql toneMint
class node_hdfs_raw,node_hbase,node_hdfs_curated,node_hive toneRose
```

## Components

### Event Ingestion

- [Event Generator](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/generator/gerador.py): `generator/gerador.py`
- [Flume Ingestion](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flume/flume.conf): `flume/flume.conf`
- [FileRoll Sink](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flume/flume.conf): `flume/flume.conf`
- [File Promoter](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/scripts/promote_stream_files.py): `scripts/promote_stream_files.py`

### Real-time Trends

- [Ready File Source](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java): `flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java`
- [Event Parsing](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java): `flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java`
- [Trend Windows](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java): `flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java`
- [HBase REST Sink](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java): `flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java`

### Batch Analytics

- [RDD Aggregation](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/spark/etl_historico.py): `spark/etl_historico.py`
- [Spark SQL Metrics](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/spark/etl_historico.py): `spark/etl_historico.py`

### Storage and Query

- **Raw HDFS**
- [Trend Alerts](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/hbase/README.md): `hbase/README.md`
- **Curated Parquet**
- [Hive Metrics Table](https://github.com/igormoraestech/big-data-ecommerce-grupo6/blob/main/hive/hive-site.xml): `hive/hive-site.xml`

## Connections

- Event Generator → Flume Ingestion: streams events
- Flume Ingestion → Raw HDFS: writes raw events
- Flume Ingestion → FileRoll Sink: routes events
- FileRoll Sink → File Promoter: places files
- File Promoter → Ready File Source: moves finalized files
- Ready File Source → Event Parsing: reads lines
- Event Parsing → Trend Windows: filters and groups
- Trend Windows → HBase REST Sink: sends alerts
- HBase REST Sink → Trend Alerts: writes via REST
- Raw HDFS → RDD Aggregation: reads history
- RDD Aggregation → Spark SQL Metrics: creates metrics
- Spark SQL Metrics → Curated Parquet: writes Parquet
- Spark SQL Metrics → Hive Metrics Table: registers and queries
- Hive Metrics Table → Curated Parquet: reads table data