How igormoraestech/big-data-ecommerce-grupo6 works
This repository implements real-time e-commerce trend detection alongside daily historical product analytics. In the streaming path, generated JSON events are ingested and split between raw HDFS storage and a file-based feed for Flink; Flink produces windowed alerts stored in HBase. In the batch path, Spark reads raw HDFS events, aggregates product metrics, writes curated Parquet data, and registers and queries it through Hive. The Flink excerpt is partial, so its documented processing parameters are represented without asserting unsampled implementation details.
Main entry points: flume.conf Flume Ingestion · etl_historico.py Spark SQL Metrics · promote_stream_files.py File Promoter · EcommerceTrendJob.java Ready File Source
Components 14
Event Ingestion
- Event Generator
generator/gerador.py - Flume Ingestion
flume/flume.conf - FileRoll Sink
flume/flume.conf - File Promoter
scripts/promote_stream_files.py
Real-time Trends
- Ready File Source
flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java - Event Parsing
flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java - Trend Windows
flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java - HBase REST Sink
flink/src/main/java/br/com/ecommerce/EcommerceTrendJob.java
Batch Analytics
- RDD Aggregation
spark/etl_historico.py - Spark SQL Metrics
spark/etl_historico.py
Storage and Query
- Raw HDFS
- Trend Alerts
hbase/README.md - Curated Parquet
- Hive Metrics Table
hive/hive-site.xml
Connections 14
- Event Generator → Flume Ingestion: streams events
- Flume Ingestion → Raw HDFS: writes raw events
- Flume Ingestion → FileRoll Sink: routes events
- FileRoll Sink → File Promoter: places files
- File Promoter → Ready File Source: moves finalized files
- Ready File Source → Event Parsing: reads lines
- Event Parsing → Trend Windows: filters and groups
- Trend Windows → HBase REST Sink: sends alerts
- HBase REST Sink → Trend Alerts: writes via REST
- Raw HDFS → RDD Aggregation: reads history
- RDD Aggregation → Spark SQL Metrics: creates metrics
- Spark SQL Metrics → Curated Parquet: writes Parquet
- Spark SQL Metrics → Hive Metrics Table: registers and queries
- Hive Metrics Table → Curated Parquet: reads table data