# doubong/customersegmentation-recommendationsystem architecture

> Architecture diagram of the GitHub repository doubong/customersegmentation-recommendationsystem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/doubong/customersegmentation-recommendationsystem
- Repository: https://github.com/doubong/customersegmentation-recommendationsystem

## Overview

##### File Structure
- Top‐level folders correspond to distinct pipeline stages:
– Data Preprocessing: Jupyter notebooks and CSVs for cleaning raw data
– HadoopStreaming: Python Mapper.py and Reducer.py using Hadoop Streaming
– Mapreduce Work: Java/Maven-based MapReduce (CustomerMapper, CustomerReducer, Driver)
– K-Means: Jupyter notebook applying clustering on aggregated output
- Deployment/config: pom.xml for Java build, target/*.jar for execution, hadoop-local result docs.

##### Extract Key Insights from README
- “Phát triển một hệ thống … bằng Hadoop trên Microsoft Azure.”

##### Main Components & Their Relationships
a. Data Sources & Storage
– Raw transaction CSV (“Online Retail.csv”) stored in Azure Blob Storage or Data Lake.
b. Data Preprocessing
– Jupyter notebooks (Python/Pandas) read raw CSV, produce cleaned_retail_data.csv.
– Output written back to Blob Storage (or HDFS on HDInsight).
c. MapReduce Layer
– Hadoop Streaming job (Mapper.py, Reducer.py) runs on HDInsight, produces intermediate aggregates (result_mapreduce.csv).
– Java MapReduce (App.java, CustomerMapper/CustomerReducer) processes cleaned data in a more scalable/prod manner.
– Both write outputs to HDFS/Azure Blob.
d. Clustering
– K-Means notebook loads MapReduce output, uses scikit-learn to assign customers to clusters, writes clustered_customers_done.csv.
e. Visualization & Reporting
– Power BI dashboards (Customer_Segmentation_Dashboard.pbix) connect to clustered output (via direct file or Azure SQL/Data Lake).
– Final PDF/PBIX for stakeholders.
f. (Optional) Recommendation Engine
– Future component that consumes clusters to generate personalized suggestions.

##### Architectural Patterns & Design Principles
- Batch data‐processing pipeline on Azure/HDInsight
- Modularity: each stage isolated (cleaning, streaming, Java MR, clustering, viz)
- Separation of concerns: compute jobs vs. notebooks vs. dashboards
- Scalability: Hadoop cluster for large-scale MapReduce
- Extensibility: can add new ML stages (recommendation) or swap Java for Spark.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Stage 1: Data Ingestion & Preprocessing
    RawDataBlob["Raw Transaction CSV\n(Online Retail.csv)"]:::storage
    click RawDataBlob "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Data Preprocessing/Online Retail.csv"
    PreprocessNotebook["Data Preprocessing Notebook\n(Data-preprocessing.ipynb)"]:::compute
    click PreprocessNotebook "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Data Preprocessing/Data-preprocessing.ipynb"
    CleanedCSV["Cleaned Retail Data CSV\n(cleaned_retail_data.csv)"]:::storage
    click CleanedCSV "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Data Preprocessing/cleaned_retail_data.csv"

    RawDataBlob --> PreprocessNotebook --> CleanedCSV

    %% Stage 2 & 3: Hadoop Jobs on Azure HDInsight
    subgraph "Azure HDInsight" 
        direction TB
        subgraph "Hadoop Streaming Layer"
            direction TB
            Mapper["Mapper.py"]:::compute
            click Mapper "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/HadoopStreaming/Mapper.py"
            Reducer["Reducer.py"]:::compute
            click Reducer "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/HadoopStreaming/Reducer.py"
        end
        subgraph "Java MapReduce Layer"
            direction TB
            Pom["pom.xml"]:::compute
            click Pom "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/pom.xml"
            App["App.java\n(Driver)"]:::compute
            click App "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/src/main/java/com/customeranalysis/App.java"
            CustomerMapper["CustomerMapper.java"]:::compute
            click CustomerMapper "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/src/main/java/com/customeranalysis/CustomerMapper.java"
            CustomerReducer["CustomerReducer.java"]:::compute
            click CustomerReducer "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/src/main/java/com/customeranalysis/CustomerReducer.java"
            Driver["Driver.java"]:::compute
            click Driver "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/src/main/java/com/customeranalysis/Driver.java"
            Jar["customeranalysis-1.0-SNAPSHOT.jar"]:::compute
            click Jar "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce Work/target/customeranalysis-1.0-SNAPSHOT.jar"
        end
    end

    CleanedCSV --> Mapper --> Reducer --> ResultStreaming["result_mapreduce.csv"]:::storage
    click ResultStreaming "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/K-Means/result_mapreduce.csv"
    CleanedCSV --> Pom --> App --> CustomerMapper --> CustomerReducer --> Driver --> Jar --> PartFile["part-r-00000.txt"]:::storage
    click PartFile "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/K-Means/part-r-00000.txt"

    %% Stage 4: Clustering
    KMeansNotebook["K-Means Clustering Notebook\n(kmean-code.ipynb)"]:::compute
    click KMeansNotebook "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/K-Means/kmean-code.ipynb"
    ClusteredData["Clustered Customers CSV\n(clustered_customers_done.csv)"]:::storage
    click ClusteredData "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/K-Means/clustered_customers_done.csv"

    ResultStreaming --> KMeansNotebook
    PartFile --> KMeansNotebook
    KMeansNotebook --> ClusteredData

    %% Stage 5: Visualization & Reporting
    subgraph "Visualization & Reporting"
        direction TB
        PowerBI["Power BI Dashboard\n(Customer_Segmentation_Dashboard.pbix)"]:::viz
        click PowerBI "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Visualize/Customer_Segmentation_Dashboard.pbix"
        PBIPDF["Dashboard PDF\n(Customer_Segmentation_Dashboard.pdf)"]:::viz
        click PBIPDF "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Visualize/Customer_Segmentation_Dashboard.pdf"
        AltPBIX["Alternative PBIX\n(Result-visualize.pbix)"]:::viz
        click AltPBIX "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Visualize/Result-visualize.pbix"
        ReportLinks["Report Links.txt"]:::storage
        click ReportLinks "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Report/Report Links.txt"
    end

    ClusteredData --> PowerBI --> PBIPDF
    ClusteredData --> AltPBIX
    ClusteredData --> ReportLinks

    %% Stage 6: Future Extension
    FutureRec["Future: Recommendation Engine"]:::future
    ClusteredData --> FutureRec

    %% Project Docs
    README["README.md"]:::storage
    click README "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/README.md"
    MRDoc["MapReduce Result Doc"]:::storage
    click MRDoc "https://github.com/doubong/customersegmentation-recommendationsystem/blob/main/Mapreduce-result/hadoop-local/kết quả mapreduce chạy trên hadoop sau 3 lần chạy.docx"

    %% Legend
    subgraph Legend
        direction LR
        S[Storage]:::storage
        C[Compute]:::compute
        V[Visualization]:::viz
        F[Future]:::future
    end

    classDef storage fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef compute fill:#C8E6C9,stroke:#43A047,color:#1B5E20;
    classDef viz fill:#FFE0B2,stroke:#FB8C00,color:#E65100;
    classDef future fill:#E1BEE7,stroke:#8E24AA,color:#4A148C;
```
