# huynhtloi/mining-of-massive-datasets architecture

> Architecture diagram of the GitHub repository huynhtloi/mining-of-massive-datasets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huynhtloi/mining-of-massive-datasets
- Repository: https://github.com/huynhtloi/mining-of-massive-datasets

## Overview

##### Main Components
- Big Data Processing Framework – Represent Apache Spark at the center since all sub-projects (e.g., Apriori, LSH, Recommendations) are built on Spark techniques.
- Data Ingestion/Data Sources – Include the CSV, TXT, and other data files found in each project (e.g., store_data.csv, movies.csv, ratings.csv, bank.csv).
- Processing and Algorithms – Identify distinct jobs:
- Apriori Algorithm component (both notebook and Python script versions)
- LSH component (with both PySpark and standard scripts)
- Recommendation Systems (Books and Movies): Show that these modules leverage different filtering techniques (Collaborative, Content-based, Demographic)
- Spark DataFrame, Spark SQL, Spark RDD, and Spark With Machine Learning modules, each of which showcases a specific subset of Spark functionality.
- Execution Environment – Represent notebooks (Jupyter) and Python scripts as “client interfaces” for job submission.
- External Tools – Optionally, outline any auxiliary files like report.docx and other documents that document findings or results.

##### Out Relationships and Interactions
- From the Data Sources to the Spark processing layer (data ingestion stage).
- Then from Spark’s processing modules to the output layers (possibly saving results or creating recommendations).
- That various modules (e.g., Spark RDD, DataFrame, SQL) are sub-components or modules within Spark, interacting and sometimes overlapping in functionality.

##### Architectural Patterns and Design Principles
- Decomposition by functionality: Each sub-directory (e.g., Spark - Apriori Algorithm, Spark - LSH, etc.) is treated as its own component within a broader Spark ecosystem.
- Modular design: Emphasize that each algorithm or processing technique is self-contained but uses shared resources (data files, common Spark infrastructure).
- Data flow architecture: Clearly represent the flow from raw data ingestion to transformation (via Spark jobs) to final outputs (results, reports, or recommendation outputs).

##### Relevant Technologies, Frameworks, or Libraries
- Apache Spark (and its sub-libraries like MLlib for machine learning)
- Python (scripts and Jupyter notebooks)
- Data Storage formats (CSV, TXT)
- MapReduce (as a fundamental model supported by some exercises)
- Any libraries or packages implicitly relied on by these projects (e.g., libraries for using Spark DataFrame or SQL operations)

## Diagram (Mermaid)

```mermaid
flowchart TD
    dataSources["Data Ingestion - CSV/TXT Data Sources"]:::data
    clientUI["Client Interfaces - Notebooks & Scripts"]:::client
    sparkCluster["Apache Spark Cluster"]:::spark

    subgraph "Processing Modules"
        apriori["Apriori Algorithm"]:::processing
        lsh["LSH (Locality-Sensitive Hashing)"]:::processing
        booksRec["Books Recommendation System"]:::processing
        moviesRec["Movies Recommendation System"]:::processing
        sparkDF["Spark DataFrame Module"]:::processing
        sparkRDD["Spark RDD Module"]:::processing
        sparkSQL["Spark SQL Module"]:::processing
        sparkML["Spark With Machine Learning"]:::processing
        sparkMR["Spark and MapReduce"]:::processing
    end

    subgraph "External Documentation"
        lshReport["LSH Report"]:::external
        moviesReport["Movies Rec Report"]:::external
    end

    dataSources --> sparkCluster
    clientUI --> sparkCluster

    sparkCluster --> apriori
    sparkCluster --> lsh
    sparkCluster --> booksRec
    sparkCluster --> moviesRec
    sparkCluster --> sparkDF
    sparkCluster --> sparkRDD
    sparkCluster --> sparkSQL
    sparkCluster --> sparkML
    sparkCluster --> sparkMR

    lsh --> lshReport
    moviesRec --> moviesReport

    click apriori "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark - Apriori Algorithm"
    click lsh "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark - LSH (Locality-Sensitive Hashing)"
    click booksRec "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark - Books Recommendation System"
    click moviesRec "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark - Movies Recommendation System"
    click sparkDF "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark Dataframe"
    click sparkRDD "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark RDD"
    click sparkSQL "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark SQL"
    click sparkML "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark With Machine Learning"
    click sparkMR "https://github.com/huynhtloi/mining-of-massive-datasets/tree/main/Spark and MapReduce"
    click lshReport "https://github.com/huynhtloi/mining-of-massive-datasets/blob/main/Spark - LSH (Locality-Sensitive Hashing)/report.docx"
    click moviesReport "https://github.com/huynhtloi/mining-of-massive-datasets/blob/main/Spark - Movies Recommendation System/51800590_51800574_51800900_RecomendationSystem.docx"

    classDef data fill:#CEF6F5,stroke:#1f77b4,stroke-width:2px;
    classDef spark fill:#F5F1A4,stroke:#FF5733,stroke-width:2px;
    classDef processing fill:#C1F0C1,stroke:#2E8B57,stroke-width:2px;
    classDef client fill:#FFDAB9,stroke:#FF8C00,stroke-width:2px;
    classDef external fill:#FADBD8,stroke:#C0392B,stroke-width:2px;
```
