How huynhtloi/mining-of-massive-datasets works
Main Components
• Big Data Processing Framework – Represent Apache Spark at the center since all sub-projects (e.g., Apriori, LSH, Recommendations) are built on Spark techniques.
• Data Ingestion/Data Sources – Include the CSV, TXT, and other data files found in each project (e.g., store_data.csv, movies.csv, ratings.csv, bank.csv).
• Processing and Algorithms – Identify distinct jobs:
• Apriori Algorithm component (both notebook and Python script versions)
• LSH component (with both PySpark and standard scripts)
• Recommendation Systems (Books and Movies): Show that these modules leverage different filtering techniques (Collaborative, Content-based, Demographic)
• Spark DataFrame, Spark SQL, Spark RDD, and Spark With Machine Learning modules, each of which showcases a specific subset of Spark functionality.
• Execution Environment – Represent notebooks (Jupyter) and Python scripts as “client interfaces” for job submission.
• External Tools – Optionally, outline any auxiliary files like report.docx and other documents that document findings or results.
Out Relationships and Interactions
• From the Data Sources to the Spark processing layer (data ingestion stage).
• Then from Spark’s processing modules to the output layers (possibly saving results or creating recommendations).
• That various modules (e.g., Spark RDD, DataFrame, SQL) are sub-components or modules within Spark, interacting and sometimes overlapping in functionality.
Architectural Patterns and Design Principles
• Decomposition by functionality: Each sub-directory (e.g., Spark - Apriori Algorithm, Spark - LSH, etc.) is treated as its own component within a broader Spark ecosystem.
• Modular design: Emphasize that each algorithm or processing technique is self-contained but uses shared resources (data files, common Spark infrastructure).
• Data flow architecture: Clearly represent the flow from raw data ingestion to transformation (via Spark jobs) to final outputs (results, reports, or recommendation outputs).
Relevant Technologies, Frameworks, or Libraries
• Apache Spark (and its sub-libraries like MLlib for machine learning)
• Python (scripts and Jupyter notebooks)
• Data Storage formats (CSV, TXT)
• MapReduce (as a fundamental model supported by some exercises)
• Any libraries or packages implicitly relied on by these projects (e.g., libraries for using Spark DataFrame or SQL operations)