How sickmazin/bigdata_hotelreview_analysis works
Main Components of the System
• Data Source & Ingestion:
• Dataset (from Kaggle/Booking.com) as the starting point of the system.
• An element that reads and loads the raw review data.
• Data Preprocessing:
• Preprocessing steps done in the preprocessing.py script. This includes cleaning operations, date conversions, missing value handling, and saving data in a parquet format.
• Data Processing & Analysis:
• PySpark-based pipeline which is the core engine for processing the large dataset. Link this to files such as best_model.py and sentiment_analysis.py that implement the analytical models (sentiment analysis, clustering, etc.).
• Use of external libraries (PySpark, pandas) that support these operations.
• Dashboard / Interactive Visualization:
• UI component built with Streamlit. This includes the Homepage.py and the pages folder which contain several specific dashboard views (like "Filtro Recensioni", "Query", "Recensioni Controverse", "Sentiment Analysis").
• External Dependencies & Environment:
• Optionally, include a block for the dependencies listed (pyspark, streamlit, folium, etc.) and configuration files such as requirements.txt and project-specific files (.idea, *.iml).
Architectural Patterns and Design Principles
• Separation of concerns: data ingestion/processing is decoupled from the user interface.
• Modular design with distinct components for data handling (preprocessing and analysis scripts) and for UI (homepage and page-specific scripts).
• That a microservice-like separation is mimicked in the codebase, even if all modules reside in a single code repository.
Specific Technologies and Libraries
• Clearly label usage of Apache Spark (via PySpark) for big data processing.
• Use of Streamlit as the primary front-end framework for creating interactive dashboards.
• Additional visualization and data processing libraries (pandas, folium, wordcloud, matplotlib, plotly) and explain their roles in supporting the project’s functionalities.
Elements
• Clear labels for each component (Data Source, Preprocessing, Data Processing, Dashboard, etc.).
• Annotation or legends explaining the role of each component and how they interact.