# sickmazin/bigdata_hotelreview_analysis architecture

> Architecture diagram of the GitHub repository sickmazin/bigdata_hotelreview_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sickmazin/bigdata_hotelreview_analysis
- Repository: https://github.com/sickmazin/bigdata_hotelreview_analysis

## Overview

##### Main Components of the System
- Data Source & Ingestion:
- Dataset (from Kaggle/Booking.com) as the starting point of the system.
- An element that reads and loads the raw review data.
- Data Preprocessing:
- Preprocessing steps done in the preprocessing.py script. This includes cleaning operations, date conversions, missing value handling, and saving data in a parquet format.
- Data Processing & Analysis:
- PySpark-based pipeline which is the core engine for processing the large dataset. Link this to files such as best_model.py and sentiment_analysis.py that implement the analytical models (sentiment analysis, clustering, etc.).
- Use of external libraries (PySpark, pandas) that support these operations.
- Dashboard / Interactive Visualization:
- UI component built with Streamlit. This includes the Homepage.py and the pages folder which contain several specific dashboard views (like "Filtro Recensioni", "Query", "Recensioni Controverse", "Sentiment Analysis").
- External Dependencies & Environment:
- Optionally, include a block for the dependencies listed (pyspark, streamlit, folium, etc.) and configuration files such as requirements.txt and project-specific files (.idea, *.iml).

##### Architectural Patterns and Design Principles
- Separation of concerns: data ingestion/processing is decoupled from the user interface.
- Modular design with distinct components for data handling (preprocessing and analysis scripts) and for UI (homepage and page-specific scripts).
- That a microservice-like separation is mimicked in the codebase, even if all modules reside in a single code repository.

##### Specific Technologies and Libraries
- Clearly label usage of Apache Spark (via PySpark) for big data processing.
- Use of Streamlit as the primary front-end framework for creating interactive dashboards.
- Additional visualization and data processing libraries (pandas, folium, wordcloud, matplotlib, plotly) and explain their roles in supporting the project’s functionalities.

##### Elements
- Clear labels for each component (Data Source, Preprocessing, Data Processing, Dashboard, etc.).
- Annotation or legends explaining the role of each component and how they interact.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer: Data Source
    A["Raw Dataset (Booking.com Reviews)"]:::data

    %% Layer: Data Preprocessing
    B["Data Preprocessing"]:::backend
    click B "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/preprocessing.py"

    %% Layer: Data Processing & Analysis
    subgraph "Data Processing & Analysis"
        C["PySpark Pipeline & Model Building"]:::backend
        click C "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/best_model.py"
        D["Sentiment Analysis"]:::backend
        click D "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/sentiment_analysis.py"
    end

    %% Layer: Dashboard / Interactive Visualization
    subgraph "Dashboard / Interactive Visualization"
        E["Main Dashboard Entry Point"]:::frontend
        click E "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/Homepage.py"
        F["Filtro Recensioni Page"]:::frontend
        click F "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/pages/Filtro Recensioni.py"
        G["Query Page"]:::frontend
        click G "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/pages/Query.py"
        H["Recensioni Controverse Page"]:::frontend
        click H "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/pages/Recensioni Controverse.py"
        I["Sentiment Analysis Page"]:::frontend
        click I "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/src/pages/Sentiment Analysis.py"
    end

    %% Layer: External Dependencies & Configuration
    subgraph "External Dependencies & Configuration"
        J["Dependencies Definition"]:::external
        click J "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/requirements.txt"
        K["Project Config (.idea)"]:::external
        click K "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/.idea"
        L["Project Config (pyspark-hotel-review.iml)"]:::external
        click L "https://github.com/sickmazin/bigdata_hotelreview_analysis/blob/main/pyspark-hotel-review.iml"
    end

    %% Data Flow Arrows
    A -->|"ingests"| B
    B -->|"cleans/transforms"| C
    B -->|"prepares sentiment data"| D
    C -->|"drives analysis"| E
    D -->|"drives sentiment UI"| I
    E -->|"navigates to"| F
    E -->|"navigates to"| G
    E -->|"navigates to"| H
    E -->|"navigates to"| I

    %% External Dependencies Relationships
    J ---|"supports"| B
    J ---|"supports"| C
    J ---|"supports"| D
    J ---|"supports"| E

    K ---|"configures"| B
    K ---|"configures"| C
    L ---|"configures"| C

    %% Styles
    classDef data fill:#ffdddd,stroke:#aa4444,stroke-width:2px;
    classDef backend fill:#ddeeff,stroke:#4466aa,stroke-width:2px;
    classDef frontend fill:#ddffdd,stroke:#44aa44,stroke-width:2px;
    classDef external fill:#ffffd0,stroke:#aaaa00,stroke-width:2px;
```
