# phamdanganuit/ie212_bigdata architecture

> Architecture diagram of the GitHub repository phamdanganuit/ie212_bigdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phamdanganuit/ie212_bigdata
- Repository: https://github.com/phamdanganuit/ie212_bigdata

## Overview

##### Project Type and Purpose
- This is a full-stack application that combines real-time comment analysis (toxic comment detection) with machine learning powered by the phoBERT model and data streaming using PySpark.
- The project has multiple facets: a user interface layer (React-based front-end), a Node.js API back-end (in the BE directory), and several ML components (Python scripts in the BE/python folder and Jupyter notebooks in “Train Model” and “Preprocessing”).
- It also mentions integration with MongoDB and use of Google Colab for preprocessing, emphasizing both the live-stream analytics and model training aspects.

##### Analyzing the File Structure
- At the top level, the “Interface” folder contains two main subdirectories:
a. FE (Front-end): Contains ReactJS project files (Vite configuration, main.jsx, assets, routes, etc.) that comprise the UI where users input the livestream URL and view prediction results.
b. BE (Back-end): Contains Node.js server files (app.js, configuration, routes, models) and a Python subdirectory with a script (crawlData.py) that likely helps in data crawling or integration between Node and Python ML routines.
- Outside “Interface”, there are additional folders and notebooks (“Train Model”, “Preprocessing”, logs, etc.) that relate to training and evaluation of the ML model, showing the offline training/preprocessing pipeline.
- Notice the configuration files for both back-end (package.json, config/database.js) and front-end (eslint, tailwind, Vite config) which suggest clear separation of concerns.

##### Key Components
a. Front-End (FE):
- User interface built with ReactJS.

b. Back-End (BE):
- Node.js Express server.
- A sub-component for Python integration (Python/crawlData.py) which possibly assists in data fetching or triggers ML functions.

c. Database Layer:

d. Machine Learning Pipeline:
- Two parts:
i. Model Training and Preprocessing: “Train Model” and “Preprocessing” notebooks (Preprocessing_for_ML.ipynb, TrainML.ipynb, phoBERT_train.ipynb, etc.). These are used offline (Google Colab) to develop and refine the ML model.
ii. Real-Time Prediction/Streaming: Mention the integration of PySpark for streaming processing, as described in the README. This could be drawn as an independent module labeled “Streaming & Data Processing (PySpark)” that feeds processed data into the decision-making engine.
- If applicable, represent a flow showing that the trained phoBERT model (downloaded via instructions after cloning) is deployed within the back-end or as a separate microservice that the back-end queries for predictions.

e. External Services:

##### Architectural Patterns and Technology Highlights
- Separation of concerns between the UI layer, API/back-end layer, and the ML/streaming pipelines.
- Use of Node.js for API orchestration, Python (with scripts and Jupyter notebooks) for ML training and data crawling, and real-time processing with PySpark.
- Labels for each technology where relevant (e.g., “ReactJS/Vite” for FE, “NodeJS/Express” for BE, “MongoDB” for Database, “phoBERT Model”, “PySpark” for streaming).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Services"
        extYT["YouTube Live Data"]:::external
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        FE["Frontend (React/Vite)"]:::frontend
    end

    %% Application Layer
    subgraph "Application Layer"
        BE["Backend (NodeJS/Express)"]:::backend
        PD["Python Data Crawler"]:::python
    end

    %% ML Processing Layer
    subgraph "ML Processing Layer"
        MLPre["ML Preprocessing (Jupyter)"]:::ml
        MLTrain1["ML Training: phoBERT"]:::ml
        MLTrain2["ML Training: TrainML"]:::ml
        Spark["Streaming & Data Processing (PySpark)"]:::ml
    end

    %% Data Layer
    subgraph "Data Layer"
        DB["Database (MongoDB)"]:::database
    end

    %% Connections
    extYT -->|"Live_Data"| PD
    FE -->|"API_Calls"| BE
    BE -->|"Data_Storage"| DB
    BE -->|"Trigger_Data_Crawl"| PD
    BE -->|"Streaming_Inference"| Spark
    Spark -->|"Predictions"| BE
    MLPre -->|"Preprocess"| MLTrain1
    MLPre -->|"Preprocess"| MLTrain2
    MLTrain1 -->|"Deploy_Model"| BE
    MLTrain2 -->|"Deploy_Model"| BE

    %% Click Events
    click FE "https://github.com/phamdanganuit/ie212_bigdata/tree/master/Interface/FE"
    click BE "https://github.com/phamdanganuit/ie212_bigdata/tree/master/Interface/BE"
    click PD "https://github.com/phamdanganuit/ie212_bigdata/blob/master/Interface/BE/python/crawlData.py"
    click DB "https://github.com/phamdanganuit/ie212_bigdata/blob/master/Interface/BE/config/database.js"
    click MLPre "https://github.com/phamdanganuit/ie212_bigdata/blob/master/Preprocessing/Preprocessing_for_ML.ipynb"
    click MLTrain1 "https://github.com/phamdanganuit/ie212_bigdata/blob/master/Train Model/phoBERT_train.ipynb"
    click MLTrain2 "https://github.com/phamdanganuit/ie212_bigdata/blob/master/Train Model/TrainML.ipynb"

    %% Styles
    classDef frontend fill:#f8c471,stroke:#b9770e,stroke-width:2px;
    classDef backend fill:#82e0aa,stroke:#196f3d,stroke-width:2px;
    classDef python fill:#d2b4de,stroke:#7d3c98,stroke-width:2px;
    classDef ml fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef database fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef external fill:#d7bde2,stroke:#6c3483,stroke-width:2px;
```
