How phamdanganuit/ie212_bigdata works
Project Type and Purpose
• This is a full-stack application that combines real-time comment analysis (toxic comment detection) with machine learning powered by the phoBERT model and data streaming using PySpark.
• The project has multiple facets: a user interface layer (React-based front-end), a Node.js API back-end (in the BE directory), and several ML components (Python scripts in the BE/python folder and Jupyter notebooks in “Train Model” and “Preprocessing”).
• It also mentions integration with MongoDB and use of Google Colab for preprocessing, emphasizing both the live-stream analytics and model training aspects.
Analyzing the File Structure
• At the top level, the “Interface” folder contains two main subdirectories:
a. FE (Front-end): Contains ReactJS project files (Vite configuration, main.jsx, assets, routes, etc.) that comprise the UI where users input the livestream URL and view prediction results.
b. BE (Back-end): Contains Node.js server files (app.js, configuration, routes, models) and a Python subdirectory with a script (crawlData.py) that likely helps in data crawling or integration between Node and Python ML routines.
• Outside “Interface”, there are additional folders and notebooks (“Train Model”, “Preprocessing”, logs, etc.) that relate to training and evaluation of the ML model, showing the offline training/preprocessing pipeline.
• Notice the configuration files for both back-end (package.json, config/database.js) and front-end (eslint, tailwind, Vite config) which suggest clear separation of concerns.
Key Components
a. Front-End (FE):
• User interface built with ReactJS.
b. Back-End (BE):
• Node.js Express server.
• A sub-component for Python integration (Python/crawlData.py) which possibly assists in data fetching or triggers ML functions.
c. Database Layer:
d. Machine Learning Pipeline:
• Two parts:
i. Model Training and Preprocessing: “Train Model” and “Preprocessing” notebooks (Preprocessing_for_ML.ipynb, TrainML.ipynb, phoBERT_train.ipynb, etc.). These are used offline (Google Colab) to develop and refine the ML model.
ii. Real-Time Prediction/Streaming: Mention the integration of PySpark for streaming processing, as described in the README. This could be drawn as an independent module labeled “Streaming & Data Processing (PySpark)” that feeds processed data into the decision-making engine.
• If applicable, represent a flow showing that the trained phoBERT model (downloaded via instructions after cloning) is deployed within the back-end or as a separate microservice that the back-end queries for predictions.
e. External Services:
Architectural Patterns and Technology Highlights
• Separation of concerns between the UI layer, API/back-end layer, and the ML/streaming pipelines.
• Use of Node.js for API orchestration, Python (with scripts and Jupyter notebooks) for ML training and data crawling, and real-time processing with PySpark.
• Labels for each technology where relevant (e.g., “ReactJS/Vite” for FE, “NodeJS/Express” for BE, “MongoDB” for Database, “phoBERT Model”, “PySpark” for streaming).