How muhammederbay10/text-sentiment-analysis works
Project Type & Purpose
• This is a Python‐based full‐stack ML application combining a model training pipeline (using Hugging Face’s BERT fine-tuning) with a lightweight Streamlit web interface for live inference.
• Key use case: train on IMDB reviews, persist the fine-tuned model, and expose it via Streamlit for real-time sentiment prediction.
High-Level Components
a. Data Layer
• CSV data source (data/Movie_Dataset.csv)
• Local filesystem storage for checkpoints and any cached tokenizers
b. Preprocessing & Data Loading
• src/data_loader.py: reads, cleans, tokenizes (via Hugging Face AutoTokenizer), and prepares PyTorch or TensorFlow datasets
c. Training Pipeline
• src/train.py: configures the Hugging Face Trainer API, sets hyperparameters, kicks off fine-tuning, writes model checkpoints
d. Evaluation Module
• src/evaluate.py: loads a saved checkpoint, runs validation on held-out data, computes metrics (accuracy, F1, precision, recall)
e. Model Persistence
• Local folder (implicitly created by Trainer) storing model checkpoints, tokenizer config, training metrics
f. Inference / Serving Layer
• app.py (Streamlit): loads the final checkpoint and tokenizer, provides a web form for users to submit text, displays predictions in real time
g. External Dependencies
• Python >=3.8, streamlit, transformers, pandas, scikit-learn, torch or tensorflow
Component Interactions & Data Flow
1. Developer invokes src/train.py
– Data Loader fetches CSV → Tokenizer → Datasets → Trainer API → Model Checkpoint
2. Post-training, src/evaluate.py loads checkpoint → runs validation → outputs metrics
3. app.py (Streamlit) on startup loads the same checkpoint & tokenizer → waits for user input → runs inference → displays sentiment label
4. requirements.txt lists all dependencies; app.py & src modules import these libraries
Architectural Patterns & Design Principles
• Pipeline Pattern: clear separation of data loading, training, evaluation, and serving.
• Modularity: each major function is in its own script/module (single responsibility).
• Reproducibility: Trainer API ensures consistent checkpointing; requirements.txt pins dependencies.
• Lightweight Serving: uses Streamlit rather than a full web framework to minimize overhead.