How lidizz/imdb-sentiment-analysis works
1. Project purpose and overall shape
This repository is a sentiment-analysis experiment pipeline for the IMDB 50K review dataset. It is organized as a comparison of multiple modeling approaches rather than a single production app: classic ML baselines, an LSTM model, and a pre-trained Transformer baseline.
2. Primary data flow
The core flow is: raw IMDB CSV in data/ → text cleaning and preprocessing → train/validation/test split → feature generation → model training/evaluation → saved metrics/plots/models in results/ and models/. The same held-out test set is used across models for fair comparison.
3. Reusable implementation layer: src/
src/ is the main architecture boundary and holds the project logic:
- data_loader.py for loading the dataset and producing splits
- preprocessing.py for HTML removal, lowercasing, punctuation/digit removal, stopword filtering, and lemmatization
- features.py for branch-specific feature preparation
- classic_models.py for scikit-learn classifiers
- dl_model.py for the Keras/TensorFlow LSTM
- evaluation.py for metrics, confusion matrices, and comparison tables
- config.py for shared paths/constants
4. Two feature/model branches
The pipeline splits into two major modeling paths:
- Classic ML path: preprocessed text is transformed with TF-IDF, then fed to Logistic Regression, Naive Bayes, Linear SVM, and Random Forest.
- Deep learning path: text is tokenized, padded, and passed through trainable embeddings into an LSTM.
These branches share preprocessing but diverge in representation and model stack.
5. Notebook layer as orchestration and analysis
notebooks/ contains the narrative workflow and visual analysis, but not the core logic. Each notebook imports functions from src/, so the notebooks act as presentation/orchestration:
- EDA and data inspection
- preprocessing walkthrough
- classic ML training
- LSTM training
- final comparison and error analysis
This separation keeps the implementation centralized and reusable.
6. Comparison and baseline evaluation
The final comparison notebook combines all model outputs and adds a pre-trained DistilBERT sentiment model as a zero-training baseline. That baseline operates as an external inference-only component, distinct from the models trained in this repo.
7. Artifacts and persistence
Generated outputs are material to the architecture:
- models/ stores saved trained artifacts
- results/metrics/ stores CSV summaries
- results/figures/ stores plots and diagnostic visuals
These artifacts are consumed later by the comparison notebook and the Streamlit app.
8. Streamlit app as the consumption layer
app/ is a separate front-end layer for presentation and live demo. It does not train models; it reads the saved metrics/model artifacts and presents:
- preprocessing explanation
- classic ML results
- deep learning results
- model comparison
- live sentiment prediction across the available models
9. Pipeline automation
run_pipeline.py is a convenience orchestrator for running the notebook sequence end-to-end. It supports reproducibility by generating the downstream artifacts in the expected order.
10. External dependencies that shape the architecture
The stack is materially influenced by:
- Python 3.11
- scikit-learn for classic models and TF-IDF
- TensorFlow/Keras for the LSTM
- NLTK for stopwords and lemmatization data
- Hugging Face for the DistilBERT comparison baseline
The repo includes download_nltk_data.py because preprocessing depends on external NLP resources.
11. Static documentation layer
docs/ contains generated source documentation, which mirrors the src/ modules. It is a secondary support artifact, not part of the training or inference path.
12. Architectural boundary summary
The cleanest diagram is: data and preprocessing at the core, split into TF-IDF/classic ML and tokenized/LSTM branches, then merged again in evaluation/comparison, with notebooks and Streamlit as presentation layers over a shared src/ implementation layer.
Main entry points: evaluation.py Evaluation · config.py Config · classic_models.py Classic models · dl_model.py LSTM model
Components 17
Core logic
Shared implementation layer under src/
- Data loader · Loads the dataset and creates train/validation/test splits
src/data_loader.py - Preprocessing · HTML removal, normalization, stopwords, and lemmatization
src/preprocessing.py - Feature prep · Branch-specific feature preparation for classic ML and deep learning
src/features.py - Classic models · TF-IDF plus Logistic Regression, Naive Bayes, Linear SVM, and Random Forest
src/classic_models.py - LSTM model · Tokenization, padding, embeddings, and LSTM training
src/dl_model.py - Evaluation · Metrics, confusion matrices, comparison tables, and error analysis
src/evaluation.py - Config · Shared paths and constants used across the project
src/config.py
Workflow
Notebook-driven training and comparison flow
- Notebooks · Exploration, training, and comparison notebooks built on src/
Presentation
Streamlit app and docs
- Streamlit app · Reads saved outputs and offers model comparison and live demo pages
app/app.py - App pages · Preprocessing, classic ML, deep learning, comparison, and live demo views
app/pages/4_Model_Comparison.py - Docs · Static documentation mirroring src/ modules
docs/index.html
Artifacts
Persisted outputs consumed across runs
- Metrics & plots · CSV summaries and figures used for comparison and reporting
results/metrics/model_comparison_metrics.csv - Saved models · Persisted trained model artifacts consumed by the app
Support
Dataset setup and orchestration
- IMDB CSV · Raw review data source for the pipeline
- Pipeline runner · Runs the notebook sequence end-to-end for reproducible artifact generation
run_pipeline.py - NLTK data · Downloaded resources required for stopwords and lemmatization
download_nltk_data.py - DistilBERT · Pre-trained sentiment baseline used for final comparison only
Connections 31
- IMDB CSV → Data loader: loads · Raw IMDB reviews are ingested for splitting
- Data loader → Preprocessing: prepares text · Splits feed the shared cleaning pipeline
- Preprocessing → Feature prep: normalizes · Cleaned text is converted to branch-specific representations
- Feature prep → Classic models: TF-IDF · Classic branch uses vectorized text features
- Feature prep → LSTM model: tokens · Deep learning branch uses padded token sequences
- Classic models → Evaluation: scores · Model predictions are evaluated and compared
- LSTM model → Evaluation: scores · LSTM outputs feed shared evaluation
- Evaluation → Metrics & plots: writes · Metrics, plots, and error samples are persisted
- Classic models → Saved models: saves · Trained classic estimators can be reused later
- LSTM model → Saved models: saves · Trained LSTM artifacts are persisted
- Metrics & plots → Streamlit app: feeds · The app reads metrics and figures for presentation
- Saved models → Streamlit app: loads · Live demo pages load saved model artifacts
- Pipeline runner → Notebooks: runs · Automation executes the notebook workflow in order
- Notebooks → Data loader: imports · Notebook workflow depends on the shared src layer
- Notebooks → Classic models: trains · Classic ML notebook orchestrates model training
- Notebooks → LSTM model: trains · Deep learning notebook orchestrates LSTM training
- Notebooks → Evaluation: compares · Comparison notebook aggregates model outputs
- NLTK data → Preprocessing: supports · External NLP resources are required by preprocessing
- Config → Data loader: shared paths · Central configuration is used by core modules
- Config → Preprocessing: shared paths · Central configuration is used by core modules
- Config → Feature prep: shared paths · Central configuration is used by core modules
- Config → Classic models: shared paths · Central configuration is used by core modules
- Config → LSTM model: shared paths · Central configuration is used by core modules
- Config → Evaluation: shared paths · Central configuration is used by core modules
- Docs → Data loader: mirrors · Generated docs mirror src modules
- Docs → Preprocessing: mirrors · Generated docs mirror src modules
- Docs → Classic models: mirrors · Generated docs mirror src modules
- Docs → LSTM model: mirrors · Generated docs mirror src modules
- Docs → Evaluation: mirrors · Generated docs mirror src modules
- App pages → Streamlit app: renders · Pages compose the Streamlit experience
- DistilBERT → Evaluation: baseline · External zero-training baseline joins final comparison