# lidizz/imdb-sentiment-analysis architecture

> Architecture diagram of the GitHub repository lidizz/imdb-sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lidizz/imdb-sentiment-analysis
- Repository: https://github.com/lidizz/imdb-sentiment-analysis

## Overview

1. **Project purpose and overall shape**
   This repository is a sentiment-analysis experiment pipeline for the IMDB 50K review dataset. It is organized as a comparison of multiple modeling approaches rather than a single production app: classic ML baselines, an LSTM model, and a pre-trained Transformer baseline.

2. **Primary data flow**
   The core flow is: raw IMDB CSV in `data/` → text cleaning and preprocessing → train/validation/test split → feature generation → model training/evaluation → saved metrics/plots/models in `results/` and `models/`. The same held-out test set is used across models for fair comparison.

3. **Reusable implementation layer: `src/`**
   `src/` is the main architecture boundary and holds the project logic:
   - `data_loader.py` for loading the dataset and producing splits
   - `preprocessing.py` for HTML removal, lowercasing, punctuation/digit removal, stopword filtering, and lemmatization
   - `features.py` for branch-specific feature preparation
   - `classic_models.py` for scikit-learn classifiers
   - `dl_model.py` for the Keras/TensorFlow LSTM
   - `evaluation.py` for metrics, confusion matrices, and comparison tables
   - `config.py` for shared paths/constants

4. **Two feature/model branches**
   The pipeline splits into two major modeling paths:
   - **Classic ML path:** preprocessed text is transformed with TF-IDF, then fed to Logistic Regression, Naive Bayes, Linear SVM, and Random Forest.
   - **Deep learning path:** text is tokenized, padded, and passed through trainable embeddings into an LSTM.
   These branches share preprocessing but diverge in representation and model stack.

5. **Notebook layer as orchestration and analysis**
   `notebooks/` contains the narrative workflow and visual analysis, but not the core logic. Each notebook imports functions from `src/`, so the notebooks act as presentation/orchestration:
   - EDA and data inspection
   - preprocessing walkthrough
   - classic ML training
   - LSTM training
   - final comparison and error analysis
   This separation keeps the implementation centralized and reusable.

6. **Comparison and baseline evaluation**
   The final comparison notebook combines all model outputs and adds a **pre-trained DistilBERT sentiment model** as a zero-training baseline. That baseline operates as an external inference-only component, distinct from the models trained in this repo.

7. **Artifacts and persistence**
   Generated outputs are material to the architecture:
   - `models/` stores saved trained artifacts
   - `results/metrics/` stores CSV summaries
   - `results/figures/` stores plots and diagnostic visuals
   These artifacts are consumed later by the comparison notebook and the Streamlit app.

8. **Streamlit app as the consumption layer**
   `app/` is a separate front-end layer for presentation and live demo. It does not train models; it reads the saved metrics/model artifacts and presents:
   - preprocessing explanation
   - classic ML results
   - deep learning results
   - model comparison
   - live sentiment prediction across the available models

9. **Pipeline automation**
   `run_pipeline.py` is a convenience orchestrator for running the notebook sequence end-to-end. It supports reproducibility by generating the downstream artifacts in the expected order.

10. **External dependencies that shape the architecture**
   The stack is materially influenced by:
   - **Python 3.11**
   - **scikit-learn** for classic models and TF-IDF
   - **TensorFlow/Keras** for the LSTM
   - **NLTK** for stopwords and lemmatization data
   - **Hugging Face** for the DistilBERT comparison baseline
   The repo includes `download_nltk_data.py` because preprocessing depends on external NLP resources.

11. **Static documentation layer**
   `docs/` contains generated source documentation, which mirrors the `src/` modules. It is a secondary support artifact, not part of the training or inference path.

12. **Architectural boundary summary**
   The cleanest diagram is: **data and preprocessing at the core, split into TF-IDF/classic ML and tokenized/LSTM branches, then merged again in evaluation/comparison, with notebooks and Streamlit as presentation layers over a shared `src/` implementation layer.**

Main entry points: [`src/evaluation.py`](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/evaluation.py) (Evaluation), [`src/config.py`](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/config.py) (Config), [`src/classic_models.py`](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/classic_models.py) (Classic models), [`src/dl_model.py`](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/dl_model.py) (LSTM model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_core["Core logic"]
  node_n_data_loader["Data loader<br/>[data_loader.py]"]
  node_n_preprocessing["Preprocessing<br/>text cleaning<br/>[preprocessing.py]"]
  node_n_features["Feature prep<br/>representation<br/>[features.py]"]
  node_n_classic["Classic models<br/>sklearn<br/>[classic_models.py]"]
  node_n_dl["LSTM model<br/>keras<br/>[dl_model.py]"]
  node_n_eval["Evaluation<br/>metrics<br/>[evaluation.py]"]
  node_n_config["Config<br/>paths/constants<br/>[config.py]"]
end

subgraph group_grp_flow["Workflow"]
  node_n_notebooks["Notebooks<br/>orchestration"]
end

subgraph group_grp_presentation["Presentation"]
  node_n_app["Streamlit app<br/>presentation<br/>[app.py]"]
  node_n_app_pages["App pages<br/>ui pages"]
  node_n_docs["Docs<br/>[index.html]"]
end

subgraph group_grp_artifacts["Artifacts"]
  node_n_artifacts[("Metrics & plots<br/>results")]
  node_n_models[("Saved models<br/>artifacts")]
end

subgraph group_grp_support["Support"]
  node_dataset[("IMDB CSV<br/>dataset")]
  node_n_pipeline["Pipeline runner<br/>orchestrator<br/>[run_pipeline.py]"]
  node_n_nltk["NLTK data<br/>external dependency"]
  node_n_distilbert["DistilBERT<br/>external baseline"]
end

node_dataset -->|"loads"| node_n_data_loader
node_n_data_loader -->|"prepares text"| node_n_preprocessing
node_n_preprocessing -->|"normalizes"| node_n_features
node_n_features -->|"TF-IDF"| node_n_classic
node_n_features -->|"tokens"| node_n_dl
node_n_classic -->|"scores"| node_n_eval
node_n_dl -->|"scores"| node_n_eval
node_n_eval -->|"writes"| node_n_artifacts
node_n_classic -->|"saves"| node_n_models
node_n_dl -->|"saves"| node_n_models
node_n_artifacts -->|"feeds"| node_n_app
node_n_models -->|"loads"| node_n_app
node_n_pipeline -->|"runs"| node_n_notebooks
node_n_notebooks -->|"imports"| node_n_data_loader
node_n_notebooks -->|"trains"| node_n_classic
node_n_notebooks -->|"trains"| node_n_dl
node_n_notebooks -->|"compares"| node_n_eval
node_n_nltk -.->|"supports"| node_n_preprocessing
node_n_config -->|"shared paths"| node_n_data_loader
node_n_config -->|"shared paths"| node_n_preprocessing
node_n_config -->|"shared paths"| node_n_features
node_n_config -->|"shared paths"| node_n_classic
node_n_config -->|"shared paths"| node_n_dl
node_n_config -->|"shared paths"| node_n_eval
node_n_docs -.->|"mirrors"| node_n_data_loader
node_n_docs -.->|"mirrors"| node_n_preprocessing
node_n_docs -.->|"mirrors"| node_n_classic
node_n_docs -.->|"mirrors"| node_n_dl
node_n_docs -.->|"mirrors"| node_n_eval
node_n_app_pages -->|"renders"| node_n_app
node_n_distilbert -.->|"baseline"| node_n_eval

click node_n_data_loader "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/data_loader.py"
click node_n_preprocessing "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/preprocessing.py"
click node_n_features "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/features.py"
click node_n_classic "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/classic_models.py"
click node_n_dl "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/dl_model.py"
click node_n_eval "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/evaluation.py"
click node_n_config "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/config.py"
click node_n_pipeline "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/run_pipeline.py"
click node_n_nltk "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/download_nltk_data.py"
click node_n_artifacts "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/results/metrics/model_comparison_metrics.csv"
click node_n_app "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/app/app.py"
click node_n_app_pages "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/app/pages/4_Model_Comparison.py"
click node_n_docs "https://github.com/lidizz/imdb-sentiment-analysis/blob/main/docs/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_data_loader,node_n_preprocessing,node_n_features,node_n_classic,node_n_dl,node_n_eval,node_n_config toneBlue
class node_n_notebooks toneAmber
class node_n_app,node_n_app_pages,node_n_docs toneMint
class node_n_artifacts,node_n_models toneRose
class node_dataset,node_n_pipeline,node_n_nltk,node_n_distilbert toneIndigo
```

## Components

### Core logic

Shared implementation layer under src/

- [Data loader](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/data_loader.py): loader · Loads the dataset and creates train/validation/test splits · `src/data_loader.py`
- [Preprocessing](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/preprocessing.py): text cleaning · HTML removal, normalization, stopwords, and lemmatization · `src/preprocessing.py`
- [Feature prep](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/features.py): representation · Branch-specific feature preparation for classic ML and deep learning · `src/features.py`
- [Classic models](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/classic_models.py): sklearn · TF-IDF plus Logistic Regression, Naive Bayes, Linear SVM, and Random Forest · `src/classic_models.py`
- [LSTM model](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/dl_model.py): keras · Tokenization, padding, embeddings, and LSTM training · `src/dl_model.py`
- [Evaluation](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/evaluation.py): metrics · Metrics, confusion matrices, comparison tables, and error analysis · `src/evaluation.py`
- [Config](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/src/config.py): paths/constants · Shared paths and constants used across the project · `src/config.py`

### Workflow

Notebook-driven training and comparison flow

- **Notebooks**: orchestration · Exploration, training, and comparison notebooks built on src/

### Presentation

Streamlit app and docs

- [Streamlit app](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/app/app.py): presentation · Reads saved outputs and offers model comparison and live demo pages · `app/app.py`
- [App pages](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/app/pages/4_Model_Comparison.py): ui pages · Preprocessing, classic ML, deep learning, comparison, and live demo views · `app/pages/4_Model_Comparison.py`
- [Docs](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/docs/index.html): generated docs · Static documentation mirroring src/ modules · `docs/index.html`

### Artifacts

Persisted outputs consumed across runs

- [Metrics & plots](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/results/metrics/model_comparison_metrics.csv): results · CSV summaries and figures used for comparison and reporting · `results/metrics/model_comparison_metrics.csv`
- **Saved models**: artifacts · Persisted trained model artifacts consumed by the app

### Support

Dataset setup and orchestration

- **IMDB CSV**: dataset · Raw review data source for the pipeline
- [Pipeline runner](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/run_pipeline.py): orchestrator · Runs the notebook sequence end-to-end for reproducible artifact generation · `run_pipeline.py`
- [NLTK data](https://github.com/lidizz/imdb-sentiment-analysis/blob/main/download_nltk_data.py): external dependency · Downloaded resources required for stopwords and lemmatization · `download_nltk_data.py`
- **DistilBERT**: external baseline · Pre-trained sentiment baseline used for final comparison only

## Connections

- IMDB CSV → Data loader: loads · Raw IMDB reviews are ingested for splitting
- Data loader → Preprocessing: prepares text · Splits feed the shared cleaning pipeline
- Preprocessing → Feature prep: normalizes · Cleaned text is converted to branch-specific representations
- Feature prep → Classic models: TF-IDF · Classic branch uses vectorized text features
- Feature prep → LSTM model: tokens · Deep learning branch uses padded token sequences
- Classic models → Evaluation: scores · Model predictions are evaluated and compared
- LSTM model → Evaluation: scores · LSTM outputs feed shared evaluation
- Evaluation → Metrics & plots: writes · Metrics, plots, and error samples are persisted
- Classic models → Saved models: saves · Trained classic estimators can be reused later
- LSTM model → Saved models: saves · Trained LSTM artifacts are persisted
- Metrics & plots → Streamlit app: feeds · The app reads metrics and figures for presentation
- Saved models → Streamlit app: loads · Live demo pages load saved model artifacts
- Pipeline runner → Notebooks: runs · Automation executes the notebook workflow in order
- Notebooks → Data loader: imports · Notebook workflow depends on the shared src layer
- Notebooks → Classic models: trains · Classic ML notebook orchestrates model training
- Notebooks → LSTM model: trains · Deep learning notebook orchestrates LSTM training
- Notebooks → Evaluation: compares · Comparison notebook aggregates model outputs
- NLTK data → Preprocessing: supports · External NLP resources are required by preprocessing
- Config → Data loader: shared paths · Central configuration is used by core modules
- Config → Preprocessing: shared paths · Central configuration is used by core modules
- Config → Feature prep: shared paths · Central configuration is used by core modules
- Config → Classic models: shared paths · Central configuration is used by core modules
- Config → LSTM model: shared paths · Central configuration is used by core modules
- Config → Evaluation: shared paths · Central configuration is used by core modules
- Docs → Data loader: mirrors · Generated docs mirror src modules
- Docs → Preprocessing: mirrors · Generated docs mirror src modules
- Docs → Classic models: mirrors · Generated docs mirror src modules
- Docs → LSTM model: mirrors · Generated docs mirror src modules
- Docs → Evaluation: mirrors · Generated docs mirror src modules
- App pages → Streamlit app: renders · Pages compose the Streamlit experience
- DistilBERT → Evaluation: baseline · External zero-training baseline joins final comparison
