How shawongazi7/spamshield-ai works
1. This repository is a notebook-driven machine learning project, not a deployed application. The core “system” lives in two Jupyter notebooks: one complete exploratory/educational version and one cleaned implementation version. All model training, evaluation, and interactive prediction happen inside the notebook environment.
2. The primary entry points are:
• SpamShield_AI_Complete.ipynb: the full end-to-end analysis, comparison of models, visualization, and the interactive prediction UI.
• SpamShield_AI_Clean.ipynb: a streamlined version of the same pipeline, likely meant for clearer reuse or presentation.
There is no separate Python package, service layer, or API layer in this repo.
3. The data layer is file-based and local:
• spam.csv is the main training corpus with 5,572 labeled emails.
• spam_ham_test_dataset.csv is a small external validation set with 79 emails from a different source.
The architecture depends heavily on comparing in-domain training results against out-of-domain test performance.
4. The processing pipeline is classic NLP classification:
raw email text is cleaned/preprocessed, then transformed into numeric features using TF-IDF. The README indicates unigram + bigram support and a cap of about 5,000 features. This feature-extraction step is the main boundary between text input and model-ready vectors.
5. Class imbalance is a first-class architectural concern. The training set is heavily skewed toward ham, so the notebooks evaluate balancing strategies such as class_weight='balanced' and SMOTE from imblearn. In the project narrative, class-weighting appears to be the preferred production-like choice over synthetic oversampling.
6. The modeling subsystem compares four supervised classifiers on the same feature matrix:
• Logistic Regression
• Naive Bayes
• Linear SVM
• Random Forest
This is a model-selection architecture, not a single-model pipeline; the notebooks are designed to benchmark speed, accuracy, F1, and generalization across candidates.
7. Evaluation is multi-stage. The notebooks use training metrics, cross-validation, and an external holdout dataset. That external dataset is architecturally important because it reveals the gap between internal performance and real-world generalization, and it drives the final model choice.
8. The project’s decision boundary is based on generalization, not just training scores. Logistic Regression emerges as the best overall model because it performs best on unseen data, even though several models show similarly high training metrics. That makes the evaluation loop a central part of the architecture rather than a reporting afterthought.
9. There is an interactive prediction layer inside the notebook using ipywidgets. This functions like a lightweight UI for entering email text and receiving a spam/ham prediction, but it is notebook-local rather than a deployed frontend.
10. The technology stack is standard Python ML tooling:
• pandas and numpy for data handling
• scikit-learn for vectorization, model training, validation, and metrics
• imblearn for SMOTE experiments
• matplotlib and seaborn for analysis/visualization
• ipywidgets for interactive notebook controls
This stack shapes the architecture into an offline analytics and modeling workflow.
11. The repo also contains documentation artifacts that support architecture understanding:
• TECHNICAL_DOCUMENTATION.md for deeper technical notes
• DEVELOPMENT_LOG.md for iteration history and decisions
• LEARNING_JOURNEY.md for narrative context
These files don’t execute any logic, but they explain why the pipeline and model choices look the way they do.
12. requirements.txt defines the execution boundary for reproducing the notebooks. Since the project is notebook-based and depends on scientific Python libraries, environment consistency matters more than application deployment plumbing.
13. The current architecture stops short of production deployment. The README explicitly mentions future API deployment, online learning, and deep learning experiments, but the repo itself is still an experiment-and-validation system centered on local CSV data, notebook execution, and comparative model analysis.
Main entry points: SpamShield_AI_Complete.ipynb Complete notebook · TECHNICAL_DOCUMENTATION.md Technical docs · SpamShield_AI_Clean.ipynb Clean notebook · spam.csv Training data
Components 17
Notebook workflow
Interactive analysis and model selection live here.
- Complete notebook · End-to-end exploratory version with model comparison, visualization, and interactive prediction.
SpamShield_AI_Complete.ipynb - Clean notebook · Streamlined implementation-oriented version of the spam classifier workflow.
SpamShield_AI_Clean.ipynb - Prediction UI · ipywidgets-based interactive text entry and prediction inside the notebook.
ML pipeline
Text processing, feature extraction, modeling, and evaluation.
- Text cleaning · Raw message normalization and preparation before vectorization.
- TF-IDF vectors · Unigram/bigram TF-IDF representation with feature cap for model input.
- Imbalance handling · Class weighting and SMOTE experiments for skewed label distribution.
- Logistic regression · Preferred model due to best generalization on unseen data.
- Naive Bayes · Baseline probabilistic classifier compared on the same feature matrix.
- Linear SVM · Margin-based linear classifier included in the benchmark set.
- Random forest · Tree ensemble baseline used for comparison against linear models.
- Evaluation loop · Training metrics, cross-validation, and external holdout evaluation.
Support files
Data, documentation, and environment definitions.
- Training data · Primary spam/ham dataset used to train and compare classifiers.
spam.csv - External test set · Smaller out-of-domain dataset used to probe real-world generalization.
spam_ham_test_dataset.csv - Technical docs · Narrative and technical notes explaining design choices and project evolution.
TECHNICAL_DOCUMENTATION.md - Dev log · Iteration history and decisions during development.
DEVELOPMENT_LOG.md - Learning notes · Contextual write-up of the learning and project process.
LEARNING_JOURNEY.md - Requirements · Pinned Python dependency set for reproducing the notebook workflow.
requirements.txt
Connections 26
- Complete notebook → Text cleaning: runs · The complete notebook executes the preprocessing workflow.
- Clean notebook → Text cleaning: runs · The cleaned notebook applies the same text preparation stage.
- Training data → Text cleaning: feeds · Training emails are cleaned before feature extraction.
- External test set → Text cleaning: feeds · External evaluation emails are processed through the same text pipeline.
- Text cleaning → TF-IDF vectors: transforms to · Normalized text is converted into numeric TF-IDF features.
- TF-IDF vectors → Imbalance handling: optionally balances · Feature matrix is paired with balancing strategies for class skew.
- Imbalance handling → Logistic regression: trains · Balanced training is evaluated on logistic regression.
- Imbalance handling → Naive Bayes: trains · Balanced training is evaluated on naive Bayes.
- Imbalance handling → Linear SVM: trains · Balanced training is evaluated on linear SVM.
- Imbalance handling → Random forest: trains · Balanced training is evaluated on random forest.
- TF-IDF vectors → Logistic regression: features for · TF-IDF vectors are the shared input to the classifiers.
- TF-IDF vectors → Naive Bayes: features for · TF-IDF vectors are the shared input to the classifiers.
- TF-IDF vectors → Linear SVM: features for · TF-IDF vectors are the shared input to the classifiers.
- TF-IDF vectors → Random forest: features for · TF-IDF vectors are the shared input to the classifiers.
- Logistic regression → Evaluation loop: measured by · Logistic regression is assessed with internal and external validation.
- Naive Bayes → Evaluation loop: measured by · Naive Bayes is assessed with internal and external validation.
- Linear SVM → Evaluation loop: measured by · Linear SVM is assessed with internal and external validation.
- Random forest → Evaluation loop: measured by · Random forest is assessed with internal and external validation.
- Evaluation loop → Complete notebook: reported in · Results are visualized and compared in the full notebook.
- Evaluation loop → Clean notebook: reported in · Selected workflow is summarized in the cleaned notebook.
- Evaluation loop → Prediction UI: powers · The chosen model is exposed through interactive notebook prediction.
- Prediction UI → Complete notebook: embedded in · The UI lives inside the complete notebook environment.
- Technical docs → Complete notebook: explains · Documentation provides technical context for the main notebook.
- Dev log → Technical docs: supports · Development history supplements the technical documentation.
- Learning notes → Technical docs: supports · Learning narrative supplements the technical documentation.
- Requirements → Complete notebook: reproduces · Dependency pinning defines the notebook execution environment.