How hawkyvan/anti-spam_model_project works
This project is a data science and machine learning pipeline focused on anti-spam email classification. The core purpose is to parse .eml files, extract relevant email data (sender, recipient, date, text, and spam/ham labels), store the data in a CSV file, perform exploratory data analysis, and train/test several classification models (Logistic Regression, SVM, Neural Network, Bernoulli Naive Bayes).
Main components
• Data Sources: The raw email files organized in the “data” directory (with subfolders “ham” and “spam”). These are your unprocessed .eml files.
• Parser Module: Represented by the single file “parser.py”. This module is responsible for scanning the directories, extracting information (sender, recipient, date, text, spam label), and converting the emails into a structured format.
• Data Storage: The output CSV file (eml_dataset.csv) that stores the parsed data. It serves as the central repository for downstream analysis.
• Data Analysis Module: Illustrated by “analysis.ipynb”. This notebook uses the CSV file for performing descriptive statistics, visualizations (for sender frequencies, temporal distributions, etc.), and key insights.
• Machine Learning Module: Depicted by “models.ipynb”. This module contains experiments for training multiple classifiers. It reads the CSV data, performs preprocessing (e.g., vectorization), splits the data into training/testing subsets, tunes hyperparameters, and evaluates model performance.
• Dependency Management: “requirements.txt” can be noted as a configuration artifact indicating the libraries used (e.g., scikit-learn, pandas, numpy, matplotlib, etc.).
Relationships and data flow between components
• Optionally, include data feedback loops if subsequent refined datasets are generated from analysis steps.
Architectural patterns and technologies
• Use a pipeline (ETL) pattern: Extract (raw .eml files), Transform (parsing and cleaning), and Load (CSV file).
• Modular separation between data preprocessing, exploratory analysis, and model training. Although not a full-stack web application, the separation of concerns is clear from the different notebooks and scripts.
• Labels for the classification models implemented – mention Logistic Regression, SVM, Neural Network (MLP), and Bernoulli Naive Bayes.
• Optionally, reference standard libraries (like scikit-learn for ML, Pandas for data manipulation, and Matplotlib/Seaborn for visualization) near the respective modules.