How dataphish/phishfuzzer works
This repository produces a metadata-rich email dataset and uses it to benchmark LLM email classification. The documented data workflow enriches seed emails with motivation and structural metadata, normalizes the records, and generates entity- and length-controlled variants; classification runs then produce results for reliability, provenance, confusion-matrix, and flip-score analysis. README methodology supports the broad dataset-stage links; source excerpts show model calls, expansion prompt use, and result handling. Some stage internals and analysis implementations are unsampled, so their connections are documented rather than inferred from imports.
Main entry points: results_raw_gemini.jsonl Gemini results · results_raw_qwen.jsonl Qwen results · expand.py Seeded expansion · classify_Gemini.py Gemini classifier
Components 21
Dataset preparation
- Curated seed emails
DataSet_Creation/1_Original_DataSet/emails_base.json - Motivation labeling
DataSet_Creation/2_Motivation_Labeling/Motivaitonal_labeling.py - Labeled emails
DataSet_Creation/2_Motivation_Labeling/emails_labeled_gemini.json - Metadata population
DataSet_Creation/3_Population/populate_prompt.py - Populated emails
DataSet_Creation/3_Population/emails_populated_non_manual_gemini.json - Schema normalization
DataSet_Creation/4_Normalize/normalize.py - Normalized emails
DataSet_Creation/4_Normalize/emails_normalized.json
Variant generation
- Variant prompts
DataSet_Creation/5_RePhrase/prompts.py - Seeded expansion
DataSet_Creation/5_RePhrase/expand.py - Seed dataset
PhishFuzzer_emails_original_seed_v1.json - Rephrased dataset
PhishFuzzer_emails_entity_rephrased_v1.json
Classification benchmark
- Gemini classifier
Classification/classify_Gemini.py - Qwen classifier
Classification/classify_Qwen.py - Gemini results
Classification/Results/results_raw_gemini.jsonl - Qwen results
Classification/Results/results_raw_qwen.jsonl
Benchmark analysis
- Confusion matrices
Classification/Analyze/aggregate_confusion_matrix.py - Provenance comparison
Classification/Analyze/private_public_original_rephrased.py - Reliability metrics
Classification/Analyze/template_reliability_metrics.py - TFS matrices
Classification/Analyze/tfs_visual_matrices.py - TFS percentage matrices
Classification/Analyze/tfs_visual_matrices_percentage.py
Other components
- OpenRouter API
Connections 30
- Curated seed emails → Motivation labeling: labels intent
ReadME.md - Motivation labeling → Labeled emails: writes labels
ReadME.md - Labeled emails → Metadata population: enriches records
ReadME.md - Metadata population → OpenRouter API: queries model
DataSet_Creation/3_Population/populate_prompt.py - Metadata population → Populated emails: writes metadata
ReadME.md - Populated emails → Schema normalization: standardizes
ReadME.md - Schema normalization → Normalized emails: writes schema
DataSet_Creation/4_Normalize/normalize.py - Normalized emails → Seeded expansion: seeds variants
ReadME.md - Seeded expansion → Variant prompts: uses prompts
DataSet_Creation/5_RePhrase/expand.py - Seeded expansion → OpenRouter API: requests variants
DataSet_Creation/5_RePhrase/expand.py - Seeded expansion → Seed dataset: preserves seeds
DataSet_Creation/5_RePhrase/expand.py - Seeded expansion → Rephrased dataset: writes variants
ReadME.md - Seed dataset → Gemini classifier: classifies emails
- Rephrased dataset → Gemini classifier: classifies variants
- Seed dataset → Qwen classifier: classifies emails
- Rephrased dataset → Qwen classifier: classifies variants
- Gemini classifier → OpenRouter API: queries model
Classification/classify_Gemini.py - Qwen classifier → OpenRouter API: queries model
Classification/classify_Qwen.py - Gemini classifier → Gemini results: writes predictions
Classification/classify_Gemini.py - Qwen classifier → Qwen results: writes predictions
Classification/classify_Qwen.py - Gemini results → Confusion matrices: measures errors
ReadME.md - Qwen results → Confusion matrices: measures errors
ReadME.md - Gemini results → Provenance comparison: compares provenance
Classification/Analyze/private_public_original_rephrased.py - Qwen results → Provenance comparison: compares provenance
Classification/Analyze/private_public_original_rephrased.py - Gemini results → Reliability metrics: computes reliability
ReadME.md - Qwen results → Reliability metrics: computes reliability
ReadME.md - Gemini results → TFS matrices: analyzes failures
Classification/Analyze/tfs_visual_matrices.py - Qwen results → TFS matrices: analyzes failures
Classification/Analyze/tfs_visual_matrices.py - Gemini results → TFS percentage matrices: analyzes failures
Classification/Analyze/tfs_visual_matrices_percentage.py - Qwen results → TFS percentage matrices: analyzes failures
Classification/Analyze/tfs_visual_matrices_percentage.py