# dataphish/phishfuzzer architecture

> Architecture diagram of the GitHub repository dataphish/phishfuzzer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataphish/phishfuzzer
- Repository: https://github.com/dataphish/phishfuzzer

## Overview

This repository produces a metadata-rich email dataset and uses it to benchmark LLM email classification. The documented data workflow enriches seed emails with motivation and structural metadata, normalizes the records, and generates entity- and length-controlled variants; classification runs then produce results for reliability, provenance, confusion-matrix, and flip-score analysis. README methodology supports the broad dataset-stage links; source excerpts show model calls, expansion prompt use, and result handling. Some stage internals and analysis implementations are unsampled, so their connections are documented rather than inferred from imports.

Main entry points: [`Classification/Results/results_raw_gemini.jsonl`](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_gemini.jsonl) (Gemini results), [`Classification/Results/results_raw_qwen.jsonl`](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_qwen.jsonl) (Qwen results), [`DataSet_Creation/5_RePhrase/expand.py`](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/5_RePhrase/expand.py) (Seeded expansion), [`Classification/classify_Gemini.py`](https://github.com/dataphish/phishfuzzer/blob/main/Classification/classify_Gemini.py) (Gemini classifier)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_dataset["Dataset preparation"]
  node_curated_seeds[("Curated seed emails<br/>[emails_base.json]")]
  node_motivation_labels["Motivation labeling"]
  node_labeled_emails[("Labeled emails")]
  node_metadata_population["Metadata population<br/>[populate_prompt.py]"]
  node_populated_emails[("Populated emails")]
  node_normalization["Schema normalization<br/>[normalize.py]"]
  node_normalized_emails[("Normalized emails")]
end

subgraph group_generation["Variant generation"]
  node_variant_prompts["Variant prompts<br/>[prompts.py]"]
  node_expansion["Seeded expansion<br/>[expand.py]"]
  node_seed_release[("Seed dataset")]
  node_expanded_release[("Rephrased dataset")]
end

subgraph group_benchmark["Classification benchmark"]
  node_gemini_classifier["Gemini classifier<br/>[classify_Gemini.py]"]
  node_qwen_classifier["Qwen classifier<br/>[classify_Qwen.py]"]
  node_gemini_results[("Gemini results")]
  node_qwen_results[("Qwen results")]
end

subgraph group_analysis["Benchmark analysis"]
  node_confusion_analysis["Confusion matrices"]
  node_provenance_analysis["Provenance comparison"]
  node_reliability_analysis["Reliability metrics"]
  node_tfs_analysis["TFS matrices"]
  node_tfs_percentage_analysis["TFS percentage matrices"]
end

node_openrouter{{"OpenRouter API"}}

node_curated_seeds -->|"labels intent"| node_motivation_labels
node_motivation_labels -->|"writes labels"| node_labeled_emails
node_labeled_emails -->|"enriches records"| node_metadata_population
node_metadata_population -->|"queries model"| node_openrouter
node_metadata_population -->|"writes metadata"| node_populated_emails
node_populated_emails -->|"standardizes"| node_normalization
node_normalization -->|"writes schema"| node_normalized_emails
node_normalized_emails -->|"seeds variants"| node_expansion
node_expansion -->|"uses prompts"| node_variant_prompts
node_expansion -->|"requests variants"| node_openrouter
node_expansion -->|"preserves seeds"| node_seed_release
node_expansion -->|"writes variants"| node_expanded_release
node_seed_release -.->|"classifies emails"| node_gemini_classifier
node_expanded_release -.->|"classifies variants"| node_gemini_classifier
node_seed_release -.->|"classifies emails"| node_qwen_classifier
node_expanded_release -.->|"classifies variants"| node_qwen_classifier
node_gemini_classifier -->|"queries model"| node_openrouter
node_qwen_classifier -->|"queries model"| node_openrouter
node_gemini_classifier -->|"writes predictions"| node_gemini_results
node_qwen_classifier -->|"writes predictions"| node_qwen_results
node_gemini_results -->|"measures errors"| node_confusion_analysis
node_qwen_results -->|"measures errors"| node_confusion_analysis
node_gemini_results -->|"compares provenance"| node_provenance_analysis
node_qwen_results -->|"compares provenance"| node_provenance_analysis
node_gemini_results -->|"computes reliability"| node_reliability_analysis
node_qwen_results -->|"computes reliability"| node_reliability_analysis
node_gemini_results -->|"analyzes failures"| node_tfs_analysis
node_qwen_results -->|"analyzes failures"| node_tfs_analysis
node_gemini_results -->|"analyzes failures"| node_tfs_percentage_analysis
node_qwen_results -->|"analyzes failures"| node_tfs_percentage_analysis

click node_curated_seeds "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/1_Original_DataSet/emails_base.json"
click node_motivation_labels "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/2_Motivation_Labeling/Motivaitonal_labeling.py"
click node_labeled_emails "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/2_Motivation_Labeling/emails_labeled_gemini.json"
click node_metadata_population "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/3_Population/populate_prompt.py"
click node_populated_emails "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/3_Population/emails_populated_non_manual_gemini.json"
click node_normalization "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/4_Normalize/normalize.py"
click node_normalized_emails "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/4_Normalize/emails_normalized.json"
click node_variant_prompts "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/5_RePhrase/prompts.py"
click node_expansion "https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/5_RePhrase/expand.py"
click node_seed_release "https://github.com/dataphish/phishfuzzer/blob/main/PhishFuzzer_emails_original_seed_v1.json"
click node_expanded_release "https://github.com/dataphish/phishfuzzer/blob/main/PhishFuzzer_emails_entity_rephrased_v1.json"
click node_gemini_classifier "https://github.com/dataphish/phishfuzzer/blob/main/Classification/classify_Gemini.py"
click node_qwen_classifier "https://github.com/dataphish/phishfuzzer/blob/main/Classification/classify_Qwen.py"
click node_gemini_results "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_gemini.jsonl"
click node_qwen_results "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_qwen.jsonl"
click node_confusion_analysis "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/aggregate_confusion_matrix.py"
click node_provenance_analysis "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/private_public_original_rephrased.py"
click node_reliability_analysis "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/template_reliability_metrics.py"
click node_tfs_analysis "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/tfs_visual_matrices.py"
click node_tfs_percentage_analysis "https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/tfs_visual_matrices_percentage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_curated_seeds,node_motivation_labels,node_labeled_emails,node_metadata_population,node_populated_emails,node_normalization,node_normalized_emails toneBlue
class node_variant_prompts,node_expansion,node_seed_release,node_expanded_release toneAmber
class node_gemini_classifier,node_qwen_classifier,node_gemini_results,node_qwen_results,node_openrouter toneMint
class node_confusion_analysis,node_provenance_analysis,node_reliability_analysis,node_tfs_analysis,node_tfs_percentage_analysis toneRose
```

## Components

### Dataset preparation

- [Curated seed emails](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/1_Original_DataSet/emails_base.json): `DataSet_Creation/1_Original_DataSet/emails_base.json`
- [Motivation labeling](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/2_Motivation_Labeling/Motivaitonal_labeling.py): `DataSet_Creation/2_Motivation_Labeling/Motivaitonal_labeling.py`
- [Labeled emails](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/2_Motivation_Labeling/emails_labeled_gemini.json): `DataSet_Creation/2_Motivation_Labeling/emails_labeled_gemini.json`
- [Metadata population](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/3_Population/populate_prompt.py): `DataSet_Creation/3_Population/populate_prompt.py`
- [Populated emails](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/3_Population/emails_populated_non_manual_gemini.json): `DataSet_Creation/3_Population/emails_populated_non_manual_gemini.json`
- [Schema normalization](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/4_Normalize/normalize.py): `DataSet_Creation/4_Normalize/normalize.py`
- [Normalized emails](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/4_Normalize/emails_normalized.json): `DataSet_Creation/4_Normalize/emails_normalized.json`

### Variant generation

- [Variant prompts](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/5_RePhrase/prompts.py): `DataSet_Creation/5_RePhrase/prompts.py`
- [Seeded expansion](https://github.com/dataphish/phishfuzzer/blob/main/DataSet_Creation/5_RePhrase/expand.py): `DataSet_Creation/5_RePhrase/expand.py`
- [Seed dataset](https://github.com/dataphish/phishfuzzer/blob/main/PhishFuzzer_emails_original_seed_v1.json): `PhishFuzzer_emails_original_seed_v1.json`
- [Rephrased dataset](https://github.com/dataphish/phishfuzzer/blob/main/PhishFuzzer_emails_entity_rephrased_v1.json): `PhishFuzzer_emails_entity_rephrased_v1.json`

### Classification benchmark

- [Gemini classifier](https://github.com/dataphish/phishfuzzer/blob/main/Classification/classify_Gemini.py): `Classification/classify_Gemini.py`
- [Qwen classifier](https://github.com/dataphish/phishfuzzer/blob/main/Classification/classify_Qwen.py): `Classification/classify_Qwen.py`
- [Gemini results](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_gemini.jsonl): `Classification/Results/results_raw_gemini.jsonl`
- [Qwen results](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Results/results_raw_qwen.jsonl): `Classification/Results/results_raw_qwen.jsonl`

### Benchmark analysis

- [Confusion matrices](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/aggregate_confusion_matrix.py): `Classification/Analyze/aggregate_confusion_matrix.py`
- [Provenance comparison](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/private_public_original_rephrased.py): `Classification/Analyze/private_public_original_rephrased.py`
- [Reliability metrics](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/template_reliability_metrics.py): `Classification/Analyze/template_reliability_metrics.py`
- [TFS matrices](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/tfs_visual_matrices.py): `Classification/Analyze/tfs_visual_matrices.py`
- [TFS percentage matrices](https://github.com/dataphish/phishfuzzer/blob/main/Classification/Analyze/tfs_visual_matrices_percentage.py): `Classification/Analyze/tfs_visual_matrices_percentage.py`

### Other components

- **OpenRouter API**

## Connections

- Curated seed emails → Motivation labeling: labels intent (evidence: `ReadME.md`)
- Motivation labeling → Labeled emails: writes labels (evidence: `ReadME.md`)
- Labeled emails → Metadata population: enriches records (evidence: `ReadME.md`)
- Metadata population → OpenRouter API: queries model (evidence: `DataSet_Creation/3_Population/populate_prompt.py`)
- Metadata population → Populated emails: writes metadata (evidence: `ReadME.md`)
- Populated emails → Schema normalization: standardizes (evidence: `ReadME.md`)
- Schema normalization → Normalized emails: writes schema (evidence: `DataSet_Creation/4_Normalize/normalize.py`)
- Normalized emails → Seeded expansion: seeds variants (evidence: `ReadME.md`)
- Seeded expansion → Variant prompts: uses prompts (evidence: `DataSet_Creation/5_RePhrase/expand.py`)
- Seeded expansion → OpenRouter API: requests variants (evidence: `DataSet_Creation/5_RePhrase/expand.py`)
- Seeded expansion → Seed dataset: preserves seeds (evidence: `DataSet_Creation/5_RePhrase/expand.py`)
- Seeded expansion → Rephrased dataset: writes variants (evidence: `ReadME.md`)
- Seed dataset → Gemini classifier: classifies emails
- Rephrased dataset → Gemini classifier: classifies variants
- Seed dataset → Qwen classifier: classifies emails
- Rephrased dataset → Qwen classifier: classifies variants
- Gemini classifier → OpenRouter API: queries model (evidence: `Classification/classify_Gemini.py`)
- Qwen classifier → OpenRouter API: queries model (evidence: `Classification/classify_Qwen.py`)
- Gemini classifier → Gemini results: writes predictions (evidence: `Classification/classify_Gemini.py`)
- Qwen classifier → Qwen results: writes predictions (evidence: `Classification/classify_Qwen.py`)
- Gemini results → Confusion matrices: measures errors (evidence: `ReadME.md`)
- Qwen results → Confusion matrices: measures errors (evidence: `ReadME.md`)
- Gemini results → Provenance comparison: compares provenance (evidence: `Classification/Analyze/private_public_original_rephrased.py`)
- Qwen results → Provenance comparison: compares provenance (evidence: `Classification/Analyze/private_public_original_rephrased.py`)
- Gemini results → Reliability metrics: computes reliability (evidence: `ReadME.md`)
- Qwen results → Reliability metrics: computes reliability (evidence: `ReadME.md`)
- Gemini results → TFS matrices: analyzes failures (evidence: `Classification/Analyze/tfs_visual_matrices.py`)
- Qwen results → TFS matrices: analyzes failures (evidence: `Classification/Analyze/tfs_visual_matrices.py`)
- Gemini results → TFS percentage matrices: analyzes failures (evidence: `Classification/Analyze/tfs_visual_matrices_percentage.py`)
- Qwen results → TFS percentage matrices: analyzes failures (evidence: `Classification/Analyze/tfs_visual_matrices_percentage.py`)
