# jjwang1118/segment_pw_cracking architecture

> Architecture diagram of the GitHub repository jjwang1118/segment_pw_cracking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jjwang1118/segment_pw_cracking
- Repository: https://github.com/jjwang1118/segment_pw_cracking

## Overview

This repository is a password-guessing research pipeline: it converts leaked-password datasets into segmented, PCFG-tagged training examples, fine-tunes an LLM, then generates candidates and measures crack rates. The current PCFG-native path is the main route; BPE remains a documented legacy alternative. The graph retains both, plus the distinct generation and evaluation workflows. Relationships come from README workflow statements and the supplied source index; unsampled implementation details are not inferred beyond those references.

Main entry points: [`run_eval.py`](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_eval.py) (Evaluation runner), [`run_pcfg_segment.py`](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_pcfg_segment.py) (PCFG-native pipeline), [`processData.py`](https://github.com/jjwang1118/segment_pw_cracking/blob/main/processData.py) (Clean passwords), [`util/train.py`](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/train.py) (LLM and LoRA training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_raw["Raw datasets"]
  node_clean["Clean passwords<br/>[processData.py]"]
  node_cleanlib["Cleaning utilities<br/>[data.py]"]
end

subgraph group_segment["Segmentation and tagging"]
  node_bpe_train["BPE tokenizer training<br/>[BPE.py]"]
  node_bpe_route["BPE pipeline runner<br/>[trainBPE.py]"]
  node_bpe_tag["BPE tokenization and tagging<br/>[Tokenize.py]"]
  node_pcfg_route["PCFG-native pipeline"]
  node_pcfg_segment["PCFG segmenter<br/>[PCFGSegment.py]"]
  node_tagmean["Tag explanations<br/>[pcfg_tags.py]"]
  node_split["Dataset splitting<br/>[Dataprocess.py]"]
  node_processed["Tagged train/test data"]
end

subgraph group_model["Model training"]
  node_train_entry["Training entry point<br/>[run_train.py]"]
  node_train["LLM and LoRA training<br/>[train.py]"]
  node_encode["Training encoding<br/>[pw_tokenize.py]"]
  node_prompts["Prompt templates<br/>[prompt_template.py]"]
  node_checkpoint["Model checkpoint"]
end

subgraph group_generate["Password generation"]
  node_search_entry["Generation runner<br/>[run_search.py]"]
  node_search["Candidate search<br/>[search.py]"]
  node_candidates["Candidate passwords"]
end

subgraph group_evaluate["Evaluation"]
  node_eval_entry["Evaluation runner<br/>[run_eval.py]"]
  node_crack_results["Crack-rate results"]
end

node_user(("Researcher"))
node_external_tagger["semantic-guesser"]

node_user -->|"starts pipeline"| node_clean
node_raw -->|"cleans data"| node_clean
node_clean -->|"uses utilities"| node_cleanlib
node_clean -->|"feeds legacy path"| node_bpe_route
node_bpe_route -->|"trains tokenizer"| node_bpe_train
node_bpe_train -->|"provides tokenizer"| node_bpe_tag
node_bpe_tag -->|"tags tokens"| node_external_tagger
node_clean -->|"feeds current path"| node_pcfg_route
node_pcfg_route -->|"segments passwords"| node_pcfg_segment
node_pcfg_route -->|"applies PCFG tags"| node_external_tagger
node_pcfg_route -->|"splits tagged data"| node_split
node_pcfg_route -.->|"uses tag definitions"| node_tagmean
node_bpe_tag -->|"feeds legacy split"| node_split
node_split -->|"writes datasets"| node_processed
node_processed -->|"supplies training data"| node_train_entry
node_train_entry -->|"dispatches training"| node_train
node_train -->|"prepares examples"| node_encode
node_train -->|"uses prompt template"| node_prompts
node_train -->|"saves weights"| node_checkpoint
node_user -->|"requests generation"| node_search_entry
node_checkpoint -->|"loads model"| node_search_entry
node_search_entry -->|"dispatches search"| node_search
node_search_entry -->|"builds model prompt"| node_prompts
node_search -->|"writes candidates"| node_candidates
node_user -->|"requests evaluation"| node_eval_entry
node_processed -->|"supplies test data"| node_eval_entry
node_checkpoint -->|"loads model"| node_eval_entry
node_eval_entry -->|"builds test prompt"| node_prompts
node_eval_entry -->|"generates guesses"| node_search
node_eval_entry -->|"records crack rates"| node_crack_results

click node_raw "https://github.com/jjwang1118/segment_pw_cracking/tree/main/datasets"
click node_clean "https://github.com/jjwang1118/segment_pw_cracking/blob/main/processData.py"
click node_cleanlib "https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/data.py"
click node_bpe_train "https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/BPE.py"
click node_bpe_route "https://github.com/jjwang1118/segment_pw_cracking/blob/main/trainBPE.py"
click node_bpe_tag "https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/Tokenize.py"
click node_pcfg_route "https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_pcfg_segment.py"
click node_pcfg_segment "https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/PCFGSegment.py"
click node_tagmean "https://github.com/jjwang1118/segment_pw_cracking/blob/main/pcfg_tags.py"
click node_split "https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/Dataprocess.py"
click node_processed "https://github.com/jjwang1118/segment_pw_cracking/tree/main/datasets/processed"
click node_train_entry "https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_train.py"
click node_train "https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/train.py"
click node_encode "https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/pw_tokenize.py"
click node_prompts "https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/prompt_template.py"
click node_search_entry "https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_search.py"
click node_search "https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/search.py"
click node_eval_entry "https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw,node_clean,node_cleanlib toneBlue
class node_bpe_train,node_bpe_route,node_bpe_tag,node_pcfg_route,node_pcfg_segment,node_tagmean,node_split,node_processed toneAmber
class node_train_entry,node_train,node_encode,node_prompts,node_checkpoint toneMint
class node_search_entry,node_search,node_candidates toneRose
class node_eval_entry,node_crack_results,node_user,node_external_tagger toneIndigo
```

## Components

### Data preparation

- [Raw datasets](https://github.com/jjwang1118/segment_pw_cracking/tree/main/datasets): `datasets`
- [Clean passwords](https://github.com/jjwang1118/segment_pw_cracking/blob/main/processData.py): `processData.py`
- [Cleaning utilities](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/data.py): `util/data.py`

### Segmentation and tagging

- [BPE tokenizer training](https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/BPE.py): `src/BPE.py`
- [BPE pipeline runner](https://github.com/jjwang1118/segment_pw_cracking/blob/main/trainBPE.py): `trainBPE.py`
- [BPE tokenization and tagging](https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/Tokenize.py): `src/Tokenize.py`
- [PCFG-native pipeline](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_pcfg_segment.py): `run_pcfg_segment.py`
- [PCFG segmenter](https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/PCFGSegment.py): `src/PCFGSegment.py`
- [Tag explanations](https://github.com/jjwang1118/segment_pw_cracking/blob/main/pcfg_tags.py): `pcfg_tags.py`
- [Dataset splitting](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/Dataprocess.py): `util/Dataprocess.py`
- [Tagged train/test data](https://github.com/jjwang1118/segment_pw_cracking/tree/main/datasets/processed): `datasets/processed`

### Model training

- [Training entry point](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_train.py): `run_train.py`
- [LLM and LoRA training](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/train.py): `util/train.py`
- [Training encoding](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/pw_tokenize.py): `util/pw_tokenize.py`
- [Prompt templates](https://github.com/jjwang1118/segment_pw_cracking/blob/main/src/prompt_template.py): `src/prompt_template.py`
- **Model checkpoint**

### Password generation

- [Generation runner](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_search.py): `run_search.py`
- [Candidate search](https://github.com/jjwang1118/segment_pw_cracking/blob/main/util/search.py): `util/search.py`
- **Candidate passwords**

### Evaluation

- [Evaluation runner](https://github.com/jjwang1118/segment_pw_cracking/blob/main/run_eval.py): `run_eval.py`
- **Crack-rate results**

### Other components

- **Researcher**
- **semantic-guesser**

## Connections

- Researcher → Clean passwords: starts pipeline (evidence: `README.md`)
- Raw datasets → Clean passwords: cleans data (evidence: `README.md`)
- Clean passwords → Cleaning utilities: uses utilities (evidence: `processData.py`)
- Clean passwords → BPE pipeline runner: feeds legacy path (evidence: `README.md`)
- BPE pipeline runner → BPE tokenizer training: trains tokenizer (evidence: `trainBPE.py`)
- BPE tokenizer training → BPE tokenization and tagging: provides tokenizer (evidence: `README.md`)
- BPE tokenization and tagging → semantic-guesser: tags tokens (evidence: `README.md`)
- Clean passwords → PCFG-native pipeline: feeds current path (evidence: `README.md`)
- PCFG-native pipeline → PCFG segmenter: segments passwords (evidence: `run_pcfg_segment.py`)
- PCFG-native pipeline → semantic-guesser: applies PCFG tags (evidence: `README.md`)
- PCFG-native pipeline → Dataset splitting: splits tagged data (evidence: `README.md`)
- PCFG-native pipeline → Tag explanations: uses tag definitions
- BPE tokenization and tagging → Dataset splitting: feeds legacy split (evidence: `README.md`)
- Dataset splitting → Tagged train/test data: writes datasets (evidence: `README.md`)
- Tagged train/test data → Training entry point: supplies training data (evidence: `README.md`)
- Training entry point → LLM and LoRA training: dispatches training (evidence: `run_train.py`)
- LLM and LoRA training → Training encoding: prepares examples (evidence: `util/train.py`)
- LLM and LoRA training → Prompt templates: uses prompt template (evidence: `util/train.py`)
- LLM and LoRA training → Model checkpoint: saves weights (evidence: `README.md`)
- Researcher → Generation runner: requests generation (evidence: `README.md`)
- Model checkpoint → Generation runner: loads model (evidence: `README.md`)
- Generation runner → Candidate search: dispatches search (evidence: `run_search.py`)
- Generation runner → Prompt templates: builds model prompt (evidence: `run_search.py`)
- Candidate search → Candidate passwords: writes candidates
- Researcher → Evaluation runner: requests evaluation (evidence: `README.md`)
- Tagged train/test data → Evaluation runner: supplies test data (evidence: `README.md`)
- Model checkpoint → Evaluation runner: loads model (evidence: `README.md`)
- Evaluation runner → Prompt templates: builds test prompt (evidence: `run_eval.py`)
- Evaluation runner → Candidate search: generates guesses (evidence: `run_eval.py`)
- Evaluation runner → Crack-rate results: records crack rates (evidence: `README.md`)
