How jjwang1118/segment_pw_cracking works
This repository is a password-guessing research pipeline: it converts leaked-password datasets into segmented, PCFG-tagged training examples, fine-tunes an LLM, then generates candidates and measures crack rates. The current PCFG-native path is the main route; BPE remains a documented legacy alternative. The graph retains both, plus the distinct generation and evaluation workflows. Relationships come from README workflow statements and the supplied source index; unsampled implementation details are not inferred beyond those references.
Main entry points: run_eval.py Evaluation runner · run_pcfg_segment.py PCFG-native pipeline · processData.py Clean passwords · train.py LLM and LoRA training
Components 23
Data preparation
- Raw datasets
datasets - Clean passwords
processData.py - Cleaning utilities
util/data.py
Segmentation and tagging
- BPE tokenizer training
src/BPE.py - BPE pipeline runner
trainBPE.py - BPE tokenization and tagging
src/Tokenize.py - PCFG-native pipeline
run_pcfg_segment.py - PCFG segmenter
src/PCFGSegment.py - Tag explanations
pcfg_tags.py - Dataset splitting
util/Dataprocess.py - Tagged train/test data
datasets/processed
Model training
- Training entry point
run_train.py - LLM and LoRA training
util/train.py - Training encoding
util/pw_tokenize.py - Prompt templates
src/prompt_template.py - Model checkpoint
Password generation
- Generation runner
run_search.py - Candidate search
util/search.py - Candidate passwords
Evaluation
- Evaluation runner
run_eval.py - Crack-rate results
Other components
- Researcher
- semantic-guesser
Connections 30
- Researcher → Clean passwords: starts pipeline
README.md - Raw datasets → Clean passwords: cleans data
README.md - Clean passwords → Cleaning utilities: uses utilities
processData.py - Clean passwords → BPE pipeline runner: feeds legacy path
README.md - BPE pipeline runner → BPE tokenizer training: trains tokenizer
trainBPE.py - BPE tokenizer training → BPE tokenization and tagging: provides tokenizer
README.md - BPE tokenization and tagging → semantic-guesser: tags tokens
README.md - Clean passwords → PCFG-native pipeline: feeds current path
README.md - PCFG-native pipeline → PCFG segmenter: segments passwords
run_pcfg_segment.py - PCFG-native pipeline → semantic-guesser: applies PCFG tags
README.md - PCFG-native pipeline → Dataset splitting: splits tagged data
README.md - PCFG-native pipeline → Tag explanations: uses tag definitions
- BPE tokenization and tagging → Dataset splitting: feeds legacy split
README.md - Dataset splitting → Tagged train/test data: writes datasets
README.md - Tagged train/test data → Training entry point: supplies training data
README.md - Training entry point → LLM and LoRA training: dispatches training
run_train.py - LLM and LoRA training → Training encoding: prepares examples
util/train.py - LLM and LoRA training → Prompt templates: uses prompt template
util/train.py - LLM and LoRA training → Model checkpoint: saves weights
README.md - Researcher → Generation runner: requests generation
README.md - Model checkpoint → Generation runner: loads model
README.md - Generation runner → Candidate search: dispatches search
run_search.py - Generation runner → Prompt templates: builds model prompt
run_search.py - Candidate search → Candidate passwords: writes candidates
- Researcher → Evaluation runner: requests evaluation
README.md - Tagged train/test data → Evaluation runner: supplies test data
README.md - Model checkpoint → Evaluation runner: loads model
README.md - Evaluation runner → Prompt templates: builds test prompt
run_eval.py - Evaluation runner → Candidate search: generates guesses
run_eval.py - Evaluation runner → Crack-rate results: records crack rates
README.md