# carolqian31/fepguesser architecture

> Architecture diagram of the GitHub repository carolqian31/fepguesser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/carolqian31/fepguesser
- Repository: https://github.com/carolqian31/fepguesser

## Overview

##### Project Type and Purpose
- FEPGuesser is a single codebase for a machine-learning-driven password guessing system. It is composed of three tightly related components:
- PSVG: a password segmentation and wordlist generation module
- PassExBert: a BERT-based masked language model pretraining module that incorporates custom mixed vocabularies
- PassExBertVAE: a variational autoencoder module built on top of the pre-trained BERT embeddings to generate candidate passwords and perform attack experiments
- The system’s primary goal is to take raw password corpora, segment them, build a richer vocabulary, train a language model, refine it with a VAE, and then sample guess candidates.

##### File-Structure Analysis
- Top-level directories:
- PSVG: scripts and submodule for segmentation, rules for detecting tokens, and utilities to produce segmented output
- passExBert: building mixed vocab from segments, ExBERT model code (tokenization, modeling, optimization), a pretraining driver, plus configuration and vocabulary storage
- passExBertVAE: VAE wrapper modules (encoder, decoder, sampler), training and sampling scripts, parameter utilities, and experiment code
- Configuration files and vocab reside under config_and_vocab and bert_config_and_vocab, respectively
- requirements.txt lists external dependencies (PyTorch, Huggingface transformers, etc.)
- The workflow is driven by CLI scripts that chain data files, config JSONs, pretrained BERT weights, and output storage folders.

##### Key Components and Relationships
a. Data Input
- Raw password datasets (plain text files)
- Pretrained BERT weights from Huggingface
b. PSVG Module
- segmntr.py: segments passwords into subword tokens
- get_wordlist_from_segment.py: filters segments by frequency threshold
- detection_rules: custom rules for segment detection
→ Output: a wordlist.txt
c. Vocabulary Builder
- build_pass_vocab_from_word_list.py: merges custom wordlist with original BERT vocab
→ Output: mixed_vocab.txt
d. PassExBert Pretraining
- pretraining_pass.py: fine-tunes BERT on password data with masks
- exBERT subpackage: tokenization, modeling, optimization routines
- Output: trained PassExBert checkpoint and dict file
e. PassExBertVAE Training
- train_chunk.py: loads PassExBert embeddings, encodes sequences into latent space, decodes them, applies VAE losses
- selfModules & model subpackages: encoder, decoder, VAE wrapper
- Output: VAE checkpoints
f. Sampling and Attack
- PassExBert_sample.py: samples from VAE latent space using beam search or Gaussian noise, generates password guesses
- guess.py: utilities for scoring or testing guesses
g. Storage and Outputs
- Storage folders (./storage and data/...) for model checkpoints, logs, and generated samples
- Logger utilities under utils/

##### Architectural Patterns and Design Principles
- Modular ML pipeline: each stage (segmentation, vocab, pretraining, VAE) is a standalone CLI that takes inputs and produces outputs for the next stage
- Use of configuration files (JSON) to parameterize each stage, enabling reproducibility
- Separation of concerns: PSVG handles data preprocessing, PassExBert handles language modeling, PassExBertVAE handles generative modeling
- Extensibility: detection_rules, selfModules, and config_and_vocab directories show plug-and-play pattern

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Data Sources and External Dependencies
    RawData(["Raw Password Datasets"]):::external
    HFWeights(["Pretrained BERT Weights"]):::external

    %% Layer 2: PSVG Preprocessing Module
    subgraph "PSVG Preprocessing" 
        direction TB
        Segmentation["segmntr.py"]:::preprocess
        WordlistGen["get_wordlist_from_segment.py"]:::preprocess
        DetectionRules["detection_rules/"]:::preprocess
    end

    WordlistStore(["wordlist.txt"]):::store

    %% Layer 3: Vocabulary Builder
    subgraph "Vocabulary Builder" 
        direction TB
        BuildVocab["build_pass_vocab_from_word_list.py"]:::preprocess
        VocabConfig["config_and_vocab/"]:::preprocess
    end

    MixedVocabStore(["mixed_vocab.txt"]):::store

    %% Layer 4: PassExBert Pretraining Module
    subgraph "PassExBert Pretraining" 
        direction TB
        PretrainDriver["pretraining_pass.py"]:::model
        subgraph "exBERT Subpackage"
            direction TB
            Tokenization["tokenization.py"]:::model
            Modeling["modeling.py"]:::model
            Optimization["optimization.py"]:::model
            FileUtils["file_utils.py"]:::model
        end
        subgraph "Utils"
            direction TB
            PELogger["Logger.py"]:::model
            PEHelpers["pretraining_pass.py"]:::model
        end
        BaseVocab["bert_base_cased/"]:::store
    end

    PassExBertCheckpoint(["PassExBert Checkpoint (.pt)"]):::store
    PassExBertDict(["PassExBert Dict"]):::store

    %% Layer 5: PassExBertVAE Training Module
    subgraph "PassExBertVAE Training" 
        direction TB
        VAETrain["train_chunk.py"]:::gen
        subgraph "Model Components"
            direction TB
            Encoder["passExBert_encoder.py"]:::gen
            Decoder["passExBert_decoder.py"]:::gen
            VAEWrapper["passexbert_rvae.py"]:::gen
        end
        SelfEmbed["bert_embedding.py"]:::gen
        subgraph "VAE Utils"
            direction TB
            BeamSearch["beam_search.py"]:::gen
            SampleMethod["sample_method.py"]:::gen
            Parameters["parameters.py"]:::gen
            DataLoader["passExBert_data_loader.py"]:::gen
            Functional["functional.py"]:::gen
            VAELogger["Logger.py"]:::gen
        end
        VAEConfig["bert_config_and_vocab/"]:::gen
    end

    VAECheckpoint(["VAE Checkpoint (.pt)"]):::store

    %% Layer 6: Sampling and Attack Module
    subgraph "Sampling & Attack"
        direction TB
        SampleMain["PassExBert_sample.py"]:::gen
        SampleCross["PassExBert_sample_cross_attack.py"]:::gen
        GuessUtils["guess.py"]:::gen
    end

    %% Layer 7: Storage
    StorageFolders(["./storage and data/..."]):::store

    %% Connections
    RawData --> Segmentation
    Segmentation --> WordlistGen
    Segmentation --> DetectionRules
    WordlistGen --> WordlistStore
    WordlistStore --> BuildVocab
    VocabConfig --> BuildVocab
    BuildVocab --> MixedVocabStore
    MixedVocabStore --> PretrainDriver
    HFWeights --> PretrainDriver
    BaseVocab --> PretrainDriver
    PretrainDriver --> PassExBertCheckpoint
    PretrainDriver --> PassExBertDict
    PassExBertCheckpoint --> VAETrain
    VAETrain --> VAECheckpoint
    VAECheckpoint --> SampleMain
    SampleMain --> StorageFolders
    SampleCross --> StorageFolders
    GuessUtils --> StorageFolders

    %% Click Events
    click Segmentation "https://github.com/carolqian31/fepguesser/blob/main/PSVG/segmntr.py"
    click WordlistGen "https://github.com/carolqian31/fepguesser/blob/main/PSVG/get_wordlist_from_segment.py"
    click DetectionRules "https://github.com/carolqian31/fepguesser/tree/main/PSVG/lib_trainer/detection_rules/"
    click BuildVocab "https://github.com/carolqian31/fepguesser/blob/main/passExBert/build_pass_vocab_from_word_list.py"
    click VocabConfig "https://github.com/carolqian31/fepguesser/tree/main/passExBert/config_and_vocab/"
    click PretrainDriver "https://github.com/carolqian31/fepguesser/blob/main/passExBert/pretraining_pass.py"
    click Tokenization "https://github.com/carolqian31/fepguesser/blob/main/passExBert/exBERT/tokenization.py"
    click Modeling "https://github.com/carolqian31/fepguesser/blob/main/passExBert/exBERT/modeling.py"
    click Optimization "https://github.com/carolqian31/fepguesser/blob/main/passExBert/exBERT/optimization.py"
    click FileUtils "https://github.com/carolqian31/fepguesser/blob/main/passExBert/exBERT/file_utils.py"
    click PELogger "https://github.com/carolqian31/fepguesser/blob/main/passExBert/utils/Logger.py"
    click PEHelpers "https://github.com/carolqian31/fepguesser/blob/main/passExBert/utils/pretraining_pass.py"
    click BaseVocab "https://github.com/carolqian31/fepguesser/tree/main/passExBert/bert_base_cased/"
    click VAETrain "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/train_chunk.py"
    click Encoder "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/model/passExBert_encoder.py"
    click Decoder "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/model/passExBert_decoder.py"
    click VAEWrapper "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/model/passexbert_rvae.py"
    click SelfEmbed "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/selfModules/bert_embedding.py"
    click BeamSearch "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/beam_search.py"
    click SampleMethod "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/sample_method.py"
    click Parameters "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/parameters.py"
    click DataLoader "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/passExBert_data_loader.py"
    click Functional "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/functional.py"
    click VAELogger "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/utils/Logger.py"
    click VAEConfig "https://github.com/carolqian31/fepguesser/tree/main/passExBertVAE/bert_config_and_vocab/"
    click SampleMain "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/PassExBert_sample.py"
    click SampleCross "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/PassExBert_sample_cross_attack.py"
    click GuessUtils "https://github.com/carolqian31/fepguesser/blob/main/passExBertVAE/guess.py"

    %% Styles
    classDef external fill:#D3D3F9,stroke:#333,stroke-width:1px;
    classDef preprocess fill:#BBDEFB,stroke:#0D47A1,stroke-width:1px;
    classDef model fill:#C8E6C9,stroke:#2E7D32,stroke-width:1px;
    classDef gen fill:#FFE0B2,stroke:#EF6C00,stroke-width:1px;
    classDef store fill:#E0E0E0,stroke:#616161,stroke-width:1px;
```
