How rub-syssec/nemo works
Project Type and Purpose
• This is a command-line, research-quality Python tool for building and applying Markov n-gram models to estimate password guessability (strength estimation, guessing, adaptive encoders). Not a web or GUI app, but a modular CLI pipeline with training and evaluation phases.
High-Level Architecture Components
a. Configuration Loader
• configs/configure.py + JSON files (dev.json, main.json)
• Reads experiment parameters (alphabet, ngram size, lengths, file names, CPU count, progress bar)
b. Data Sources
• input/training.txt – raw password distribution for training
• input/eval.txt – passwords to evaluate
c. Training Pipeline
• train.py – orchestrates multi-processing workers
• ngram/ngram_creator.py – builds three n-gram probability structures:
– IP: initial (n-1)-grams
– CP: conditional n-grams
– EP: ending (n-1)-grams
• Uses Python lists + custom n-gram ↔ list-index functions (_n2i, _i2n)
• Smoothing, count, probability calculation
d. Model Storage
• trained/*.pack – serialized IP/CP/EP lists via u-msgpack-python
• results/.keep and results/eval_result.txt for evaluation output
e. Evaluation (Strength Estimation) Pipeline
• meter.py – spawns threads that:
1. Load .pack files into memory via ngram_creator.load()
2. Read input/eval.txt, compute per-password probabilities (product of IP • CP^k • EP)
3. Write tab-separated results to results/eval_result.txt
f. Utilities
• utils/info.py – inspects password file lengths and alphabet
• utils/sortresult.py – sorts evaluation output
g. Logging
• log/multiprocessinglog.py + rainbow_logging_handler – colorful, multi-process-safe logging
h. External Dependencies
• tqdm – progress bars
• u-msgpack-python – model serialization
• virtual environment & runtime (PyPy recommended for performance)
Component Interactions and Data Flow
1. Config Loader → train.py & meter.py (injects parameters)
2. input/training.txt → train.py → ngram_creator → IP/CP/EP lists → trained/*.pack
3. trained/*.pack → meter.py → ngram_creator.load() → in-memory lists
4. input/eval.txt → meter.py → probability computation → results/eval_result.txt
5. utils/info.py and utils/sortresult.py used pre- and post- pipeline
6. Logging taps into train.py, meter.py, ngram_creator for status and errors
Architectural Patterns and Principles
• Modular separation of concerns: configuration, data I/O, processing (training vs. evaluation), serialization, logging, utilities
• Pipeline/batch processing with parallelism (multiprocessing for training, multithreading for evaluation)
• Memory-vs-compute trade-off: list-based storage vs. dicts for large n-gram sets
• Lightweight serialization (MessagePack) for fast load/save