# rub-syssec/nemo architecture

> Architecture diagram of the GitHub repository rub-syssec/nemo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rub-syssec/nemo
- Repository: https://github.com/rub-syssec/nemo

## Overview

##### Project Type and Purpose
- This is a command-line, research-quality Python tool for building and applying Markov n-gram models to estimate password guessability (strength estimation, guessing, adaptive encoders). Not a web or GUI app, but a modular CLI pipeline with training and evaluation phases.

##### High-Level Architecture Components
a. Configuration Loader
- configs/configure.py + JSON files (dev.json, main.json)
- Reads experiment parameters (alphabet, ngram size, lengths, file names, CPU count, progress bar)
b. Data Sources
- input/training.txt – raw password distribution for training
- input/eval.txt – passwords to evaluate
c. Training Pipeline
- train.py – orchestrates multi-processing workers
- ngram/ngram_creator.py – builds three n-gram probability structures:
– IP: initial (n-1)-grams
– CP: conditional n-grams
– EP: ending (n-1)-grams
- Uses Python lists + custom n-gram ↔ list-index functions (_n2i, _i2n)
- Smoothing, count, probability calculation
d. Model Storage
- trained/*.pack – serialized IP/CP/EP lists via u-msgpack-python
- results/.keep and results/eval_result.txt for evaluation output
e. Evaluation (Strength Estimation) Pipeline
- meter.py – spawns threads that:
1. Load .pack files into memory via ngram_creator.load()
2. Read input/eval.txt, compute per-password probabilities (product of IP • CP^k • EP)
3. Write tab-separated results to results/eval_result.txt
f. Utilities
- utils/info.py – inspects password file lengths and alphabet
- utils/sortresult.py – sorts evaluation output
g. Logging
- log/multiprocessinglog.py + rainbow_logging_handler – colorful, multi-process-safe logging
h. External Dependencies
- tqdm – progress bars
- u-msgpack-python – model serialization
- virtual environment & runtime (PyPy recommended for performance)

##### Component Interactions and Data Flow
1. Config Loader → train.py & meter.py (injects parameters)
2. input/training.txt → train.py → ngram_creator → IP/CP/EP lists → trained/*.pack
3. trained/*.pack → meter.py → ngram_creator.load() → in-memory lists
4. input/eval.txt → meter.py → probability computation → results/eval_result.txt
5. utils/info.py and utils/sortresult.py used pre- and post- pipeline
6. Logging taps into train.py, meter.py, ngram_creator for status and errors

##### Architectural Patterns and Principles
- Modular separation of concerns: configuration, data I/O, processing (training vs. evaluation), serialization, logging, utilities
- Pipeline/batch processing with parallelism (multiprocessing for training, multithreading for evaluation)
- Memory-vs-compute trade-off: list-based storage vs. dicts for large n-gram sets
- Lightweight serialization (MessagePack) for fast load/save

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration
    subgraph "Configuration" 
        ConfigLoader["Configuration Loader\n(configs/configure.py)"]:::io
    end

    %% Data Sources
    Input["Data Sources\ninput/"]:::io

    %% Training Phase
    subgraph "Training Phase"
        Train["train.py\n(Orchestrator)"]:::processing
        NGramBuilder["NGramCreator\n(ngram/ngram_creator.py)"]:::processing
        ModelStorage["Model Storage\ntrained/"]:::storage
    end

    %% Evaluation Phase
    subgraph "Evaluation Phase"
        Meter["meter.py\n(Orchestrator)"]:::processing
        ModelLoad["Load .pack via ngram_creator.load()"]:::processing
        EvalResults["Evaluation Output\nresults/eval_result.txt"]:::storage
    end

    %% Utilities
    subgraph "Utilities"
        Info["File Inspection\n(utils/info.py)"]:::utility
        SortResult["Result Sorting\n(utils/sortresult.py)"]:::utility
    end

    %% Logging
    Logging["Logging Module\n(log/multiprocessinglog.py)"]:::logging

    %% External Dependencies
    subgraph "External Dependencies"
        PyPy["PyPy Runtime"]:::external
        TQDM["tqdm"]:::external
        MsgPack["u-msgpack-python"]:::external
    end

    %% Data Flow
    ConfigLoader -->|"parameters"| Train
    ConfigLoader -->|"parameters"| Meter

    Input -->|"training.txt"| Train
    Train -->|"counts & probs"| NGramBuilder
    NGramBuilder -->|"IP/CP/EP lists"| ModelStorage

    ModelStorage -->|".pack files"| Meter
    Meter -->|"loads models"| ModelLoad
    Input -->|"eval.txt"| Meter
    ModelLoad -->|"probabilities per password"| Meter
    Meter -->|"tab-separated results"| EvalResults

    Info -->|pre-pipeline| Train
    Info -->|pre-pipeline| Meter
    SortResult -->|post-pipeline| EvalResults

    Train -->|logs status/errors| Logging
    Meter -->|logs status/errors| Logging
    NGramBuilder -->|logs status/errors| Logging

    %% External usage
    Train -.->|uses| PyPy
    Meter -.->|uses| PyPy
    Train -.->|progress bars| TQDM
    NGramBuilder -.->|serializes| MsgPack
    Meter -.->|loads via| MsgPack

    %% Click Events
    click ConfigLoader "https://github.com/rub-syssec/nemo/blob/master/configs/configure.py"
    click Input "https://github.com/rub-syssec/nemo/tree/master/input/"
    click Train "https://github.com/rub-syssec/nemo/blob/master/train.py"
    click NGramBuilder "https://github.com/rub-syssec/nemo/blob/master/ngram/ngram_creator.py"
    click ModelStorage "https://github.com/rub-syssec/nemo/tree/master/trained/"
    click Meter "https://github.com/rub-syssec/nemo/blob/master/meter.py"
    click EvalResults "https://github.com/rub-syssec/nemo/blob/master/results/eval_result.txt"
    click Info "https://github.com/rub-syssec/nemo/blob/master/utils/info.py"
    click SortResult "https://github.com/rub-syssec/nemo/blob/master/utils/sortresult.py"
    click Logging "https://github.com/rub-syssec/nemo/blob/master/log/multiprocessinglog.py"

    %% Styles
    classDef processing fill:#add8e6,stroke:#000,stroke-width:1px
    classDef storage fill:#90ee90,stroke:#000,stroke-width:1px
    classDef io fill:#ffff99,stroke:#000,stroke-width:1px
    classDef utility fill:#ffcc99,stroke:#000,stroke-width:1px
    classDef logging fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef external fill:#dda0dd,stroke:#000,stroke-width:1px
```
