# preligens-lab/textnoisr architecture

> Architecture diagram of the GitHub repository preligens-lab/textnoisr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/preligens-lab/textnoisr
- Repository: https://github.com/preligens-lab/textnoisr

## Overview

##### Project Type & Purpose
- This is a Python package library intended for adding random noise to text datasets, with a key focus on precise noise calibration.
- The README explains its usage for various “noise” operations (insert, delete, substitute, swap) and emphasizes the swap unbiasing method, which is implemented with Markov Chains.
- With installation via pip, the project is designed for integration in data pre-processing pipelines for NLP tasks.

##### Key Components
a. Core Library (textnoisr Package):
- Module: noise.py — Contains functions for generic noise actions (inserting, deleting, substituting characters).
- Module: noise_dataset.py — Provides operations over datasets (applying noise transformation over collections of text).
- Module: noise_unbiasing.py — Houses specialized logic for the swap operation’s unbiasing, including the Markov Chain algorithm.

b. Supporting Scripts & Tools:
- Scripts directory: Contains helper scripts such as generate_figures.py (likely used for documentation or testing the noise effects) and natural_calibration.py (potentially a calibration utility).

c. Documentation and Testing:
- Docs directory: Provides detailed documentation, including API reference, tutorial, mechanism explanation, and swap unbiasing details.
- .github/workflows: Listing CI/CD workflows (build-doc, code-style, nightly-test, unit-test) ensuring code quality and continuous integration.
- Tests directory: Contains unit tests for noise functions and dataset operations.
- Configuration files: pyproject.toml, mkdocs.yml, and pre-commit-config.yaml that set up build, formatting, and deployment guidelines.

##### Relationships & Data Flow
- Flow from a user (or external system) invoking the textnoisr API.
- How input text data is processed through the noise_dataset functions, which in turn call core noise functions.
- Specifically detail the advanced swap operation: how noise.py interfaces with noise_unbiasing.py to handle the swap bias using Markov Chain logic.

##### Architectural Patterns & Design Principles
- Modular design: The separation into different Python modules (noise, noise_dataset, noise_unbiasing) clearly segregates concerns.
- Extensibility: The package is built as a modular library with a single calibration parameter controlling noise injection.
- Testing & CI: A strong emphasis on automated testing and documentation through designated GitHub workflows and configuration.
- Calibration Logic: Specific design around statistically unbiasing the swap action using Markov Chains highlights a non-trivial algorithm layered within the overall system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entity
    User["User/External System"]:::external

    %% Core Library Subgraph
    subgraph "Core Library (textnoisr Package)"
        dataset["noise_dataset.py"]:::core
        noise["noise.py"]:::core
        unbiasing["noise_unbiasing.py - Markov Chain"]:::core
    end

    %% Supporting Scripts Subgraph
    subgraph "Supporting Scripts"
        figures["generate_figures.py"]:::support
        calibration["natural_calibration.py"]:::support
    end

    %% Documentation Subgraph
    subgraph "Documentation"
        docs["Docs (API, Tutorial, etc.)"]:::doc
    end

    %% CI & Testing Subgraph
    subgraph "CI & Testing"
        ci["CI Workflows"]:::ci
        tests["Unit Tests"]:::ci
    end

    %% Configuration Files Subgraph
    subgraph "Configuration"
        config["Configuration Files"]:::config
    end

    %% Relationships & Data Flow
    User -->|"invoke"| dataset
    dataset -->|"calls noise"| noise
    noise -->|"swap calls unbiasing"| unbiasing
    calibration -->|"sets calibration"| dataset
    ci -->|"triggers docs generation"| docs

    %% Click Events
    click noise "https://github.com/preligens-lab/textnoisr/blob/main/textnoisr/noise.py"
    click dataset "https://github.com/preligens-lab/textnoisr/blob/main/textnoisr/noise_dataset.py"
    click unbiasing "https://github.com/preligens-lab/textnoisr/blob/main/textnoisr/noise_unbiasing.py"
    click figures "https://github.com/preligens-lab/textnoisr/blob/main/scripts/generate_figures.py"
    click calibration "https://github.com/preligens-lab/textnoisr/blob/main/scripts/natural_calibration.py"
    click docs "https://github.com/preligens-lab/textnoisr/tree/main/docs/"
    click ci "https://github.com/preligens-lab/textnoisr/tree/main/.github/workflows/"
    click tests "https://github.com/preligens-lab/textnoisr/tree/main/tests/"
    click config "https://github.com/preligens-lab/textnoisr/blob/main/pyproject.toml"

    %% Styles
    classDef core fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef support fill:#d6eaf8,stroke:#3498db,stroke-width:2px;
    classDef doc fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef ci fill:#f5cba7,stroke:#e67e22,stroke-width:2px;
    classDef config fill:#f0b27a,stroke:#e74c3c,stroke-width:2px;
    classDef external fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
```
