How preligens-lab/textnoisr works
Project Type & Purpose
• This is a Python package library intended for adding random noise to text datasets, with a key focus on precise noise calibration.
• The README explains its usage for various “noise” operations (insert, delete, substitute, swap) and emphasizes the swap unbiasing method, which is implemented with Markov Chains.
• With installation via pip, the project is designed for integration in data pre-processing pipelines for NLP tasks.
Key Components
a. Core Library (textnoisr Package):
• Module: noise.py — Contains functions for generic noise actions (inserting, deleting, substituting characters).
• Module: noise_dataset.py — Provides operations over datasets (applying noise transformation over collections of text).
• Module: noise_unbiasing.py — Houses specialized logic for the swap operation’s unbiasing, including the Markov Chain algorithm.
b. Supporting Scripts & Tools:
• Scripts directory: Contains helper scripts such as generate_figures.py (likely used for documentation or testing the noise effects) and natural_calibration.py (potentially a calibration utility).
c. Documentation and Testing:
• Docs directory: Provides detailed documentation, including API reference, tutorial, mechanism explanation, and swap unbiasing details.
• .github/workflows: Listing CI/CD workflows (build-doc, code-style, nightly-test, unit-test) ensuring code quality and continuous integration.
• Tests directory: Contains unit tests for noise functions and dataset operations.
• Configuration files: pyproject.toml, mkdocs.yml, and pre-commit-config.yaml that set up build, formatting, and deployment guidelines.
Relationships & Data Flow
• Flow from a user (or external system) invoking the textnoisr API.
• How input text data is processed through the noise_dataset functions, which in turn call core noise functions.
• Specifically detail the advanced swap operation: how noise.py interfaces with noise_unbiasing.py to handle the swap bias using Markov Chain logic.
Architectural Patterns & Design Principles
• Modular design: The separation into different Python modules (noise, noise_dataset, noise_unbiasing) clearly segregates concerns.
• Extensibility: The package is built as a modular library with a single calibration parameter controlling noise injection.
• Testing & CI: A strong emphasis on automated testing and documentation through designated GitHub workflows and configuration.
• Calibration Logic: Specific design around statistically unbiasing the swap action using Markov Chains highlights a non-trivial algorithm layered within the overall system.