# rspeer/python-ftfy architecture

> Architecture diagram of the GitHub repository rspeer/python-ftfy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rspeer/python-ftfy
- Repository: https://github.com/rspeer/python-ftfy

## Overview

##### Project type and purpose
- This is an open-source Python library/CLI tool (ftfy: “fixes text for you”) for detecting and correcting mojibake and other text‐encoding problems.
- The README describes a programmatic API (`fix_text`, `fix_encoding`), a CLI entry point (`ftfy/cli.py`), and extensive documentation and test coverage.

##### File structure
- Top-level directories:
- ftfy/       → core library modules
- docs/       → Sphinx documentation sources
- tests/      → pytest test suite and test-case data
- scripts/    → auxiliary code (e.g. data-table generator)
- notebook/   → presentation notebook
- Configuration/build/deployment files:
- pyproject.toml, MANIFEST.in, setup in .github/workflows/publish.yml
- pytest.ini, mypy.ini, tox.ini, .readthedocs.yaml
- Data modules under `ftfy/bad_codecs/` and `ftfy/chardata.py` hint at a data-driven design.

##### Key insights from the README
- Core functionality: detect and fix encoding mix-ups, HTML entities, curly quotes, etc.
- Describes API usage, CLI usage, configuration, and citations.
- Mentions documentation on ReadTheDocs and package publishing to PyPI.

##### Main components and their interactions
a. Components:
1. CLI layer (ftfy/cli.py)
2. Core API modules:
- fix logic (`ftfy/fixes.py`, `ftfy/badness.py`)
- formatting (`ftfy/formatting.py`)
- character data (`ftfy/chardata.py`)
- bad-codec handling (`ftfy/bad_codecs/*.py`)
3. Data assets: JSON/text files in tests/test-cases
4. Test suite (pytest)
5. Documentation (Sphinx: docs/, docs/conf.py)
6. Packaging & CI (GitHub Actions, pyproject.toml)
7. Auxiliary scripts (scripts/char_data_table.py)
8. External services:
- PyPI (package distribution)
- ReadTheDocs (docs hosting)
- GitHub (source, CI)

b. Relationships and data flows:
- The CLI invokes the core API, passing user input to `fix_text` or `fix_encoding`.
- Core API modules collaborate:
- Input passes through detection/heuristics (`badness.py`).
- Character mapping and codec heuristics in `chardata.py` and `bad_codecs/`.
- Corrections applied in `fixes.py` and `formatting.py`.
- Scripts generate data artifacts used in docs or tests.
- Tests exercise both the CLI and API layers against test-case data.
- Documentation is generated from `docs/`, referencing code comments and examples.
- CI (GitHub Actions) builds, tests, and publishes to PyPI on tag.

c. Architectural patterns and principles:
- Modular plugin pattern for “bad codecs” – new codecs = new module.
- Data-driven design: large characterization tables live in code/data modules.
- Clear separation: API core vs. CLI vs. data vs. docs vs. tests.
- Use of standard Python packaging (PEP 517/518, pyproject.toml).

d. Key technologies and frameworks:
- Python 3 (PEP 484 typing, `py.typed`)
- pytest for testing
- Sphinx for documentation
- uv (task runner)
- GitHub Actions for CI/CD
- Publishing to PyPI, hosting docs on ReadTheDocs

##### Detailed capture of essential elements
- Do not over-simplify: show each module grouping (fixes, formatting, bad_codecs, chardata).
- Test data flow: test suite consumes library + test-case JSON.
- Documentation pipeline: docs sources + scripts → HTML on ReadTheDocs.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI["CLI: ftfy/cli.py"]:::cli
    end

    %% Core API
    subgraph "Core API" 
        direction TB
        Fixes["fixes.py"]:::core
        Badness["badness.py"]:::core
        Formatting["formatting.py"]:::core
        CharData["chardata.py"]:::core
        subgraph "Bad-Codec Plugins"
            direction TB
            BadCodecs["ftfy/bad_codecs/"]:::plugin
        end
    end

    %% Data Modules
    subgraph "Data Assets"
        direction TB
        DataAssets["Test-case JSON files"]:::data
    end

    %% Scripts
    subgraph "Auxiliary Scripts"
        direction TB
        Script["char_data_table.py"]:::script
    end

    %% Tests
    subgraph "Test Suite"
        direction TB
        Tests["pytest tests/*.py"]:::test
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        Docs["Sphinx sources (docs/)"]:::docs
    end

    %% Packaging & CI
    subgraph "Packaging & CI"
        direction TB
        BuildConfig["pyproject.toml / MANIFEST.in"]:::ops
        CIWorkflow[".github/workflows/publish.yml"]:::ops
        RTDConfig[".readthedocs.yaml"]:::ops
    end

    %% External Services
    subgraph "External Services"
        direction TB
        GitHub["GitHub"]:::external
        PyPI["PyPI"]:::external
        ReadTheDocs["ReadTheDocs"]:::external
    end

    %% Relationships
    CLI -->|"invokes"| Fixes
    CLI --> Fixes
    Fixes -->|"uses heuristics"| Badness
    Fixes -->|"applies formatting"| Formatting
    Fixes -->|"maps characters"| CharData
    Fixes -->|"loads plugins"| BadCodecs

    Script -->|"generates data"| Docs
    Script --> DataAssets

    Tests -->|"load test data"| DataAssets
    Tests -->|"exercise CLI/API"| CLI
    Tests --> CLI
    Tests --> Fixes

    Docs -->|"includes examples from"| CLI
    Docs --> Fixes

    CIWorkflow -->|"triggers tests"| Tests
    CIWorkflow -->|"publishes package"| PyPI
    CIWorkflow -->|"deploys docs"| ReadTheDocs

    GitHub --> CIWorkflow
    GitHub -->|"source repo"| CLI
    GitHub --> Fixes

    %% Click Events
    click CLI "https://github.com/rspeer/python-ftfy/blob/main/ftfy/cli.py"
    click Fixes "https://github.com/rspeer/python-ftfy/blob/main/ftfy/fixes.py"
    click Badness "https://github.com/rspeer/python-ftfy/blob/main/ftfy/badness.py"
    click Formatting "https://github.com/rspeer/python-ftfy/blob/main/ftfy/formatting.py"
    click CharData "https://github.com/rspeer/python-ftfy/blob/main/ftfy/chardata.py"
    click BadCodecs "https://github.com/rspeer/python-ftfy/tree/main/ftfy/bad_codecs/"
    click DataAssets "https://github.com/rspeer/python-ftfy/tree/main/tests/test-cases/"
    click Tests "https://github.com/rspeer/python-ftfy/blob/main/tests/test_cli.py"
    click Docs "https://github.com/rspeer/python-ftfy/tree/main/docs/"
    click Script "https://github.com/rspeer/python-ftfy/blob/main/scripts/char_data_table.py"
    click BuildConfig "https://github.com/rspeer/python-ftfy/blob/main/pyproject.toml"
    click CIWorkflow "https://github.com/rspeer/python-ftfy/blob/main/.github/workflows/publish.yml"
    click RTDConfig "https://github.com/rspeer/python-ftfy/blob/main/.readthedocs.yaml"

    %% Styles
    classDef cli fill:#f9e79f,stroke:#d4ac0d,color:#000;
    classDef core fill:#aed6f1,stroke:#2e86c1,color:#000;
    classDef plugin fill:#d7bde2,stroke:#8e44ad,color:#000;
    classDef data fill:#a3e4d7,stroke:#16a085,color:#000;
    classDef script fill:#fadbd8,stroke:#c0392b,color:#000;
    classDef test fill:#ededed,stroke:#7f8c8d,color:#000;
    classDef docs fill:#fcf3cf,stroke:#b7950b,color:#000;
    classDef ops fill:#d5dbdb,stroke:#566573,color:#000;
    classDef external fill:#f5cba7,stroke:#b9770e,color:#000;
```
