How rspeer/python-ftfy works
Project type and purpose
• This is an open-source Python library/CLI tool (ftfy: “fixes text for you”) for detecting and correcting mojibake and other text‐encoding problems.
• The README describes a programmatic API (fix_text, fix_encoding), a CLI entry point (ftfy/cli.py), and extensive documentation and test coverage.
File structure
• Top-level directories:
• ftfy/ → core library modules
• docs/ → Sphinx documentation sources
• tests/ → pytest test suite and test-case data
• scripts/ → auxiliary code (e.g. data-table generator)
• notebook/ → presentation notebook
• Configuration/build/deployment files:
• pyproject.toml, MANIFEST.in, setup in .github/workflows/publish.yml
• pytest.ini, mypy.ini, tox.ini, .readthedocs.yaml
• Data modules under ftfy/bad_codecs/ and ftfy/chardata.py hint at a data-driven design.
Key insights from the README
• Core functionality: detect and fix encoding mix-ups, HTML entities, curly quotes, etc.
• Describes API usage, CLI usage, configuration, and citations.
• Mentions documentation on ReadTheDocs and package publishing to PyPI.
Main components and their interactions
a. Components:
1. CLI layer (ftfy/cli.py)
2. Core API modules:
• fix logic (ftfy/fixes.py, ftfy/badness.py)
• formatting (ftfy/formatting.py)
• character data (ftfy/chardata.py)
• bad-codec handling (ftfy/bad_codecs/*.py)
3. Data assets: JSON/text files in tests/test-cases
4. Test suite (pytest)
5. Documentation (Sphinx: docs/, docs/conf.py)
6. Packaging & CI (GitHub Actions, pyproject.toml)
7. Auxiliary scripts (scripts/char_data_table.py)
8. External services:
• PyPI (package distribution)
• ReadTheDocs (docs hosting)
• GitHub (source, CI)
b. Relationships and data flows:
• The CLI invokes the core API, passing user input to fix_text or fix_encoding.
• Core API modules collaborate:
• Input passes through detection/heuristics (badness.py).
• Character mapping and codec heuristics in chardata.py and bad_codecs/.
• Corrections applied in fixes.py and formatting.py.
• Scripts generate data artifacts used in docs or tests.
• Tests exercise both the CLI and API layers against test-case data.
• Documentation is generated from docs/, referencing code comments and examples.
• CI (GitHub Actions) builds, tests, and publishes to PyPI on tag.
c. Architectural patterns and principles:
• Modular plugin pattern for “bad codecs” – new codecs = new module.
• Data-driven design: large characterization tables live in code/data modules.
• Clear separation: API core vs. CLI vs. data vs. docs vs. tests.
• Use of standard Python packaging (PEP 517/518, pyproject.toml).
d. Key technologies and frameworks:
• Python 3 (PEP 484 typing, py.typed)
• pytest for testing
• Sphinx for documentation
• uv (task runner)
• GitHub Actions for CI/CD
• Publishing to PyPI, hosting docs on ReadTheDocs
Detailed capture of essential elements
• Do not over-simplify: show each module grouping (fixes, formatting, bad_codecs, chardata).
• Test data flow: test suite consumes library + test-case JSON.
• Documentation pipeline: docs sources + scripts → HTML on ReadTheDocs.