How toon-format/toon-python works
1. What this repo is
- This is a Python package, toon_format, that implements the TOON serialization format in both directions: encode() turns Python values into TOON text, and decode() parses TOON text back into Python values.
- It also exposes a command-line tool and an optional Pydantic integration for structured LLM workflows.
2. Top-level architecture
- The repository is organized as a small core library plus two adjacent surfaces:
- the core serializer/parser
- the CLI entrypoint
- the optional Pydantic adapter
- The package is typed (py.typed) and built from src/toon_format, with tests and fixtures validating spec behavior.
3. Encode pipeline
- The encoding path starts from arbitrary Python objects and converts them into TOON text.
- The likely flow is: normalize values and types, choose a representation style based on the value shape, then emit formatted text through writer/encoder components.
- Modules that suggest this split are normalize.py, encoder.py, encoders.py, and writer.py.
4. Decode pipeline
- The decode path starts from a TOON string and reconstructs Python data structures.
- The parsing stack appears layered: a scanner tokenizes the input, parsing utilities assemble structure, and validation enforces format rules such as indentation, array lengths, and delimiter correctness.
- Relevant modules here are _scanner.py, _parsing_utils.py, _validation.py, and decoder.py.
5. Format-specific subproblems
- Several modules isolate the tricky text-level concerns of TOON:
- _string_utils.py for quoting/escaping/string classification
- _literal_utils.py for primitive/literal handling
- primitives.py for primitive type logic
- constants.py and types.py for shared format definitions and option types
- This separation suggests the core logic is organized around syntax rules rather than around transport or I/O.
6. Normalization boundary
- normalize.py is a distinct preprocessing layer that converts non-canonical Python values into TOON-friendly representations.
- The README indicates this covers things like Infinity/NaN/Functions -> null, Decimal -> float, datetime -> ISO 8601, and -0 -> 0.
- That means the serializer does not simply dump raw Python types; it first enforces the format’s value semantics.
7. CLI and runtime entrypoint
- src/toon_format/__main__.py and cli.py provide the executable tool.
- The CLI is a thin boundary over the same encode/decode APIs, handling file/stdin/stdout routing and option parsing such as --encode, --decode, --delimiter, --indent, --length-marker, and --no-strict.
- This is a classic “library first, CLI on top” structure.
8. Optional Pydantic integration
- src/toon_format/pydantic/ is an extension boundary, not part of the core runtime.
- It appears to support two main flows:
- converting Pydantic schemas into compact TOON schema strings for prompts
- validating TOON-formatted model output back into Pydantic models
- Keeping this in a separate package path suggests the core library has no hard dependency on Pydantic.
9. Token counting and comparison utilities
- The README describes token counting and format comparison features that depend on tiktoken.
- These are auxiliary analysis features, not part of the encode/decode core, and are meant for measuring TOON’s efficiency against JSON.
- Architecturally, they sit beside serialization rather than inside it.
10. Testing strategy
- The test suite is broad and fixture-driven, with separate decode/encode fixtures and spec fixtures.
- There are targeted tests for scanning, parsing, string utilities, normalization, numeric detection, security, internationalization, CLI, API behavior, and Pydantic integration.
- This indicates the repository treats format conformance and edge cases as first-class architectural concerns.
11. Spec compliance and validation
- The presence of tests/fixtures.schema.json and tests/test_spec_fixtures.py suggests the implementation is checked against an external TOON specification.
- Validation is therefore an important boundary: the decoder is not just a permissive parser, but a format-enforcing component that can operate in strict or relaxed modes.
12. Tooling and delivery
- pyproject.toml, GitHub Actions workflows, and the devcontainer files show the project is built for repeatable local development and CI.
- CI covers linting, testing, and publishing, while the README and PUBLISHING.md indicate PyPI distribution is part of the delivery path.
13. How to draw the main data flows
- For encoding: application data -> normalization/type handling -> encoder/writer -> TOON text.
- For decoding: TOON text -> scanner/parser -> validation -> Python objects.
- For Pydantic: model/schema -> TOON schema or validated decode/encode path.
- The main architectural boundary is between Python object semantics and TOON syntax/format rules.
14. What matters most for an architecture diagram
- Show the core serializer/parser as the central subsystem.
- Show the CLI as a thin wrapper around the same APIs.
- Show the Pydantic package as an optional extension branch.
- Show validation and normalization as separate internal stages, because they materially shape both correctness and interoperability.
Main entry points: encoder.py Encoder · decoder.py Decoder · tests Tests · cli.py CLI
Components 16
Core library
Encoding, decoding, and shared format logic for TOON.
- Normalize · Canonicalizes Python values into TOON-friendly forms.
src/toon_format/normalize.py - Encoder · Top-level encode API and coordination layer.
src/toon_format/encoder.py - Value encoders · Shape- and type-specific encoding strategies.
src/toon_format/encoders.py - Writer · Formats TOON text output from structured encoding decisions.
src/toon_format/writer.py - Decoder · Top-level decode API and reconstruction layer.
src/toon_format/decoder.py - Scanner · Tokenizes TOON text for the parser.
src/toon_format/_scanner.py - Parse utils · Builds structured records from scanned tokens.
src/toon_format/_parsing_utils.py - Validation · Checks indentation, delimiters, lengths, and strictness rules.
src/toon_format/_validation.py - String utils · Quoting, escaping, and string classification.
src/toon_format/_string_utils.py - Literal utils · Handles primitive and literal text forms.
src/toon_format/_literal_utils.py - Primitives · Primitive type handling used by encode and decode.
src/toon_format/primitives.py - Format defs · Shared constants, option types, and format-level definitions.
Interfaces
Thin entrypoints that expose the core library to users and tools.
- CLI · User-facing command-line wrapper over encode and decode.
src/toon_format/cli.py - Entry point · Python module entrypoint for the CLI.
src/toon_format/__main__.py
Extensions
Optional integration surface built on top of the core package.
- Pydantic · Optional adapter for schema generation and model validation.
src/toon_format/pydantic/serializer.py
Quality
Tests and fixtures that lock in format behavior.
- Tests · Fixture-driven checks for spec compliance, API behavior, and edge cases.
tests
Connections 20
- Normalize → Encoder: prepares values · Encode pipeline normalizes Python objects before serialization.
- Encoder → Value encoders: dispatches · Top-level encoding delegates by value shape and type.
- Value encoders → Writer: emits text · Selected encoding strategy writes TOON output.
- Encoder → Format defs: uses rules · Encoding relies on shared format constants and option definitions.
- Decoder → Scanner: tokenizes · Decode begins by scanning raw TOON text.
- Scanner → Parse utils: feeds · Tokens are assembled into higher-level parse structures.
- Parse utils → Validation: checks · Parsed structures are validated against format constraints.
- Validation → Decoder: governs · Validation shapes permissive versus strict decode behavior.
- String utils → Encoder: supports · String quoting and escaping are required during encoding.
- Literal utils → Decoder: parses literals · Literal handling supports reconstruction of primitive values.
- Primitives → Normalize: classifies · Primitive rules inform normalization decisions.
- Format defs → Scanner: defines syntax · Shared format definitions inform lexical and structural rules.
- CLI → Encoder: wraps · CLI encoding mode routes to the library API.
- CLI → Decoder: wraps · CLI decoding mode routes to the library API.
- Entry point → CLI: starts · Module entrypoint invokes the CLI surface.
- Pydantic → Encoder: schema encode · Optional integration can serialize model data through the core encoder.
- Pydantic → Decoder: model validate · Optional integration can validate TOON output back into models.
- Tests → Encoder: covers · Fixtures and tests verify encode behavior and normalization.
- Tests → Decoder: covers · Fixtures and tests verify decode behavior and validation.
- Tests → Pydantic: covers · Dedicated tests exercise optional Pydantic integration.