# toon-format/toon-python architecture

> Architecture diagram of the GitHub repository toon-format/toon-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/toon-format/toon-python
- Repository: https://github.com/toon-format/toon-python

## Overview

1. **What this repo is**
   - This is a Python package, `toon_format`, that implements the TOON serialization format in both directions: `encode()` turns Python values into TOON text, and `decode()` parses TOON text back into Python values.
   - It also exposes a command-line tool and an optional Pydantic integration for structured LLM workflows.

2. **Top-level architecture**
   - The repository is organized as a small core library plus two adjacent surfaces:
     - the **core serializer/parser**
     - the **CLI entrypoint**
     - the **optional Pydantic adapter**
   - The package is typed (`py.typed`) and built from `src/toon_format`, with tests and fixtures validating spec behavior.

3. **Encode pipeline**
   - The encoding path starts from arbitrary Python objects and converts them into TOON text.
   - The likely flow is: normalize values and types, choose a representation style based on the value shape, then emit formatted text through writer/encoder components.
   - Modules that suggest this split are `normalize.py`, `encoder.py`, `encoders.py`, and `writer.py`.

4. **Decode pipeline**
   - The decode path starts from a TOON string and reconstructs Python data structures.
   - The parsing stack appears layered: a scanner tokenizes the input, parsing utilities assemble structure, and validation enforces format rules such as indentation, array lengths, and delimiter correctness.
   - Relevant modules here are `_scanner.py`, `_parsing_utils.py`, `_validation.py`, and `decoder.py`.

5. **Format-specific subproblems**
   - Several modules isolate the tricky text-level concerns of TOON:
     - `_string_utils.py` for quoting/escaping/string classification
     - `_literal_utils.py` for primitive/literal handling
     - `primitives.py` for primitive type logic
     - `constants.py` and `types.py` for shared format definitions and option types
   - This separation suggests the core logic is organized around syntax rules rather than around transport or I/O.

6. **Normalization boundary**
   - `normalize.py` is a distinct preprocessing layer that converts non-canonical Python values into TOON-friendly representations.
   - The README indicates this covers things like `Infinity/NaN/Functions -> null`, `Decimal -> float`, `datetime -> ISO 8601`, and `-0 -> 0`.
   - That means the serializer does not simply dump raw Python types; it first enforces the format’s value semantics.

7. **CLI and runtime entrypoint**
   - `src/toon_format/__main__.py` and `cli.py` provide the executable tool.
   - The CLI is a thin boundary over the same encode/decode APIs, handling file/stdin/stdout routing and option parsing such as `--encode`, `--decode`, `--delimiter`, `--indent`, `--length-marker`, and `--no-strict`.
   - This is a classic “library first, CLI on top” structure.

8. **Optional Pydantic integration**
   - `src/toon_format/pydantic/` is an extension boundary, not part of the core runtime.
   - It appears to support two main flows:
     - converting Pydantic schemas into compact TOON schema strings for prompts
     - validating TOON-formatted model output back into Pydantic models
   - Keeping this in a separate package path suggests the core library has no hard dependency on Pydantic.

9. **Token counting and comparison utilities**
   - The README describes token counting and format comparison features that depend on `tiktoken`.
   - These are auxiliary analysis features, not part of the encode/decode core, and are meant for measuring TOON’s efficiency against JSON.
   - Architecturally, they sit beside serialization rather than inside it.

10. **Testing strategy**
   - The test suite is broad and fixture-driven, with separate decode/encode fixtures and spec fixtures.
   - There are targeted tests for scanning, parsing, string utilities, normalization, numeric detection, security, internationalization, CLI, API behavior, and Pydantic integration.
   - This indicates the repository treats format conformance and edge cases as first-class architectural concerns.

11. **Spec compliance and validation**
   - The presence of `tests/fixtures.schema.json` and `tests/test_spec_fixtures.py` suggests the implementation is checked against an external TOON specification.
   - Validation is therefore an important boundary: the decoder is not just a permissive parser, but a format-enforcing component that can operate in strict or relaxed modes.

12. **Tooling and delivery**
   - `pyproject.toml`, GitHub Actions workflows, and the devcontainer files show the project is built for repeatable local development and CI.
   - CI covers linting, testing, and publishing, while the README and `PUBLISHING.md` indicate PyPI distribution is part of the delivery path.

13. **How to draw the main data flows**
   - For encoding: application data -> normalization/type handling -> encoder/writer -> TOON text.
   - For decoding: TOON text -> scanner/parser -> validation -> Python objects.
   - For Pydantic: model/schema -> TOON schema or validated decode/encode path.
   - The main architectural boundary is between **Python object semantics** and **TOON syntax/format rules**.

14. **What matters most for an architecture diagram**
   - Show the core serializer/parser as the central subsystem.
   - Show the CLI as a thin wrapper around the same APIs.
   - Show the Pydantic package as an optional extension branch.
   - Show validation and normalization as separate internal stages, because they materially shape both correctness and interoperability.

Main entry points: [`src/toon_format/encoder.py`](https://github.com/toon-format/toon-python/blob/main/src/toon_format/encoder.py) (Encoder), [`src/toon_format/decoder.py`](https://github.com/toon-format/toon-python/blob/main/src/toon_format/decoder.py) (Decoder), [`tests`](https://github.com/toon-format/toon-python/tree/main/tests) (Tests), [`src/toon_format/cli.py`](https://github.com/toon-format/toon-python/blob/main/src/toon_format/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core library"]
  node_normalize["Normalize<br/>preprocess<br/>[normalize.py]"]
  node_encoder["Encoder<br/>serialize<br/>[encoder.py]"]
  node_encoders["Value encoders<br/>serialize helpers<br/>[encoders.py]"]
  node_writer["Writer<br/>emitter<br/>[writer.py]"]
  node_decoder["Decoder<br/>parse<br/>[decoder.py]"]
  node_scanner["Scanner<br/>lexing<br/>[_scanner.py]"]
  node_parsing["Parse utils<br/>parsing<br/>[_parsing_utils.py]"]
  node_validation["Validation<br/>format rules<br/>[_validation.py]"]
  node_string_utils["String utils<br/>syntax helpers<br/>[_string_utils.py]"]
  node_literal_utils["Literal utils<br/>value syntax<br/>[_literal_utils.py]"]
  node_primitives["Primitives<br/>type rules<br/>[primitives.py]"]
  node_format_defs["Format defs<br/>schema defs"]
end

subgraph group_interfaces["Interfaces"]
  node_cli["CLI<br/>command line<br/>[cli.py]"]
  node_main["Entry point<br/>runtime hook<br/>[__main__.py]"]
end

subgraph group_extensions["Extensions"]
  node_pydantic_ext["Pydantic<br/>integration<br/>[serializer.py]"]
end

subgraph group_quality["Quality"]
  node_tests["Tests<br/>verification"]
end

node_normalize -->|"prepares values"| node_encoder
node_encoder -->|"dispatches"| node_encoders
node_encoders -->|"emits text"| node_writer
node_encoder -->|"uses rules"| node_format_defs
node_decoder -->|"tokenizes"| node_scanner
node_scanner -->|"feeds"| node_parsing
node_parsing -->|"checks"| node_validation
node_validation -->|"governs"| node_decoder
node_string_utils -->|"supports"| node_encoder
node_literal_utils -->|"parses literals"| node_decoder
node_primitives -->|"classifies"| node_normalize
node_format_defs -->|"defines syntax"| node_scanner
node_cli -->|"wraps"| node_encoder
node_cli -->|"wraps"| node_decoder
node_main -->|"starts"| node_cli
node_pydantic_ext -->|"schema encode"| node_encoder
node_pydantic_ext -->|"model validate"| node_decoder
node_tests -.->|"covers"| node_encoder
node_tests -.->|"covers"| node_decoder
node_tests -.->|"covers"| node_pydantic_ext

click node_normalize "https://github.com/toon-format/toon-python/blob/main/src/toon_format/normalize.py"
click node_encoder "https://github.com/toon-format/toon-python/blob/main/src/toon_format/encoder.py"
click node_encoders "https://github.com/toon-format/toon-python/blob/main/src/toon_format/encoders.py"
click node_writer "https://github.com/toon-format/toon-python/blob/main/src/toon_format/writer.py"
click node_decoder "https://github.com/toon-format/toon-python/blob/main/src/toon_format/decoder.py"
click node_scanner "https://github.com/toon-format/toon-python/blob/main/src/toon_format/_scanner.py"
click node_parsing "https://github.com/toon-format/toon-python/blob/main/src/toon_format/_parsing_utils.py"
click node_validation "https://github.com/toon-format/toon-python/blob/main/src/toon_format/_validation.py"
click node_string_utils "https://github.com/toon-format/toon-python/blob/main/src/toon_format/_string_utils.py"
click node_literal_utils "https://github.com/toon-format/toon-python/blob/main/src/toon_format/_literal_utils.py"
click node_primitives "https://github.com/toon-format/toon-python/blob/main/src/toon_format/primitives.py"
click node_cli "https://github.com/toon-format/toon-python/blob/main/src/toon_format/cli.py"
click node_main "https://github.com/toon-format/toon-python/blob/main/src/toon_format/__main__.py"
click node_pydantic_ext "https://github.com/toon-format/toon-python/blob/main/src/toon_format/pydantic/serializer.py"
click node_tests "https://github.com/toon-format/toon-python/tree/main/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_normalize,node_encoder,node_encoders,node_writer,node_decoder,node_scanner,node_parsing,node_validation,node_string_utils,node_literal_utils,node_primitives,node_format_defs toneBlue
class node_cli,node_main toneAmber
class node_pydantic_ext toneMint
class node_tests toneRose
```

## Components

### Core library

Encoding, decoding, and shared format logic for TOON.

- [Normalize](https://github.com/toon-format/toon-python/blob/main/src/toon_format/normalize.py): preprocess · Canonicalizes Python values into TOON-friendly forms. · `src/toon_format/normalize.py`
- [Encoder](https://github.com/toon-format/toon-python/blob/main/src/toon_format/encoder.py): serialize · Top-level encode API and coordination layer. · `src/toon_format/encoder.py`
- [Value encoders](https://github.com/toon-format/toon-python/blob/main/src/toon_format/encoders.py): serialize helpers · Shape- and type-specific encoding strategies. · `src/toon_format/encoders.py`
- [Writer](https://github.com/toon-format/toon-python/blob/main/src/toon_format/writer.py): emitter · Formats TOON text output from structured encoding decisions. · `src/toon_format/writer.py`
- [Decoder](https://github.com/toon-format/toon-python/blob/main/src/toon_format/decoder.py): parse · Top-level decode API and reconstruction layer. · `src/toon_format/decoder.py`
- [Scanner](https://github.com/toon-format/toon-python/blob/main/src/toon_format/_scanner.py): lexing · Tokenizes TOON text for the parser. · `src/toon_format/_scanner.py`
- [Parse utils](https://github.com/toon-format/toon-python/blob/main/src/toon_format/_parsing_utils.py): parsing · Builds structured records from scanned tokens. · `src/toon_format/_parsing_utils.py`
- [Validation](https://github.com/toon-format/toon-python/blob/main/src/toon_format/_validation.py): format rules · Checks indentation, delimiters, lengths, and strictness rules. · `src/toon_format/_validation.py`
- [String utils](https://github.com/toon-format/toon-python/blob/main/src/toon_format/_string_utils.py): syntax helpers · Quoting, escaping, and string classification. · `src/toon_format/_string_utils.py`
- [Literal utils](https://github.com/toon-format/toon-python/blob/main/src/toon_format/_literal_utils.py): value syntax · Handles primitive and literal text forms. · `src/toon_format/_literal_utils.py`
- [Primitives](https://github.com/toon-format/toon-python/blob/main/src/toon_format/primitives.py): type rules · Primitive type handling used by encode and decode. · `src/toon_format/primitives.py`
- **Format defs**: schema defs · Shared constants, option types, and format-level definitions.

### Interfaces

Thin entrypoints that expose the core library to users and tools.

- [CLI](https://github.com/toon-format/toon-python/blob/main/src/toon_format/cli.py): command line · User-facing command-line wrapper over encode and decode. · `src/toon_format/cli.py`
- [Entry point](https://github.com/toon-format/toon-python/blob/main/src/toon_format/__main__.py): runtime hook · Python module entrypoint for the CLI. · `src/toon_format/__main__.py`

### Extensions

Optional integration surface built on top of the core package.

- [Pydantic](https://github.com/toon-format/toon-python/blob/main/src/toon_format/pydantic/serializer.py): integration · Optional adapter for schema generation and model validation. · `src/toon_format/pydantic/serializer.py`

### Quality

Tests and fixtures that lock in format behavior.

- [Tests](https://github.com/toon-format/toon-python/tree/main/tests): verification · Fixture-driven checks for spec compliance, API behavior, and edge cases. · `tests`

## Connections

- Normalize → Encoder: prepares values · Encode pipeline normalizes Python objects before serialization.
- Encoder → Value encoders: dispatches · Top-level encoding delegates by value shape and type.
- Value encoders → Writer: emits text · Selected encoding strategy writes TOON output.
- Encoder → Format defs: uses rules · Encoding relies on shared format constants and option definitions.
- Decoder → Scanner: tokenizes · Decode begins by scanning raw TOON text.
- Scanner → Parse utils: feeds · Tokens are assembled into higher-level parse structures.
- Parse utils → Validation: checks · Parsed structures are validated against format constraints.
- Validation → Decoder: governs · Validation shapes permissive versus strict decode behavior.
- String utils → Encoder: supports · String quoting and escaping are required during encoding.
- Literal utils → Decoder: parses literals · Literal handling supports reconstruction of primitive values.
- Primitives → Normalize: classifies · Primitive rules inform normalization decisions.
- Format defs → Scanner: defines syntax · Shared format definitions inform lexical and structural rules.
- CLI → Encoder: wraps · CLI encoding mode routes to the library API.
- CLI → Decoder: wraps · CLI decoding mode routes to the library API.
- Entry point → CLI: starts · Module entrypoint invokes the CLI surface.
- Pydantic → Encoder: schema encode · Optional integration can serialize model data through the core encoder.
- Pydantic → Decoder: model validate · Optional integration can validate TOON output back into models.
- Tests → Encoder: covers · Fixtures and tests verify encode behavior and normalization.
- Tests → Decoder: covers · Fixtures and tests verify decode behavior and validation.
- Tests → Pydantic: covers · Dedicated tests exercise optional Pydantic integration.
