How amazon-ion/ion-java works
1. What this repository is
- ion-java is the reference Java implementation of Amazon Ion.
- It is a library-first repo with three main deliverables: the core Ion API, a command-line tool (ion-java-cli), and performance/test infrastructure.
2. Public API boundary
- The primary stable surface is src/main/java/com/amazon/ion.
- This package defines the Ion type system, values, readers/writers, exceptions, spans, symbol tables, and factory interfaces.
- Most consumers interact only with these types and the builders under com.amazon.ion.system.
3. Builder and configuration layer
- com.amazon.ion.system is the entry point for creating readers, writers, systems, and catalogs.
- IonSystemBuilder, IonReaderBuilder, IonWriterBuilder, IonTextWriterBuilder, and IonBinaryWriterBuilder encapsulate configuration and hide implementation details.
- This is the main construction boundary between application code and the internal engine.
4. Core data model
- Ion values are represented as typed objects such as IonList, IonStruct, IonString, IonSymbol, IonTimestamp, and IonDatagram.
- The model supports both mutable container-style values and the streaming-oriented reader/writer APIs.
- IonDatagram is the top-level container that bridges between parsed data and application-level manipulation.
5. Two execution styles: streaming vs DOM-like
- The library supports streaming parsing/writing through IonReader and IonWriter.
- It also supports an in-memory tree model through IonValue and IonSystem.
- Many flows start with a reader, optionally materialize into values, and then write back out through a writer.
6. Internal implementation boundary
- Nearly all real parsing, encoding, buffering, and symbol handling lives under src/main/java/com/amazon/ion/impl.
- These classes are intentionally private-ish and are not the main API contract.
- The implementation layer is split by concern: text parsing, binary parsing, tree readers, writers, symbol tables, buffering, and utility primitives.
7. Text and binary processing pipelines
- Text Ion and binary Ion are handled by different internal reader/writer stacks.
- Text-oriented classes include raw token readers, text readers, and text writers.
- Binary-oriented classes include binary readers, binary writers, and low-level byte/block encoders.
- The architecture is format-aware all the way down; text and binary share abstractions but not the same parsing/encoding machinery.
8. Binary writer architecture and Ion 1.1 support
- The impl/bin package contains the newer binary writer stack.
- It includes block allocation, write buffers, UTF-8 pooling, and encoder classes such as IonEncoder_1_1.
- This area is optimized for throughput and memory reuse, and it is separate from the older general-purpose writer code paths.
9. Lite DOM implementation
- impl/lite contains the lightweight in-memory Ion implementation.
- It provides concrete IonValueLite, container/value subclasses, IonSystemLite, IonLoaderLite, and ValueFactoryLite.
- This is the main DOM implementation used when the library needs to materialize data rather than stream it.
10. Symbol tables and cataloging
- Symbol resolution is a first-class subsystem, not an incidental detail.
- Public interfaces include SymbolTable, SymbolToken, IonCatalog, and IonMutableCatalog.
- Internally there are shared/local/substitute symbol table implementations and struct-backed forms for serialization.
- This subsystem sits on the critical path for both binary and text Ion because field names and symbols must be resolved consistently.
11. Spans, cursors, and incremental/seekable reading
- The repo includes infrastructure for source-location tracking and partial inspection: Span, TextSpan, OffsetSpan, SeekableReader, RawValueSpanProvider, and related helpers.
- These capabilities matter for tooling, diagnostics, and streaming readers that need byte/text offsets.
- There are separate pathways for seekable, non-seekable, and incremental input.
12. Utilities and cross-cutting helpers
- com.amazon.ion.util contains reusable helpers for equivalence, printing, stream handling, gzip detection, and span utilities.
- The library also includes app-facing helper classes in com.amazon.ion.apps such as PrintApp, EncodeApp, and SymtabApp.
- These are convenience layers built on top of the core reader/writer/system APIs.
13. Standalone CLI module
- ion-java-cli is a separate Gradle subproject with its own source tree and command model.
- It implements commands for comparing Ion data, reporting errors, and processing events.
- This module is an end-user tool built on the library, not part of the core runtime.
14. Testing, benchmarks, and external test data
- The repo is heavily tested, with unit, streaming, system-processing, and implementation-level tests across src/test.
- It depends on the ion-tests git submodule for shared test fixtures and canonical data.
- src/jmh contains microbenchmarks for hot paths such as text processing and hashing.
- GitHub Actions workflows cover CI, code scanning, release preparation, publishing, performance regression detection, and test-driver integration.
15. High-level data flow to diagram
- Typical input flow: bytes/streams/text → reader implementation → symbol resolution / span tracking → optional IonValue materialization.
- Typical output flow: IonValue or streaming events → writer builder/configuration → text or binary writer/encoder → output stream/bytes.
- The symbol table subsystem is shared across both directions and is central to preserving Ion semantics.
Main entry points: ion Ion types · impl Text pipeline · java Tests & benches · bin Bin writer 1.1
Components 19
Public API
Stable Ion types and top-level reader/writer contracts
- Ion types · IonValue, IonType, readers, writers, exceptions, spans, and symbol contracts
src/main/java/com/amazon/ion - System API · IonSystem, ValueFactory, loader, datagram, and catalog entry points
src/main/java/com/amazon/ion - Symbols · SymbolTable, SymbolToken, catalog, and system symbol definitions
src/main/java/com/amazon/ion - Spans · Span, TextSpan, OffsetSpan, SeekableReader, and raw span access
src/main/java/com/amazon/ion
Builders
Entry points for constructing systems, readers, and writers
- System builder · IonSystemBuilder and system-level configuration
src/main/java/com/amazon/ion/system - Reader builder · IonReaderBuilder and reader configuration
src/main/java/com/amazon/ion/system - Writer builder · IonWriterBuilder base for writer construction and tuning
src/main/java/com/amazon/ion/system - Text writer · IonTextWriterBuilder for text output configuration
src/main/java/com/amazon/ion/system - Binary writer · IonBinaryWriterBuilder for binary output configuration
src/main/java/com/amazon/ion/system
Internal Engine
Private implementation for parsing, writing, symbol handling, and buffering
- Text pipeline · Raw token reader, text reader, and text writer stack
src/main/java/com/amazon/ion/impl - Binary pipeline · Binary reader stack and legacy writer paths
src/main/java/com/amazon/ion/impl - Symbol tables · Local, shared, substitute, and struct-backed symbol table machinery
src/main/java/com/amazon/ion/impl - Buffers · Input, output, cursor, and byte/block buffering primitives
src/main/java/com/amazon/ion/impl - Bin writer 1.1 · Managed binary writer, encoder, block allocator, and UTF-8 pools
src/main/java/com/amazon/ion/impl/bin
Lite DOM
In-memory Ion value implementation
- Lite DOM · IonValueLite, container/value subclasses, loader, system, and factory
src/main/java/com/amazon/ion/impl/lite
Tooling
CLI and app-facing helpers
Quality
Tests, benchmarks, and CI-adjacent infrastructure
- Tests & benches · Unit, streaming, impl, JMH benchmarks, and CI-driven validation
src/test/java
Connections 27
- System builder → System API: configures · System builder produces the public IonSystem boundary
- Reader builder → Ion types: creates · Reader builder creates public reader instances
- Writer builder → Ion types: creates · Writer builder creates public writer instances
- Text writer → Text pipeline: targets · Text writer builder selects the text pipeline
- Binary writer → Bin writer 1.1: targets · Binary writer builder selects the optimized binary writer stack
- System API → Lite DOM: loads · IonSystem materializes values through the lite DOM implementation
- Ion types → Text pipeline: streams · Reader/writer contracts are implemented by the text stack
- Ion types → Binary pipeline: streams · Reader/writer contracts are implemented by the binary stack
- Ion types → Symbol tables: resolves · Public symbol contracts depend on internal symbol table machinery
- Spans → Buffers: tracks · Span and seek APIs are backed by buffer and cursor infrastructure
- Text pipeline → Buffers: uses · Text parsing and writing consume shared buffering primitives
- Binary pipeline → Buffers: uses · Binary parsing and writing consume shared buffering primitives
- Binary pipeline → Symbol tables: resolves · Binary readers and writers rely on symbol table resolution
- Text pipeline → Symbol tables: resolves · Text readers and writers rely on symbol table resolution
- Bin writer 1.1 → Symbol tables: encodes · Binary 1.1 writer emits symbol-aware output
- Bin writer 1.1 → Buffers: allocates · Optimized writer stack manages blocks, pools, and write buffers
- Bin writer 1.1 → Binary pipeline: extends · Managed binary writer sits alongside the broader binary pipeline
- Lite DOM → Text pipeline: materializes · DOM loading can originate from text readers
- Lite DOM → Binary pipeline: materializes · DOM loading can originate from binary readers
- Apps → System API: uses · Convenience apps are built on the public system API
- CLI tool → Ion types: uses · CLI commands consume the public Ion reader/writer surface
- Utilities → Ion types: supports · Utility helpers support printing, comparison, streams, and spans
- Tests & benches → Ion types: verifies · Tests exercise the public data model and streaming APIs
- Tests & benches → Text pipeline: verifies · Tests cover text processing internals
- Tests & benches → Binary pipeline: verifies · Tests cover binary processing internals
- Tests & benches → Lite DOM: verifies · Tests cover the lite DOM implementation
- Tests & benches → CLI tool: verifies · Tests and fixtures cover the standalone CLI