amazon-science/sted

Purpose and Entry Points

sted is a Python library and research framework for measuring reliability of LLM JSON/tool-call outputs. Its core metric is Semantic Tree Edit Distance (STED); higher-level evaluators aggregate pairwise similarities across repeated runs. Primary public entry points are package exports in sted/__init__.py, the CLI in sted/cli.py, and examples in examples/basic_usage.py.

Core STED Engine

SemanticJsonTreeConsistencyEvaluator converts nested JSON into semantic tree comparisons, combining structural constraints with embedding-based key/value similarity. Tree representation and matching/cost logic are separated across:

• sted/semantic_json_tree_consistency.py

• sted/json_tree_node.py

• sted/_matching.py

• sted/_costs.py

Array/order-invariant matching uses Hungarian-style optimal assignment, creating the principal computational boundary and O(n³) worst case for large collections.

Embedding and Similarity Boundary

Semantic comparison is delegated to pluggable embedding providers and caches. The default local path uses sentence-transformer embeddings; AWS Bedrock is an alternate remote backend. Reuse is important because pairwise scoring amplifies embedding cost.

• sted/embeddings/sentence_transformers.py

• sted/embeddings/bedrock.py

• sted/similarity_cache.py

This layer isolates model/API calls from tree-edit logic and supports precomputation/batched reuse.

Consistency Aggregation and Production API

Structural consistency converts a collection of outputs for one prompt into pairwise STED measurements and aggregate coefficients. AgentConsistencyEvaluator adds an operational interface: execute a user-supplied JSON-returning agent repeatedly or score pre-collected outputs, while separating parse validity from consistency. Async execution is a parallel adapter.

• sted/structural_consistency_analyzer.py

• sted/agent_consistency_evaluator.py

• sted/agent_consistency_async.py

Control flow is: prompts or logs → parseable-output filtering → embedding precompute/cache → pairwise STED → per-prompt and overall report.

Model Invocation and External Services

Benchmark generation routes model IDs through a centralized provider registry. Bedrock models use AWS credentials and Converse APIs; OpenAI-compatible models use API-key/base-URL configuration. Bedrock also supplies optional embeddings.

• sted/model_config.py

• sted/bedrock_utils.py

• scripts/eval/run_temperature_experiment.py

The core library does not require LLM invocation when evaluating existing logs; remote services are primarily generation/embedding dependencies.

Research Evaluation Pipeline

The repository’s experiment workflow is batch-oriented: acquire ShareGPT/Toucan data, generate repeated structured or tool-call outputs at temperatures, calculate consistency, then analyze/visualize results.

• scripts/data/download/download_sharegpt_data.py

• scripts/eval/generate_structured_outputs.py

• scripts/eval/generate_tool_calls.py

• scripts/eval/calculate_consistency_metrics.py

Synthetic-data generation and dataset-analysis scripts validate whether STED distinguishes schema, expression, and semantic changes rather than merely supporting operational scoring.

Extensions and Deployment

Risk prediction and LLM-judge modules complement direct STED scoring; telemetry integration is explicitly adapter-based. The package is containerizable and has EC2-oriented experiment runners, but is principally a Python library plus offline experimentation system, not a web service.

• sted/consistency_predictor.py

• sted/consistency_risk_assessor.py

• sted/otel_adapter.py

• Dockerfile

• docker-compose.yml