If you're building a RAG pipeline or an AI agent, you've probably noticed that structured data — retrieved documents, database rows, tool-call results — takes up a large share of every prompt. Every token in that context costs money, and every model has a hard limit on how much context it can hold at once. When you're stuffing search results into a prompt or feeding an agent's tool output back into the model turn after turn, the format you use to encode that data directly affects both your bill and how much room is left for everything else. TOON (Token-Oriented Object Notation) exists specifically for this boundary. This guide walks through where it actually helps in RAG and agent systems, with a worked example checked against the site's real conversion logic, and where it isn't worth the extra step.
💸 The Token-Cost Problem in RAG and Agent Systems
A typical RAG pipeline retrieves a handful of relevant documents or rows from a vector database or search index, then injects them into the prompt so the model can ground its answer in them. An AI agent does something similar every turn: a tool call returns a JSON payload — search results, API responses, matching database records — and that payload gets appended to the conversation history so the model can act on it. In both cases, the same structural pattern repeats: a JSON array of objects that all share the same fields. JSON encodes that by repeating every field name for every object in the array. Retrieve 20 documents with 5 fields each, and JSON sends you the same 5 key names 20 times before the model reads a single value. That repetition is pure overhead — it doesn't help the model understand the data, and you pay for it on every request.
In an agent loop, this compounds: if tool results are appended to context on every turn without being trimmed, a bulkier encoding means the context window fills up — and truncates useful history — sooner than it needs to.
🎯 Where TOON Helps Most
TOON's token savings come from one mechanism: for arrays of uniform objects — every item having the same set of fields — it states the field names once, in a header, and then writes each object as a plain comma-separated row underneath. That maps directly onto three common situations:
🔍 RAG: injecting retrieved documents or rows
A retrieval step that returns N search results, product records, or database rows to stuff into the prompt is exactly the uniform-array case TOON is built for. Every retrieved item usually has the same shape (id, title, score, source, and so on), so JSON repeats those field names N times while TOON states them once.
🛠️ Agents: tool-call results fed back into context
An agent whose tool returns a list of matching records from a database or API — and that list gets appended to the conversation on every subsequent turn — pays the JSON repetition cost repeatedly. Encoding it as TOON before it re-enters context means more turns of conversation can fit before the context window fills up and older turns get truncated.
💬 Chatbots: structured session data in the system prompt
A chatbot that injects user profile fields, session state, or a list of recent interactions into the system prompt on every message benefits the same way — that payload is sent with every single message, so trimming its encoding once pays off on every request afterward.
🧪 A Worked Example: Five Search Results
Here's a realistic RAG scenario: a retrieval step returns 5 search results, each with an id, title, relevance score, and source. This is the JSON you'd get back from a typical vector search or database query:
❌ JSON (pretty-printed, as you'd typically build it)
{
"results": [
{ "id": 101, "title": "Intro to Vector Databases", "score": 0.93, "source": "docs.example.com" },
{ "id": 102, "title": "RAG Pipeline Best Practices", "score": 0.89, "source": "blog.example.com" },
{ "id": 103, "title": "Chunking Strategies for Embeddings", "score": 0.87, "source": "docs.example.com" },
{ "id": 104, "title": "Reranking with Cross-Encoders", "score": 0.84, "source": "papers.example.com" },
{ "id": 105, "title": "Evaluating Retrieval Quality", "score": 0.81, "source": "blog.example.com" }
]
}
Running this through the same jsonToToon() logic that powers this site's
converter
produces:
✅ TOON
results[5]{id,title,score,source}:
101,Intro to Vector Databases,0.93,docs.example.com
102,RAG Pipeline Best Practices,0.89,blog.example.com
103,Chunking Strategies for Embeddings,0.87,docs.example.com
104,Reranking with Cross-Encoders,0.84,papers.example.com
105,Evaluating Retrieval Quality,0.81,blog.example.com
Because results is an array where every object has the exact same keys in the same
order and no nested objects, TOON's encoder classifies it as a uniform object array: it writes
the results[5]{id,title,score,source}: header once, then one comma-separated row per
item, indented two spaces under it. The field names id, title,
score, and source appear exactly once instead of five times.
Measuring this specific example by character count: the pretty-printed JSON above is 691 characters, the same data minified to a single line is 473 characters, and the TOON version is 324 characters — about 53% fewer characters than pretty JSON and about 32% fewer than minified JSON. Token counts track character counts closely but aren't identical, since tokenizers group characters differently; treat this as a directional measurement of this one example, not a token count. For a broader, tokenizer-based figure, TOON's own published benchmarks report roughly 42.6% fewer tokens than equivalent JSON on average across datasets, with LLM retrieval accuracy that's comparable to or slightly better than JSON (72.2% vs JSON's 71.4%). Five rows is a small example — the gap widens as the array grows, since the header cost is fixed but the per-row savings compound with every additional item.
🧠 Getting the Model to Understand TOON
TOON isn't a native input or output format for any LLM API — there's no "toon" content type you can set on a request. You convert your JSON to TOON yourself, as a plain text step, before it goes into the prompt. The model then has to read that text and understand its structure well enough to reason about it, the same way it would read a code block.
In practice this means including a short explanation of the format, or a one-line example,
somewhere in your system or user prompt — something like: "Data below is in TOON format:
key[N]{col1,col2}: is a header stating N rows with those columns, followed by one
comma-separated row per line." Most capable models can then parse TOON reliably, especially
since its indentation-based structure resembles YAML, which models have seen extensively in
training. If you want the model's own output back in TOON too — for example, so an agent's
response can be re-parsed programmatically — you need to ask for that explicitly and show it
what the format looks like, since nothing about the API enforces it.
🔁 Round-Tripping Model Output Back to JSON
If a model responds in TOON, your application still needs plain JSON downstream. This site's
JSON to TOON converter
can decode TOON back to JSON interactively for testing and debugging. To do this
programmatically in a server-side pipeline, port the same parsing rules — reading the
[N]{col1,col2}: header, then splitting each indented row on commas back into an
object per row — into whatever language your backend uses.
⚖️ When TOON Isn't Worth It
TOON adds a conversion step: something has to turn your JSON into TOON before it reaches the prompt, and possibly turn TOON back into JSON afterward. That's a fixed cost, in code and in complexity, that has to be paid regardless of how much the format itself saves. A few cases where it isn't worth paying:
Small payloads
A single object or a two- or three-item array doesn't repeat enough keys for the header-once savings to matter much. The fixed overhead of adding a conversion step can outweigh a token saving measured in the dozens.
Highly irregular, non-uniform data
When array items don't share the same set of keys, TOON can't build a single header row for them and falls back to a more expanded, per-item encoding. It still round-trips correctly, but the compact tabular savings that make TOON worth adopting for RAG-style data don't apply.
One-off scripts and prototypes
If you're calling an LLM a handful of times to test an idea, the engineering time to wire up a conversion step is very unlikely to be worth the token savings. Reach for TOON once a data pattern is going into production and repeating on every request.
🎯 The Practical Rule
Adopt TOON where a uniform, repeated array shape shows up on a large scale or on every request — RAG search results, tool-call record lists, session data injected on every message. Skip it for small, one-off, or deeply irregular payloads where the conversion step costs more engineering effort than it saves in tokens.
❓ Frequently Asked Questions
Does TOON work with the OpenAI or Anthropic APIs directly?
No. TOON is not a native input or output format for any LLM API or SDK. You convert your JSON data to TOON yourself before building the prompt, and if you want the model's response in TOON too, you need to tell it what TOON is, usually with a short inline explanation or example.
How much does TOON actually save in a RAG pipeline?
It depends on your data shape. TOON's savings come from stating field names once instead of repeating them per object, so the more uniform records you inject, the bigger the savings. According to TOON's published benchmarks, converting typical structured datasets produces roughly 42.6% fewer tokens than equivalent JSON on average, with comparable or better LLM retrieval accuracy (72.2% vs JSON's 71.4%). Always measure your own payload rather than assuming that figure applies exactly.
Will an LLM understand TOON without any explanation?
Many models can infer TOON's structure from context since it resembles YAML, but for reliable parsing you should include a brief explanation of the syntax or a short example in your prompt, especially if you expect the model to also generate TOON as output.
Should I convert every tool call result to TOON in my agent?
Only where it helps. Tool calls that return a list of uniform records benefit the most because that list gets re-sent on every subsequent turn. A tool call returning a single scalar or a small irregular object gains little and isn't worth the added conversion step.
What happens if my retrieved data isn't uniform?
TOON still encodes it correctly, but falls back to a more expanded, indentation-based form instead of the compact tabular header-plus-rows form, since it can't state a shared set of field names once for objects with different keys. The token savings on non-uniform data are smaller than on uniform arrays.
🚀 Try It Yourself
Paste your own RAG search results or agent tool-call output into the converter below and compare the JSON and TOON versions directly. If you're new to the format, start with the docs for the full syntax reference, or the introductory post for the reasoning behind TOON.