Why Agent Trace Logs Belong in Iceberg 1.12 Variant Columns

Open your agent trace table and count the fields that keep changing shape. A tool call carries args, a retrieval step carries chunks, and a framework upgrade adds reasoning_tokens on a random Tuesday. So the payload sits in a JSON string, and every query that wants one field parses the whole blob. Iceberg variant columns, with the 1.12 release behind them, are a better home for that payload.

The JSON string tax on agent trace logs

A string column is flexible but dumb. To read one field, the engine parses the entire text. JSON's type system is also thin, so timestamps and exact decimals collapse into strings and floats.

The Iceberg Variant overview describes Variant as JSON-like in flexibility but stored as compact, typed binary. The table schema declares only variant, not the fields inside it, so a new field in your traces needs no schema migration.

That fits agent traces because the shape is the part you don't control. You own trace_id and ts. You don't own what a tool returns.

Create Iceberg variant columns for trace payloads

Keep the fields you always filter on as real columns. Put the volatile payload in a Variant. Variant needs a v3 table, so set the format version explicitly.

CREATE TABLE agent_traces (
  trace_id STRING,
  span_id  STRING,
  ts       TIMESTAMP,
  agent    STRING,
  payload  VARIANT
)
USING iceberg
PARTITIONED BY (days(ts))
TBLPROPERTIES (
  'format-version' = '3',
  'write.parquet.shred-variants' = 'true'
);

write.parquet.shred-variants defaults to false. When true, Parquet stores commonly seen Variant fields as their own typed columns and keeps the leftovers in the untyped value. Shredding is Parquet-only, while Variant itself also works in Avro and ORC.

Querying changes by one function. Fields missing from a row come back as null, which suits traces where not every span has every key.

-- before: payload_json is a STRING
SELECT get_json_object(payload_json, '$.tool') AS tool, count(*) AS calls
FROM agent_traces_json
WHERE agent = 'refund-agent'
GROUP BY 1;

-- after: payload is a VARIANT
SELECT variant_get(payload, '$.tool', 'string') AS tool, count(*) AS calls
FROM agent_traces
WHERE agent = 'refund-agent'
GROUP BY 1;

The query reads the same. What differs is how much the engine has to decode to answer it.

What Iceberg 1.12 adds for reads, writes and streaming

The 1.12.0 release notes list three changes that matter for trace pipelines.

• Spark reads (PR #16292): vectorized Parquet reads for variant columns. Before this, a variant column pushed the whole table onto row-at-a-time reads. The PR covers Spark 4.0 and 4.1, so check your version before counting on it.
• Flink writes (PR #15596): Flink can now write shredded Variant. The docs list a per-write shred-variants option that overrides the table property.
• Kafka Connect writes (PR #17520): Parquet variant shredding now works for Kafka Connect and generic Record writers. Before that PR, setting write.parquet.shred-variants there silently did nothing.

The Kafka Connect one is the quiet winner. Plenty of trace pipelines already fan into a lakehouse through Kafka, and now the same table property applies to that path.

The strongest objection: it's new, and the optimizations don't stack yet

Fair. Variant needs a v3 table, so every engine reading your traces must support v3. If one of them can't, a JSON string still wins on compatibility.

The Spark read path also has edges. According to the PR description, vectorization falls back to row reads for shredded files, for variants nested inside structs, lists or maps, and for columns whose metrics mode is none or counts. So in Spark you are choosing between the vectorized path and shredding, and you should measure both on your own queries. I found no benchmark numbers in the PR, and I won't invent any.

Shredding has a quirk too. Each data file infers its layout from its own buffered rows, 100 by default, so files in one snapshot can shred different fields. If your traces vary wildly, raise write.parquet.variant-inference-buffer-size and accept the extra memory.

And if your traces already follow a fixed schema, such as spans with stable attributes, plain columns beat Variant. Use it for the part that churns.

First step: copy one day of traces and race the queries

Don't migrate the lake. Convert one day and compare.

CREATE TABLE agent_traces_variant
USING iceberg
TBLPROPERTIES ('format-version' = '3')
AS SELECT trace_id, span_id, ts, agent,
          parse_json(payload_json) AS payload
FROM agent_traces_json
WHERE ts >= TIMESTAMP '2026-10-12 00:00:00'
  AND ts <  TIMESTAMP '2026-10-13 00:00:00';

Run your three most common dashboard queries against both tables, then repeat with write.parquet.shred-variants on. The table configuration docs cover both shredding properties.

For anyone running agents in production, trace storage is where flexible payloads meet expensive queries, and a typed column is a cheaper place to keep the part that never stops changing.