Iceberg 1.12 Adds Vectorized Variant Reads in Spark

Apache Iceberg 1.12.0 shipped with one line that matters to anyone landing agent traces in a lakehouse: Spark 4.0 and 4.1 now use vectorized Parquet reads for unshredded Variant columns. This post covers what the iceberg 1.12 variant changes are, what the release does not claim, and a Monday-morning test you can run on one trace table. The official releases page dates the release September 29, 2026.

Variant is Iceberg's type for semi-structured values. One column holds JSON-like values of different shapes, so row 1 can carry a tokens field while row 2 carries retries, and nobody files a schema migration.

What shipped in Iceberg 1.12

• Variant: vectorized Parquet reads for unshredded Variant columns on Spark 4.0 and 4.1. The GitHub release notes also list fixes for shredding large decimals, UTF-8 ordering of string bounds and malformed binary parsing, plus VariantType in the REST catalog OpenAPI spec. A community weekly roundup adds that Flink and Kafka Connect can write shredded Variant data.
• Support matrix: Spark 3.4 and Flink 2.0 are removed. Flink 2.2 and 2.3 are added.
• Geometry and geography: read and write in Parquet and Avro, with Spark 4.1 type mapping.
• V4 groundwork: a v4 manifest reader, writing Parquet and Avro manifests in the V4 layout, and relative paths in the spec. Dremio's upgrade guide is blunt that this is groundwork, not a production format, so don't plan a V4 migration yet.

Why Variant fits agent trace tables

Agent traces change shape constantly. Someone adds a tool call, a retry counter or a guardrail verdict, and a typed-column design turns each one into a migration ticket and a sprint-planning argument. With Variant, events of different shapes share one column, and you read fields out with variant_get, as the Iceberg Variant post shows.

The release notes say vectorized reads for unshredded columns, so that is the claim to test. Don't assume the same path for shredded files. The sources I read publish no benchmark numbers, so measure your own.

And the catch: Variant needs a format version 3 table, and Iceberg 1.12 does not convert existing tables for you.

Monday morning: test iceberg 1.12 variant on one trace table

1. Check your Spark and Flink versions

List every job that touches the table. Anything on Spark 3.4 or Flink 2.0 has no 1.12 runtime. Per Dremio's guide, the supported lines are Spark 3.5, 4.0 and 4.1, and Flink 1.20, 2.1, 2.2 and 2.3. The Variant work in this release targets Spark 4.0 and 4.1.

2. Start Spark with the 1.12.0 runtime

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.lab=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.lab.type=hadoop \
  --conf spark.sql.catalog.lab.warehouse=/tmp/iceberg-lab

This uses a throwaway local Hadoop catalog named lab. On Spark 4.0, swap the artifact for iceberg-spark-runtime-4.0_2.13, and confirm the exact artifact on the releases page before you pin it.

3. Create a v3 table and mix payload shapes

CREATE NAMESPACE IF NOT EXISTS lab.agents;

CREATE TABLE lab.agents.traces (
  trace_id STRING,
  step_payload VARIANT
)
USING iceberg
TBLPROPERTIES ('format-version' = '3');

INSERT INTO lab.agents.traces VALUES
  ('t1', parse_json('{"step": "plan", "model": "model-a", "tokens": 812}')),
  ('t1', parse_json('{"step": "tool_call", "tool": "crm_lookup", "retries": 2}')),
  ('t2', parse_json('{"step": "answer", "tokens": 240, "guardrail": "pass"}'));

SELECT
  trace_id,
  variant_get(step_payload, '$.step', 'string') AS step,
  variant_get(step_payload, '$.tokens', 'int') AS tokens
FROM lab.agents.traces;

Three rows, three shapes, one column, no ALTER TABLE. Then copy a day of real traces into it and time the scan against your current JSON-string table.

4. Check shared tables for position deletes with row data

SELECT count(*)
FROM prod.db.your_table.position_deletes
WHERE `row` IS NOT NULL;

Dremio's guide says the Java library no longer writes position delete files that embed deleted row values, and that rewrite_position_delete_files and rewrite_table_path reject existing ones. Run this before you upgrade a table that has maintenance jobs.

Gotchas

• Format version is a separate decision. The Java library, the engine runtime and the table format version change independently. Treat a move to v3 as one-way.
• Deprecated APIs were removed across Spark, Flink, Kafka Connect and BigQuery. Recompile custom code against 1.12 before you deploy it.
• Your catalog must support v3 and Variant. A managed catalog can lag the library, so check it first.
• Flink and Kafka Connect shredded writes are new. Test large decimals, null-heavy objects and long strings before a new writer joins a shared table.

In production, this turns 'the trace schema changed again' from a migration ticket into a query change, as long as you check your engines, catalog and table format version first.