How to Upgrade to Apache Iceberg 1.12 Without Breaking Jobs

Iceberg 1.12.0 removed support for Spark 3.4 and Flink 2.0, and its Java writers no longer write position delete files that embed row data. If you upgrade Apache Iceberg 1.12 by bumping a version string, a Spark 3.4 job has no iceberg-spark-runtime-3.4 to land on, and a maintenance job can fail on a table you thought was healthy. This runbook finds those problems first, rolls the upgrade out in a safe order and ends with a Variant read as a smoke test. Version facts come from the Iceberg releases page.

Step 1: Find jobs still on Spark 3.4 or Flink 2.0

Check the engine version of every job, then grep repos and deploy scripts for the removed runtimes.

spark-submit --version 2>&1 | grep -i 'version'
<flink-home>/bin/flink --version
grep -rEn 'iceberg-(spark-runtime-3[.]4|flink-runtime-2[.]0)' . --include='*.xml' --include='*.gradle*' --include='*.sh' --include='*.yaml'

Anything that matches needs a new engine before it gets a new Iceberg. The nearest targets are Spark 3.5 (1.12 builds against 3.5.9) and Flink 2.1. For stateful Flink jobs, restore a savepoint on staging with the exact Flink and Iceberg pair before you trust it.

Step 2: Scan for position deletes that carry row data

Iceberg 1.12 stops writing position deletes with row data (deprecated in 1.11) but still reads old ones. The catch is maintenance: per pull request #17706, rewrite_position_delete_files and rewrite_table_path now fail with an IllegalArgumentException on tables that contain them, because the maintainers chose failing loudly over silently dropping row data. The position_deletes metadata table keeps its row column, filled only for those older files, so it works as a detector.

SELECT delete_file_path, count(*) AS delete_rows_with_data
FROM prod.db.events.position_deletes
WHERE `row` IS NOT NULL
GROUP BY delete_file_path;

To sweep a whole namespace:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
catalog, db = 'prod', 'db'

for t in spark.sql(f'SHOW TABLES IN {catalog}.{db}').collect():
    name = f'{catalog}.{db}.{t.tableName}'
    try:
        n = spark.sql(f'SELECT count(*) AS n FROM {name}.position_deletes WHERE `row` IS NOT NULL').first().n
        if n:
            print(f'{name}: {n} position delete rows with row data')
    except Exception as e:
        print(f'{name}: skipped ({type(e).__name__})')

Run the sweep off-peak or on a staging copy, since it reads delete files. Iceberg won't convert flagged files in place. One route to try on staging is compacting so the deletes are applied into new data files, then expiring old snapshots and re-running the scan.

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map('delete-file-threshold', '1')
);

CALL prod.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '<cutoff-timestamp>',
  retain_last => 5
);

delete-file-threshold is how many deletes must be linked to a data file before it is rewritten (the default is effectively never). Expiring snapshots ends time travel to them, so choose the cutoff deliberately.

Step 3: Upgrade Apache Iceberg 1.12 with the matching runtime jar

The 1.12.0 runtimes are named after the engine:

• Spark 3.5: iceberg-spark-runtime-3.5_2.12 or iceberg-spark-runtime-3.5_2.13
• Spark 4.0: iceberg-spark-runtime-4.0_2.13
• Spark 4.1: iceberg-spark-runtime-4.1_2.13
• Flink 1.20, 2.1, 2.2 and 2.3: iceberg-flink-runtime-<flink-version>

For Spark, swap the package version and keep your catalog settings:

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0 \\
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\
  --conf spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog \\
  --conf spark.sql.catalog.prod.type=rest \\
  --conf spark.sql.catalog.prod.uri=<rest-catalog-uri>

For Flink, replace the old runtime jar in lib/ so two versions never share a classpath:

cd <flink-home>
rm lib/iceberg-flink-runtime-*.jar
wget -P lib/ https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-2.1/1.12.0/iceberg-flink-runtime-2.1-1.12.0.jar

The Iceberg Flink docs recommend lib/ over the SQL client's -j flag because of a regression (FLINK-30035) on Flink 1.16 and later.

Step 4: Check Kafka Connect sinks and custom code

The 1.12 Kafka Connect sink surfaces commit failures instead of swallowing them, adds a metric for partial commit failures and retries transient commit exceptions a bounded number of times. Expect louder failures at first, and watch task state after the rollout.

curl -s http://<connect-host>:8083/connectors/<connector-name>/status

Deprecated members were also removed from TableReference and IcebergWriterResult, so custom transforms may stop compiling. Recompile any Java code against 1.12.0 and fix what breaks. PositionDeltaWriter.delete with a row argument is deprecated for removal in 1.13.0 and already ignores the row.

Step 5: Roll out by writer risk

Upgrade stateless readers first, then low-volume writers, then streaming writers, and maintenance jobs last. Keep one reader on the old version until the new one has survived a full maintenance cycle. Don't touch format-version in the same window: the library, the engine runtime and the table format are three separate decisions.

Step 6: Verify a Variant read on the new build

Iceberg 1.12 adds vectorized Parquet reads for Variant in Spark. Variant needs a format-version 3 table and Spark 4.0 or 4.1, as the Iceberg Variant post describes. Run this on staging with the 1.12 runtime:

CREATE TABLE prod.db.variant_smoke (id BIGINT, payload VARIANT)
USING iceberg
TBLPROPERTIES ('format-version' = '3');

INSERT INTO prod.db.variant_smoke VALUES
  (1, parse_json('{"event": "login", "country": "US"}')),
  (2, parse_json('{"event": "purchase", "country": "UK", "amount": 99}')),
  (3, parse_json('{"event": "login"}'));

SELECT id,
       variant_get(payload, '$.event', 'string') AS event,
       variant_get(payload, '$.amount', 'int')   AS amount
FROM prod.db.variant_smoke
ORDER BY id;

DROP TABLE prod.db.variant_smoke;

You should get three rows, with amount equal to 99 on row 2 and null on the others. Wrong values or an error here means stop the rollout.

Gotchas

• Opening a v3 table doesn't prove an engine supports every v3 feature. Test each one you use.
• Failing rewrite_position_delete_files or rewrite_table_path jobs on old tables are the intended behavior, not a regression.
• Spark 3.5 has two Scala builds, so match the _2.12 or _2.13 suffix to your Spark distribution.
• format-version=4 is groundwork only in 1.12, not a supported migration.

If your AI pipelines read features or training data from Iceberg tables, a stuck maintenance job or a failed sink shows up downstream as stale data, so finding it on staging is far cheaper than explaining it later.