Making the Apache Spark VARIANT Type Live Up to Its Promise: A Benchmark, Two Optimizer Fixes, and a 3.5× TPC-DS Speedup

Why we started measuring Variant

There has been a lot of interest lately in migrating from JSON-as-string columns to the new native Apache Parquet VARIANT type. The design rationale is compelling: a self-describing binary encoding that avoids re-parsing text on every read, enables typed column access, and supports shredding — extracting frequently accessed fields into dedicated Parquet columns for faster projection and filtering.

The theory is clean. A string_json column pays a JSON-parsing cost on every single query. VARIANT, especially when shredded, should let the engine skip that work and read only the typed sub-columns a query actually touches. On paper, moving to VARIANT should be close to a free performance win for semi-structured analytics.

So we put it to the test. And the first numbers told a very different story.

The first surprise: Variant was slower

We ran an early comparison on a real-world–shaped workload: one calendar day of the GitHub Archive dataset (2023-01-06), roughly 4.4 million rows, where each row is an API event with a heavy, semi-structured payload column whose shape depends on the event type. That heterogeneity — no single stable schema for the whole column — is exactly the shape of the logs and lake JSON that motivate VARIANT in the first place.

The workload was read-only analytics queries (group-bys, aggregations, nested-field access, arrays, and mixed types), each written twice against the same data: once as a STRING column using get_json_object(), and once as a VARIANT column using variant_get(). Throughout this post we report query_median_s — Apache Spark's own reported query execution time, the median of 5 timed runs after 1 warmup run, excluding JVM startup — measured on Spark 4 in local mode.

The result was counterintuitive: on several of these queries, shredded VARIANT — the layout that is supposed to help reads most — came out slower than plain JSON-as-string, which re-parses text on every read. That is the opposite of what the design intends: all the queries are read-only, and shredding's cost is paid at write time, so at read time the typed side-columns should only help.

Seeing shredding lose to text parsing was a strong signal that something in the engine — not the type — was getting in the way. We treated these numbers as a diagnostic, not a verdict. The Spark VARIANT path was (and is) under active development. But to turn "something feels wrong" into an actionable engine fix, we needed a workload we could run, re-run, and share — and we could not find a repeatable, open one.

Building an open benchmark instead of guessing

Rather than reason about VARIANT performance from one-off scripts, we built and open-sourced variant-conformance-benchmark: a standalone testbed for validating correctness and measuring performance of the Parquet VARIANT type across engines.

Two design decisions matter for reproducibility:

• Correctness is a first-class mode. The harness runs a fixed query suite and compares results row-by-row against golden reference files, using two complementary sources of truth. For the small synthetic dataset, golden files are hand-authored from the raw JSON and the conformance spec — independently-verified ground truth. For the larger datasets, they are generated by Spark against the well-tested string_json path and used as a consistency oracle: the variant results must match what the stable JSON-string representation produces on the same data. Either way, any mismatch is a conformance failure.
• Benchmark mode measures query_median_s and writes per-query CSVs to results/ for comparison across runs, so a regression (or an improvement from an engine fix) is a diff, not a hunch.
• Engine-neutral by construction. Engines plug in through an adapter interface. Today only Spark is implemented (Spark native tables and Spark + Apache Iceberg); adapters for other engines are future work and the primary contribution we are asking for.

The benchmark ships five data strategies, each designed to stress a different aspect of VARIANT:

The two TPC-DS strategies run 12 queries (q07, q12, q19, q26, q42, q52, q55, q63, q68, q73, q79, q98) that exercise multi-table joins, GROUP BY aggregation, ORDER BY, and filter predicates — the shapes of real analytical SQL, not toy one-liners. Getting to VARIANT-in-realistic-queries is what surfaced the actual root causes.

What the benchmark surfaced: two optimizer problems

VARIANT is a young type — it landed in Spark 4.0, and the optimizer work to make a brand-new type fast naturally matures over several releases. Running these strategies against Spark 4 surfaced two such issues — one a bug, one a missed optimization — both now merged and slated for Apache Spark 4.3.

The whole point of shredding is to cut scan I/O: store frequently accessed fields as their own typed Parquet columns so a query reads only the sub-columns it touches, and nothing else. Both problems below break exactly that — each in a different way, they push the scan back to reading far more than the query needs, which is why shredded VARIANT ended up slower than a plain JSON string instead of faster.

One piece of Spark internals runs through both, so it is worth naming up front: Spark has two scan paths. DSv1 is the classic file source used by native managed Parquet tables; DSv2 is the newer DataSourceV2 path used by Iceberg and by path-based file relations. Where each fix applies comes down to this split, and each section below says which path it touches.

Problem 1 — Column pruning skipped on DSv2 Variant scans

SPARK-57499 · PR apache/spark#56556 · Fixed, fixVersion 4.3.0

We hit this one while running the benchmark against Iceberg, which reads through Spark's DSv2 scan path. Pruning already works on DSv1, so this bug only bites on DSv2 — it does nothing on native tables, but it matters for the read path Iceberg sits on. The effect: a scan reads far more of the table than the query asked for.

The rule that pushes variant_get() down into a DSv2 scan interacted with column pruning in a way that turned pruning off: whenever variant pushdown kicked in, the scan went back to reading every column in the table — even columns the query never referenced. Pushdown replaced the plan node that the pruning rule keyed off of, before pruning ran, so pruning became a no-op. On top of that, the scan pushed a whole-variant reconstruction (a full-variant read at path $) of the untouched VARIANT column into the read — reading and rebuilding the entire variant value from its shredded Parquet sub-columns, on every row, for a column the query never mentions. This is the exact inverse of what shredding is for: instead of reading fewer, typed sub-columns, the scan reads all of them and pays to reassemble the value it was trying to avoid touching.

The symptom that started this was slowness, not failure — on the Iceberg read path the queries ran correctly, just far slower than they should.

This is easy to see and easy to reproduce on stock Spark, no Iceberg required. Reading a Parquet directory through a path-based temporary view — with the DSv1 shortcut disabled — forces the DSv2 scan path, exactly as Spark's own regression test for this fix does:

SET spark.sql.sources.useV1SourceList = "";

CREATE OR REPLACE TEMPORARY VIEW gha_events_payload_variant_dsv2
USING parquet
OPTIONS (path 'file:/…/vcb.db/gha_events_payload_variant');

The table has a scalar type column and a heavy payload VARIANT column. Our benchmark's simplest GHA query touches only type and never mentions payload:

SELECT type, count(*) AS event_count
FROM gha_events_payload_variant_dsv2
GROUP BY type
ORDER BY event_count DESC, type ASC;

EXPLAIN EXTENDED shows the bug in a single line. Without the fix, the DSv2 BatchScan reads the entire seven-column schema — including a full reconstruction of payload:

BatchScan ...[id, type, actor, repo, payload, public, created_at] ParquetScan
  PushedVariantExtractions: [payload:{"path":"$", ...}:VariantType]   -- rebuild whole payload, per row
  ReadSchema: struct<id:string,type:string,actor:string,repo:string,payload:variant,public:boolean,created_at:string>

With the fix, pruning works again: the scan reads only type, pushes nothing, and reconstructs nothing:

BatchScan ...[type] ParquetScan
  PushedVariantExtractions: []
  ReadSchema: struct<type:string>

Both plans return the identical 16 result rows — the fix changes only what is read, never what is computed. On this ~4.4M-row table, reading (and rebuilding) a VARIANT column the query never asked for is not a rounding error:

That is a ~25× speedup on a single trivial query, entirely from the engine reading less, with no query rewrite. The larger the unreferenced VARIANT payload — and the more VARIANT columns a scan touches — the worse the pre-fix behavior, which is exactly the kind of table (heavy, multi-column, semi-structured) that motivates adopting VARIANT in the first place.

Methodology: both times are the warm second run of the query in the same spark-sql session (the first, cold run is discarded), measured on the Spark 4.3 branch with this fix backed out versus applied — so the only variable is the fix.

Being DSv2-only, this fix does not show up in the native TPC-DS numbers below — but it is exactly what any DSv2 Variant read path needs, Iceberg included.

Problem 2 — Extraction pushdown doesn't reach through Aggregate, Sort, or Join

SPARK-58089 · PR apache/spark#57190 · Improvement, Fixed, fixVersion 4.3.0

Unlike Problem 1, this one affects both scan paths — DSv1 and DSv2 alike — and it is the one with the larger performance impact. It left extraction pushdown unused on nearly every realistic analytical query. Spark classifies it as an improvement rather than a bug — nothing crashes and results are correct — but the missed optimization is significant.

The query that surfaced it was one of the simplest in the GHA suite — q05, an average and max of a single nested numeric field over one table:

-- q05: variant_get('$.size') inside AVG and MAX, plus an IS NOT NULL filter
SELECT round(avg(variant_get(payload, '$.size', 'double')), 6) AS avg_commits,
       max(variant_get(payload, '$.size', 'long'))             AS max_commits
FROM vcb.gha_events_variant
WHERE type = 'PushEvent'
  AND variant_get(payload, '$.size', 'long') IS NOT NULL;

The existing pushVariantExtractions rule could only push a variant_get() call into the scan when it sat directly above the scan. In real analytical queries, though, the call is almost always buried one level up — inside an aggregate argument like AVG(variant_get(...)), a GROUP BY key, a JOIN condition, or an ORDER BY. The Aggregate, Join, and Sort operators sitting in between meant the rule could not yet see the extraction, so it stayed in place.

The EXPLAIN output makes the gap precise. Note that this is not "no pushdown at all" — the IS NOT NULL filter predicate pushes into the scan fine either way. What stays stuck is the aggregate. Without the fix, variant_get('$.size') remains inside the Aggregate, and the $.size value it needs is read back as a whole variant sub-column and re-decoded on every row:

-- WITHOUT the fix (Optimized Logical Plan + scan)
Aggregate [round(avg(variant_get(payload, $.size, DoubleType)), 6) …, max(variant_get(payload, $.size, LongType)) …]
+- …
   +- FileScan … ReadSchema: struct<type:string, payload:struct<0:variant, 1:bigint>>
                                              -- $.size arrives as a variant, decoded per row ↑

The fix adds a new rule, PullOutVariantExtractions, that pulls those variant_get() calls out of the Aggregate, Sort, and Join operators and drops them into a Project sitting directly above the scan (_ve0, _ve1 below). From there the existing pushdown rule can see them and push them the rest of the way in — so the aggregate operates on plain pre-computed columns, and the scan reads $.size as a directly-typed double sub-column:

-- WITH SPARK-58089 (Optimized Logical Plan + scan)
Aggregate [round(avg(_ve0), 6) …, max(_ve1) …]
+- Project [payload.0 AS _ve0, payload.1 AS _ve1]      -- variant_get lifted out of the aggregate
   +- FileScan … ReadSchema: struct<type:string, payload:struct<0:double, 1:bigint>>
                                              -- $.size now read as a typed double ↑

Every TPC-DS query in the benchmark is shaped the same way — the extraction is buried inside an aggregate, GROUP BY, join, or sort. TPC-DS q07, for instance, wraps variant_get() in AVG arguments and a join-side filter across a five-table join. So each of them left variant_get() stranded above the scan, decoding a whole variant per row instead of reading the one typed sub-column the query needed.

The numbers after the fixes

These run on native DSv1 tables, not on Iceberg: Iceberg's scan builder doesn't yet implement extraction pushdown, so there is nothing for the optimizer fixes to hook into on that path and no before/after to show there. Running on DSv1 also isolates Problem 2 cleanly — we compare the same 12 TPC-DS queries on the same machine and data across stock Spark 4.3 (extraction pullout disabled), the same build with SPARK-58089 active, and the string_json column a team would be migrating from. All numbers are query_median_s, Spark's own execution time, the median of 5 timed runs after 1 warmup run, on the Spark 4.3 branch, at TPC-DS scale factor 5, tpcds-flat (a single VARIANT column per fact table — the layout a team migrating a JSON column would most naturally adopt). Every column comes from the same run.

All three columns come from a single run on the Spark 4.3 branch: "stock" toggles SPARK-58089 off, "with SPARK-58089" toggles it on, and "String JSON baseline" is the string_json representation from the same run. Scale-factor-5 TPC-DS, single machine, local mode, native DSv1 tables. Bold marks the fastest configuration in each row.

The pushdown fix is decisive: total query time drops from 26.0 s to 7.3 s — a 3.5× speedup (72% faster) on stock shredded VARIANT.

Just as important is what that does to the migration comparison. Before the fix, shredded VARIANT was 3.2× slower than the plain JSON-string column (26.0 s vs. 8.0 s) — the very "Variant is slower" result the early diagnostic flagged. After the fix, VARIANT lands at 7.3 s versus string's 8.0 s — roughly 9% faster, essentially at parity. The fix doesn't just make VARIANT faster than its unoptimized self; it moves VARIANT from clearly-losing to slightly-ahead of the representation teams are migrating from. The two queries that barely move (q12, q19) are simple projections that never place variant_get() inside an aggregate, sort, or join — there is nothing to pull out and push down, which is exactly what we would expect. A few queries (q42, q55, q07) land fractionally behind string at this scale, because their extraction paths are already cheap enough that the typed layout does not pull ahead. The win is large and consistent, not uniform — and being able to say which queries move, and why, is a direct benefit of measuring inside a structured benchmark rather than eyeballing a total. We chose the single-column tpcds-flat layout deliberately, as the conservative case a typical migration would build; on a schema with several VARIANT columns, or on the DSv2/Iceberg path where Problem 1 also applies, the two fixes compound into a larger combined speedup than the pushdown-only number here.

These are scale-factor-5 TPC-DS results from a single developer machine in local mode — meant to be directional, isolating optimizer behavior rather than reporting production throughput. The ratios between configurations are the takeaway; absolute times will differ on production hardware and at larger scale.

What this means for Variant adoption

The early GitHub Archive results were a diagnostic, and they pointed at the engine, not the type. VARIANT's type-safe extraction and shredded column layout mean the engine should skip full Parquet reads for selective filters and typed aggregations — but only if the optimizer can actually see the variant_get() calls and read only the sub-columns they touch. When the DSv2 pruning bug (SPARK-57499) and the pushdown gap (SPARK-58089) blocked that path, shredded VARIANT not only lost the read-time benefit — it paid extra, reconstructing the whole variant value from its shredded sub-columns on every row, work a plain JSON string never does. That is precisely why it measured slower than string. Both fixes are now merged and shipped in Apache Spark 4.3.

If your workload uses variant_get() in GROUP BY, aggregation, or JOIN conditions against Spark 4, upgrading to Spark 4.3 unlocks the pushdown speedup with no query rewrites — and if you read Variant through a DSv2 source, you also get the pruning fix. Both fixes live in the Spark optimizer on the DSv2 read path Iceberg is built on, so Iceberg tables will inherit the same wins once Iceberg implements extraction pushdown on its scan — work now in flight in the Iceberg community.

Both fixes were shaped by review from the Apache Spark community, and we are grateful to the reviewers and committers who helped refine and merge them. Maturing a type as capable as VARIANT is a collaborative effort, and contributing these fixes upstream — where every Spark user benefits — was the whole point of measuring in the open.

Try the benchmark on your own data

The benchmark harness, query SQL, data-prep scripts, and result-comparison tooling are all open source at cloudera-labs/variant-conformance-benchmark. The quickstart guides under docs/ walk through each strategy, starting with a synthetic strategy that needs no downloads.

The two most useful contribution directions are:

• New engine adapters — Trino, Flink, DuckDB, or anything else with a VARIANT path. The adapter interface is the plug-in point; see docs/adding-an-engine.md.
• New data strategies — different JSON shapes, sizes, and query patterns, especially datasets where representations diverge in interesting ways. See docs/adding-a-strategy.md.

Re-running the benchmark against it on your own data is a concrete way to confirm the fixes hold for your workload — not just for TPC-DS. If you have a dataset where VARIANT shines, or where it should but doesn't, open an issue.

Resources

• Benchmark: cloudera-labs/variant-conformance-benchmark
• Spark fixes (Spark 4.3): SPARK-57499 — column pruning (PR #56556) · SPARK-58089 — extraction pullout (PR #57190)
• Parquet Variant encoding spec: VariantEncoding.md
• Apache Repos: Iceberg, Parquet, Spark