[#2771] feat(spark): Support Spark 4.2

[LuciferYang]

What changes were proposed in this pull request?

Add a mutually-exclusive spark4.2 Maven profile that builds the existing client-spark/spark4 module against Apache Spark 4.2.0, mirroring the spark4.1 profile added in #2751.

• Root pom.xml: new spark4.2 profile — a structural clone of spark4.1, differing only in spark.version=4.2.0, scala.version=2.13.18, netty.version=4.2.13.Final, jackson.version=2.21.2, and the lz4 coordinate (lz4.groupId + lz4.version, see below).
• client-spark/spark4/pom.xml: spark4.2 profile reuses the src/main/java-spark4_1 compat source root (the 4.1 Spark4Compat shim).
• client-spark/extension/pom.xml: spark4.2 profile reuses the scala-jakarta servlet source root.
• client-spark/extension — ShufflePage.scala: the Uniffle UI tab's collapsible sections use onClick="collapseTable(...)", relying on a global collapseTable() that Spark 3.x/4.0/4.1 ship in webui.js. Spark 4.2 removed that function (its own pages moved to Bootstrap 5 data-bs-toggle collapse), so under -Pspark4.2 the sections could not be expanded. Inline a fallback that defines a compatible collapseTable only when the host Spark didn't provide one — 3.x/4.0/4.1 keep Spark's implementation, 4.2 gets a working equivalent, reusing the .collapsed / arrow-open|closed CSS every Spark line still ships.
• lz4: introduce ${lz4.groupId} (org.lz4 by default, at.yawk.lz4 under spark4.2) and ${lz4.version} (1.8.1 by default, 1.11.0 under spark4.2); the two integration-test modules declare ${lz4.groupId}:lz4-java, and the at.yawk.lz4 root-dependencyManagement entry is pinned to ${lz4.version}. The default build and every non-spark4.2 profile stay on org.lz4:1.8.1 — the bump is scoped to spark4.2 only.
• CI: parallel.yml gains a spark4.2 / java 17 matrix entry; sequential.yml gains a -Pspark4.2 step gated on java-version == 17.

No new Maven module. client-spark/spark4-shaded needs no relocation change (its bouncycastle/jctools relocations are already global); a comment was added there noting its lz4 exclusion must stay org.lz4 — Maven matches exclusions on the as-declared coordinate, so parameterizing it would leak at.yawk.lz4 into the shaded jar under -Pspark4.2.

Why are the changes needed?

Fix: #2771

Users should be able to build and run Uniffle's Spark client against Spark 4.2.0 with full parity to the existing 4.1 support.

The three cross-major APIs called from client-spark/spark4 are byte-identical between 4.1.1 and 4.2.0 (verified with javap), so no shim change is required:

API	4.1.1	4.2.0
MapStatus.apply(BlockManagerId, long[], long, long)	4-arg	identical
ExternalSorter(…, RowBasedChecksum[]) ctor	6-arg	identical
WebUIPage.render(jakarta…HttpServletRequest)	jakarta	identical

Dependency notes:

• jackson → 2.21.2 (required): Spark 4.2.0 ships jackson-module-scala_2.13 2.21.2, which validates the classpath jackson-databind on registration and throws in RDDOperationScope's static initializer if it is outside [2.21.0, 2.22.0). Same class of failure the 4.1 work hit.
• netty → 4.2.13.Final: matches what Spark 4.2.0 was compiled/tested against (still Netty 4.2, same major as 4.1's 4.2.7, so [#2750] feat(spark): Support Spark 4.1 #2751's NioIoHandler/refCnt handling still applies — no code change).
• lz4 → at.yawk.lz4:1.11.0, scoped to spark4.2 (required): Spark 4.2.0 bumped lz4 to 1.11.0 and calls LZ4BlockInputStream.newBuilder() (added in the 1.11 line); without it the integration tests fail at runtime with NoSuchMethodError. Because org.lz4:1.8.1 is a relocation stub that Maven rewrites to at.yawk.lz4:1.8.1, a globally-managed at.yawk.lz4:1.11.0 would silently bump lz4 across the whole reactor (server/client/coordinator/…), which is out of this PR's scope and would override the org.lz4:1.8.1 pin from chore: Update lz4 to address CVE-2025-12183 #2693 (CVE-2025-12183). The ${lz4.groupId}/${lz4.version} properties confine the bump to the spark4.2 profile; every other build stays on 1.8.1.
• hadoop / jetty / log4j / slf4j / commons-lang3 deltas are deferred; the profile keeps the 4.1 values.

Does this PR introduce any user-facing change?

No behavioral change to existing profiles. Adds a new opt-in -Pspark4.2 build profile, and fixes the Uniffle Spark UI tab so its collapsible sections expand under Spark 4.2. Existing -Pspark4 (4.0.2) and -Pspark4.1 (4.1.1) are unchanged.

How was this patch tested?

Verified locally on JDK 17 (Zulu 17.0.18):

1. mvn clean install -Pspark4.2 -DskipTests — compiles + shades against 4.2.0 (spark-core/sql/catalyst 4.2.0, scala 2.13.18, netty 4.2.13.Final, jackson 2.21.2, lz4 at.yawk.lz4:1.11.0 resolved).
2. mvn clean install -Pspark4.1 -DskipTests + integration tests — 4.1.1 build and AQERepartitionTest / MapSideCombineTest still green (lz4 stays org.lz4:1.8.1); confirms no regression. Also confirmed the default build (no profile) and other spark profiles keep lz4 at 1.8.1.
3. AQERepartitionTest, MapSideCombineTest pass under -Pspark4.2.
4. TransportFrameDecoderTest (2 tests) passes on Netty 4.2.13 under -Pspark4.2.
5. RepartitionWithLocalFileRssTest (NOOP/ZSTD/LZ4 codecs) passes under -Pspark4.2 — exercises the lz4 runtime path.
6. dev/scripts/checkshade.sh on the spark4.2 shaded jar — "check success", 0 unshaded classes; bouncycastle/jctools relocated, no net.jpountz (lz4) leak.
7. End-to-end on a real Spark 4.2.0 distribution against a local Uniffle cluster (coordinator + shuffle server): a shuffle job runs through RssShuffleManager, and the Uniffle Spark UI tab renders and its collapsible sections (Build Info / Shuffle Server / Assignment / …) expand with no page JS errors.

CI runs the build + unit + integration tests via the new spark4.2 matrix entry / -Pspark4.2 step.

[github-actions]

Test Results

 3 833 files  +  216   3 833 suites  +216   8h 11m 42s ⏱️ + 27m 16s
 1 265 tests ±    0   1 254 ✅ ±    0  11 💤 ± 0  0 ❌ ±0 
19 153 runs  +1 098  19 105 ✅ +1 087  48 💤 +11  0 ❌ ±0 

Results for commit 9740fdf. ± Comparison against base commit ec77694.

♻️ This comment has been updated with latest results.

[LuciferYang]

all test passed

[LuciferYang]

End-to-end UI verification on a real Spark 4.2.0 cluster

Beyond the automated build/tests, I verified the Uniffle Spark UI tab end-to-end against a real Spark 4.2.0 distribution with a local Uniffle cluster (coordinator + shuffle server). A shuffle job was run through RssShuffleManager with spark.plugins=org.apache.spark.UnifflePlugin, then the Uniffle tab was driven with a headless browser.

This also surfaced and fixed a Spark 4.2-specific UI regression: ShufflePage relied on the global collapseTable() that Spark 3.x/4.0/4.1 ship in webui.js, which Spark 4.2 removed (its own pages moved to Bootstrap 5 data-bs-toggle collapse). Without the fix in this PR, clicking a section header threw collapseTable is not defined and no section could be expanded. The fix inlines a fallback that defines a compatible collapseTable only when the host Spark didn't provide one (so 3.x/4.0/4.1 keep Spark's implementation).

Uniffle tab — sections expanded (Build Info / Shuffle Server / Assignment):

The footer shows 4.2.0; the expanded tables show Version 0.11.0-SNAPSHOT, the shuffle server id / write+read bytes, and the per-shuffle assignment — confirming the listener collected real shuffle stats and the jakarta servlet variant renders correctly on 4.2. No page JS errors.

Uniffle tab — default (collapsed) view:

[LuciferYang]

cc @zuston @roryqi @wForget FYI

[wForget]

Why don't we modify the lz4 group in the common module?

[LuciferYang]

I tried exactly that and it breaks the shaded client, which is why the exclusion has to keep the literal org.lz4.

If I switch common/pom.xml (and client-spark/common/pom.xml) to ${lz4.groupId}, then under -Pspark4.2 the transitive lz4 enters the graph as at.yawk.lz4:lz4-java, but spark4-shaded's exclusion matches on the as-declared coordinate, so <groupId>org.lz4</groupId> no longer matches and lz4 leaks into the shaded jar:

unshaded count: 94
check failed.       # dev/scripts/checkshade.sh, 110 net/jpountz/** classes in the jar

Parameterizing the exclusion too (<groupId>${lz4.groupId}</groupId>) doesn't save it either — still 94 unshaded and the checkshade antrun step fails the build. With the current code (literal org.lz4 in both places) the same clean build gives net/jpountz count = 0 and check success.

There's also no benefit to changing it: the only 1.11-only API is LZ4BlockInputStream.newBuilder(), and that's called by Spark's own LZ4CompressionCodec, never by Uniffle. Uniffle's Lz4Codec only uses LZ4Factory.safeInstance() / fastCompressor / fastDecompressor / maxCompressedLength, all of which exist in 1.8.1 and 1.11.0 alike (the net.jpountz package name is unchanged by the groupId relocation). And on a real cluster the shaded client excludes lz4 and defers to Spark's env lz4 anyway.

So the split is deliberate: only the modules that must follow Spark's lz4 on the test classpath (integration-test/spark-common, integration-test/spark4) use ${lz4.groupId}; common / client-spark/common stay on the pre-relocation coordinate so the shade exclusion keeps working. That's what the comment on this line is there to prevent someone from "fixing" later. Happy to reword the comment if it reads more like a puzzle than a warning.

[wForget]

I tried that approach before as well, and it didn't seem to cause any CI failures (https://github.com/apache/uniffle/pull/2768/checks). However, as you explained, there's also no benefit to changing it.

[wForget]

Thanks @LuciferYang

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.07%. Comparing base (ec77694) to head (9740fdf).

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2772      +/-   ##
============================================
+ Coverage     51.03%   51.07%   +0.03%     
- Complexity     3362     3366       +4     
============================================
  Files           533      533              
  Lines         25998    25998              
  Branches       2379     2379              
============================================
+ Hits          13268    13278      +10     
+ Misses        11868    11852      -16     
- Partials        862      868       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[LuciferYang]

GHA passed, I will merge this one

[LuciferYang]

Merged into master. Thanks @wForget @roryqi