[CELEBORN-2408] Exclude unused Netty QUIC/HTTP3 modules from shaded clients - #3787
Open
yew1eb wants to merge 1 commit into
Open
[CELEBORN-2408] Exclude unused Netty QUIC/HTTP3 modules from shaded clients#3787yew1eb wants to merge 1 commit into
yew1eb wants to merge 1 commit into
Conversation
yew1eb
force-pushed
the
CELEBORN-2408
branch
2 times, most recently
from
August 7, 2026 09:26
a45fc9d to
2ea3bd0
Compare
…lients Netty 4.2 merged the QUIC/HTTP3 stack into netty-all: netty-codec-http3 -> netty-codec-classes-quic -> netty-codec-native-quic. The latter ships a libnetty_quiche42_* native library per platform classifier (linux/osx/windows x86_64/aarch_64), ~29MB uncompressed. Since every *-shaded module uses maven-shade-plugin with <include>io.netty:*</include>, all of it lands in every shaded client jar, which grew celeborn-client-spark-3-shaded from 15MB (0.6.3, Netty 4.1.118) to 27MB (1.0.0-SNAPSHOT, Netty 4.2.10). Celeborn only uses Netty TCP RPC; a full-repo grep confirms zero references to codec.quic / codec.http3 / quiche in source. Exclude the whole trio at the netty-all declaration in both pom.xml (maven build) and project/CelebornBuild.scala (sbt build), since sbt does not inherit maven exclusions. Also update the dev/deps/dependencies-* snapshots (19 files) to drop the removed quic artifacts, so the maven/sbt dependency-list CI checks stay green. Verified: - spark-3-shaded and flink-1.20-shaded jars drop from 27MB to 15MB, with zero quiche native libs and zero quic/http3 classes. - epoll/kqueue native libs (renamed by antrun) remain present. - common dependency:tree: quic nodes 7 -> 0 after the exclusion. - ./dev/dependencies.sh --check --module spark-3.5 (maven): no diff. - ./dev/dependencies.sh --sbt --check --module service / spark-3.5: no diff.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Exclude
netty-codec-http3,netty-codec-classes-quic, andnetty-codec-native-quicfrom thenetty-alldependency in the root pom's dependencyManagement.Why are the changes needed?
Since the Netty
4.1.118→4.2.10.Finalupgrade,netty-allpulls in the QUIC/HTTP3 stack that 4.2 merged into the mainline:netty-codec-native-quicships alibnetty_quiche42_*native library per platform classifier (linux-x86_64,linux-aarch_64,osx-x86_64,osx-aarch_64,windows-x86_64) — ~29MB uncompressed / ~15MB compressed in total, plus 156 QUIC classes + 112 HTTP3 classes.Because every
*-shadedmodule usesmaven-shade-pluginwith<include>io.netty:*</include>, the wildcard drags all of the above into every shaded client jar. As a resultceleborn-client-spark-3-shadedgrew from 15MB (0.6.3, Netty 4.1.118) to 27MB (1.0.0-SNAPSHOT, Netty 4.2.10). The same bloat applies to all 11 shaded modules (spark-3/4, flink 1.18~2.3, mr, tez) and to the runtime classpath of common/client/master/service/worker.Celeborn only uses Netty TCP RPC; a full-repo grep confirms zero references to
codec.quic/codec.http3/quichein source. These are pure dead weight.Excluding them at the
netty-alldeclaration (inherited by all modules via dependencyManagement) drops the shaded jars back to ~15MB with no functional impact.Does this PR resolve a correctness bug?
Does this PR introduce any user-facing change?
How was this patch tested?
celeborn-client-spark-3-shaded(spark-3.5 profile) andceleborn-client-flink-1.20-shaded(flink-1.20 profile). Each jar shrank from 27MB to 15MB, with zero*quiche*native libs and zerohandler/codec/quic/**/handler/codec/http3/**classes.mvn dependency:treeoncommon: quic dependency nodes dropped from 7 → 0 after the exclusion, confirming it takes effect at the dependency resolution layer for all inheriting modules.