[CELEBORN-2415] Add Celeborn Spark UI extension - #3793
Open
yew1eb wants to merge 1 commit into
Open
Conversation
yew1eb
marked this pull request as draft
August 10, 2026 11:30
yew1eb
marked this pull request as ready for review
August 10, 2026 13:38
yew1eb
force-pushed
the
celeborn-ui-mvp
branch
2 times, most recently
from
August 10, 2026 13:40
4f8bad3 to
68fc2f2
Compare
yew1eb
marked this pull request as draft
August 10, 2026 13:48
yew1eb
force-pushed
the
celeborn-ui-mvp
branch
6 times, most recently
from
August 10, 2026 16:27
7a9e389 to
47f3676
Compare
yew1eb
force-pushed
the
celeborn-ui-mvp
branch
from
August 10, 2026 16:57
47f3676 to
e053012
Compare
yew1eb
marked this pull request as ready for review
August 10, 2026 17:30
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a new module
celeborn-client-spark-3-ui— a Spark UI extension that renders a "Celeborn" tab in both the live Spark UI and the History Server (enabled viaspark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin, supports Spark 3.x and 4.x).This is an initial version that lays the foundation; it intentionally starts minimal. The page shows:
TaskMetricsspark.celeborn.*configsThe skeleton is:
CelebornPlugin(live UI) /CelebornHistoryServerPlugin(SHS, viaAppHistoryServerPluginSPI) → listener aggregates into SparkKVStore→CelebornStatusStore→CelebornUITabrenders; ajavax/jakartaservlet bridge covers Spark 3.x and 4.x. This plugin/store/render pipeline is designed so that fine-grained Celeborn client metrics can be added as collapsible sections in follow-up PRs:Why are the changes needed?
Spark UI already shows stage/task-level shuffle metrics, but nothing
Celeborn-specific: which workers served the data, push/chunk latencies, congestion and split events are invisible there, while Celeborn's own metrics sit in cluster-level Grafana dashboards that cannot be attributed back to a specific application. This tab closes that gap by putting per-application Celeborn shuffle information inside the Spark UI and History Server, alongside the job's stages and tasks.Does this PR introduce any user-facing change?
Yes.
Setting
spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin(with the extension jar on the classpath) adds a "Celeborn" tab to the Spark UI and History Server. Disabled by default; no effect otherwise.Does this PR resolve a correctness bug?
No.
How was this patch tested?
Manual end-to-end verification: