Skip to content

[CELEBORN-2415] Add Celeborn Spark UI extension - #3793

Open
yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:celeborn-ui-mvp
Open

[CELEBORN-2415] Add Celeborn Spark UI extension#3793
yew1eb wants to merge 1 commit into
apache:mainfrom
yew1eb:celeborn-ui-mvp

Conversation

@yew1eb

@yew1eb yew1eb commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a new module celeborn-client-spark-3-ui — a Spark UI extension that renders a "Celeborn" tab in both the live Spark UI and the History Server (enabled via spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin, supports Spark 3.x and 4.x).

This is an initial version that lays the foundation; it intentionally starts minimal. The page shows:

  1. Summary — shuffle write/read bytes, time, speed, and shuffle-vs-task duration ratio, aggregated from Spark's built-in TaskMetrics
  2. Celeborn Properties — a collapsible table of spark.celeborn.* configs

The skeleton is: CelebornPlugin (live UI) / CelebornHistoryServerPlugin (SHS, via AppHistoryServerPlugin SPI) → listener aggregates into Spark KVStoreCelebornStatusStoreCelebornUITab renders; a javax/jakarta servlet bridge covers Spark 3.x and 4.x. This plugin/store/render pipeline is designed so that fine-grained Celeborn client metrics can be added as collapsible sections in follow-up PRs:

  • Shuffle Write Times — serialize / copy / queue wait / compress / queue stall / inflight wait / drain wait / max push RTT / slow push
  • Shuffle Read Times — chunk wait / decompress / retry wait / max chunk RTT / slow chunk
  • Shuffle Write Servers — per-worker push bytes, push count, RTT, soft/hard splits, congested counts, last failure reason
  • Shuffle Read Servers — per-worker chunk count, read bytes, total/max RTT
  • Shuffle Assignments — partition-to-worker slot assignments

Why are the changes needed?

Spark UI already shows stage/task-level shuffle metrics, but nothing Celeborn-specific: which workers served the data, push/chunk latencies, congestion and split events are invisible there, while Celeborn's own metrics sit in cluster-level Grafana dashboards that cannot be attributed back to a specific application. This tab closes that gap by putting per-application Celeborn shuffle information inside the Spark UI and History Server, alongside the job's stages and tasks.

Does this PR introduce any user-facing change?

Yes.
Setting spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin (with the extension jar on the classpath) adds a "Celeborn" tab to the Spark UI and History Server. Disabled by default; no effect otherwise.

Does this PR resolve a correctness bug?

No.

How was this patch tested?

Manual end-to-end verification:

image

@yew1eb
yew1eb marked this pull request as draft August 10, 2026 11:30
@yew1eb
yew1eb marked this pull request as ready for review August 10, 2026 13:38
@yew1eb
yew1eb force-pushed the celeborn-ui-mvp branch 2 times, most recently from 4f8bad3 to 68fc2f2 Compare August 10, 2026 13:40
@yew1eb
yew1eb marked this pull request as draft August 10, 2026 13:48
@yew1eb
yew1eb force-pushed the celeborn-ui-mvp branch 6 times, most recently from 7a9e389 to 47f3676 Compare August 10, 2026 16:27
@yew1eb yew1eb closed this Aug 10, 2026
@yew1eb yew1eb reopened this Aug 10, 2026
@yew1eb
yew1eb marked this pull request as ready for review August 10, 2026 17:30
@yew1eb

yew1eb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant