Skip to content

[CELEBORN-2412] Propagate the worker error message through cpp MessageDispatcher failures - #3791

Closed
yugan95 wants to merge 1 commit into
apache:mainfrom
yugan95:CELEBORN-2412
Closed

[CELEBORN-2412] Propagate the worker error message through cpp MessageDispatcher failures#3791
yugan95 wants to merge 1 commit into
apache:mainfrom
yugan95:CELEBORN-2412

Conversation

@yugan95

@yugan95 yugan95 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

MessageDispatcher::read decodes the error string a worker sends back on RPC_FAILURE / CHUNK_FETCH_FAILURE and then throws it away — both paths fulfil the pending promise with a blank std::exception():

holder.msgPromise.setException(folly::exception_wrapper(std::exception()));

This PR builds the exception from the message instead:

  • RPC_FAILUREfolly::make_exception_wrapper<std::runtime_error>(failure->errorMsg())
  • CHUNK_FETCH_FAILURE → same, using the already-formatted errorMsg that includes the streamChunkSlice context.

MessageDispatcherTest is strengthened accordingly: a shared takeExceptionMessage() helper is added, the two failure tests now assert on the message content rather than merely hasException(), and sendPushDataAndReceiveFailure uses a real StatusCode name (PUSH_DATA_FAIL_PARTITION_NOT_FOUND) so it exercises the classification path it is meant to protect.

Why are the changes needed?

ShuffleClientImpl::getPushDataFailCause classifies a push failure by matching StatusCode names as substrings of the error message, and PushDataCallback::onFailure / PushMergedDataCallback feed it exception->what(). With a blank std::exception, what() is a fixed runtime string that matches none of the 13 candidate causes, so classification always falls through to PUSH_DATA_FAIL_NON_CRITICAL_CAUSE. Two consequences:

  1. excludeWorkerByCause only acts on the connection-fail / timeout causes and hits its default: branch every time, so celeborn.client.excludeWorker.enabled never actually excludes a worker on the C++ push path.
  2. When revive attempts are exhausted, classifyPushFailure raises "PUSH_DATA_FAIL_NON_CRITICAL_CAUSE: <blank>" — the worker's real reason (partition not found, worker excluded, replica write failure, …) is gone from the task-failure message, which makes these failures very hard to diagnose from client logs alone.

Chunk fetch failures lose the worker's diagnostics the same way.

This restores parity with the Java client, where TransportResponseHandler surfaces the worker's error text and ShuffleClientImpl#getPushDataFailCause can act on it.

Does this PR resolve a correctness bug?

  • Yes

Shuffle output is unaffected, so this is not a result-correctness bug. It is a failure-classification bug: worker exclusion never engages on the C++ push path, and the worker's reason is missing from the task-failure message.

Does this PR introduce any user-facing change?

  • Yes

No config, API or behaviour change. Push and fetch failure messages now carry the worker's reason instead of a placeholder.

How was this patch tested?

Unit tests in cpp/celeborn/network/tests/MessageDispatcherTest.cpp:

  • sendRpcRequestAndReceiveFailure — asserts the RPC failure message reaches the caller.
  • sendPushDataAndReceiveFailure — asserts a StatusCode name (PUSH_DATA_FAIL_PARTITION_NOT_FOUND) survives on the exception, which is exactly what getPushDataFailCause needs to see.

Covered by the Celeborn Cpp Integration Test workflow (Run Unittests of Celeborn Cpp).

@github-actions github-actions Bot added module:cpp correctness Correctness bugfix and removed correctness Correctness bugfix labels Aug 10, 2026
@yugan95 yugan95 changed the title [CELEBORN-2412] Propagate the worker error message through C++ MessageDispatcher failures [CELEBORN-2412] Propagate the worker error message through cpp MessageDispatcher failures Aug 11, 2026
@SteNicholas
SteNicholas requested a lite review from Copilot August 11, 2026 08:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the C++ client’s observability and failure classification by propagating worker-provided error strings through MessageDispatcher failures (instead of fulfilling promises with a blank std::exception()), and strengthens unit tests to assert on the surfaced message content.

Changes:

  • Wrap RPC_FAILURE and CHUNK_FETCH_FAILURE with std::runtime_error containing the worker’s error message (and chunk context for fetch failures).
  • Strengthen MessageDispatcherTest to assert that failure futures contain the expected error text (including a real StatusCode name substring for push failures).
  • Add a shared helper to extract exception messages from failed futures.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
cpp/celeborn/network/MessageDispatcher.cpp Preserve worker error text by setting promise exceptions with std::runtime_error messages for RPC and chunk-fetch failures.
cpp/celeborn/network/tests/MessageDispatcherTest.cpp Add helper + tighten failure tests to validate exception message contents.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cpp/celeborn/network/MessageDispatcher.cpp Outdated
Comment thread cpp/celeborn/network/MessageDispatcher.cpp

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@RexXiong RexXiong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

SteNicholas pushed a commit that referenced this pull request Aug 12, 2026
…eDispatcher failures

### What changes were proposed in this pull request?

`MessageDispatcher::read` decodes the error string a worker sends back on `RPC_FAILURE` / `CHUNK_FETCH_FAILURE` and then throws it away — both paths fulfil the pending promise with a blank `std::exception()`:

```cpp
holder.msgPromise.setException(folly::exception_wrapper(std::exception()));
```

This PR builds the exception from the message instead:

- `RPC_FAILURE` → `folly::make_exception_wrapper<std::runtime_error>(failure->errorMsg())`
- `CHUNK_FETCH_FAILURE` → same, using the already-formatted `errorMsg` that includes the `streamChunkSlice` context.

`MessageDispatcherTest` is strengthened accordingly: a shared `takeExceptionMessage()` helper is added, the two failure tests now assert on the message content rather than merely `hasException()`, and `sendPushDataAndReceiveFailure` uses a real StatusCode name (`PUSH_DATA_FAIL_PARTITION_NOT_FOUND`) so it exercises the classification path it is meant to protect.

### Why are the changes needed?

`ShuffleClientImpl::getPushDataFailCause` classifies a push failure by matching StatusCode names as substrings of the error message, and `PushDataCallback::onFailure` / `PushMergedDataCallback` feed it `exception->what()`. With a blank `std::exception`, `what()` is a fixed runtime string that matches none of the 13 candidate causes, so classification always falls through to `PUSH_DATA_FAIL_NON_CRITICAL_CAUSE`. Two consequences:

1. `excludeWorkerByCause` only acts on the connection-fail / timeout causes and hits its `default:` branch every time, so `celeborn.client.excludeWorker.enabled` never actually excludes a worker on the C++ push path.
2. When revive attempts are exhausted, `classifyPushFailure` raises `"PUSH_DATA_FAIL_NON_CRITICAL_CAUSE: <blank>"` — the worker's real reason (partition not found, worker excluded, replica write failure, …) is gone from the task-failure message, which makes these failures very hard to diagnose from client logs alone.

Chunk fetch failures lose the worker's diagnostics the same way.

This restores parity with the Java client, where `TransportResponseHandler` surfaces the worker's error text and `ShuffleClientImpl#getPushDataFailCause` can act on it.

### Does this PR resolve a correctness bug?

- [ ] Yes

Shuffle output is unaffected, so this is not a result-correctness bug. It is a failure-classification bug: worker exclusion never engages on the C++ push path, and the worker's reason is missing from the task-failure message.

### Does this PR introduce _any_ user-facing change?

- [ ] Yes

No config, API or behaviour change. Push and fetch failure messages now carry the worker's reason instead of a placeholder.

### How was this patch tested?

Unit tests in `cpp/celeborn/network/tests/MessageDispatcherTest.cpp`:

- `sendRpcRequestAndReceiveFailure` — asserts the RPC failure message reaches the caller.
- `sendPushDataAndReceiveFailure` — asserts a StatusCode name (`PUSH_DATA_FAIL_PARTITION_NOT_FOUND`) survives on the exception, which is exactly what `getPushDataFailCause` needs to see.

Covered by the `Celeborn Cpp Integration Test` workflow (`Run Unittests of Celeborn Cpp`).

Closes #3791 from yugan95/CELEBORN-2412.

Authored-by: Yu Gan <zhongheng.gy@alibaba-inc.com>
Signed-off-by: 子懿 <programgeek@163.com>
@SteNicholas

Copy link
Copy Markdown
Member

Merged to main(v1.0.0) and branch-0.7(v0.7.1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants