UCT/UD: Remove stale private UD connections - v1.23.x

[iyastreb]

What?

Backport #11822 to 1.23

Release a UD private endpoint once its peer is gone, instead of keeping it forever.

Why?

On first UCP wireup request:
A private endpoint is created internally on CREQ, has no UCP endpoint above it => so keepalive never checks it => and UD never signals a closed remote endpoint.
When remote peer connection dies => private UD connection stays forever pointing at a dead ep_id

On UCP wireup reconnect request => uct_ep_create() matches that stale endpoint by addr+conn_sn => reuses it as is => wireup reply goes to the dead ep_id => connection never completes

How?

Probe an idle private endpoint from the UD timer with the existing uct_ud_ep_check(), skipping it while the peer keeps sending. The threshold is the upper layer's keepalive interval. An alive peer acks; otherwise peer_timeout fires and the endpoint leaves connection matching immediately.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/uct/ib/test_ud.cc

minor — find_private_ep: elem is declared void * and cast in the loop body; ucs_ptr_array_for_each can take the typed pointer directly (as uct_ud_iface_flush does with ucs_ptr_array_for_each(ep, i, &iface->eps)), which would drop the local cast.

Note: could not fetch the live PR discussion or CI status (no GitHub access in this environment), and the reference checkout at /tmp/pr-agent-wt-11899 contained the base tree rather than the PR head, so review was against the provided diff plus base-code context. Residual gap: this is real-HW UD behavior, so confirm the two new gtests actually run (not skipped for lack of PUT_SHORT/IB device) in CI before merge.

[tvegas1]

for my reference: UD connection matching is only opportunistic and there can be valid case with one-sided uct_ep_create for non-p2p lane (one sided loadRemoteMD->ucp_ep_create with only non-p2p or ep check response creating remote private or simply early disconnect). eviction ucp ~ka interval then post ack_req, re-tx timer backoff/tick until peer_timeout.