Skip to content

Look up workers after dial failures; vendor patched libp2p-stream - #217

Merged
mo4islona merged 3 commits into
mainfrom
vendor-libp2p-stream
Oct 15, 2026
Merged

mo4islona merged 3 commits into
mainfrom
vendor-libp2p-stream

Conversation

@kalabukdima

@kalabukdima kalabukdima commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #216: its commits are folded in, so the lookup fix reaches main together with the vendored stream crate.

Why

Actors that keep connections to all workers (portal, pings collector, logs collector, SQL client) can lose a worker for good after it is unreachable for a few minutes. A failed redial evicts the worker's cached address and nothing starts a new lookup, so every request to it fails with Dial error: no addresses for peer until unrelated DHT traffic supplies the address or the process restarts.

Changes

Look up a worker after a failed dial (from #216)

  • A Transport, WrongPeerId or NoAddresses dial failure to a registered worker schedules a Kademlia lookup after reconnect_cooldown. NoAddresses is included because a lookup returns only peers that answered: it cannot find a worker that is still offline, and the dial after it has no address. The cooldown paces these lookups.
  • Retries go on while the worker stays registered. Connection closes and dial failures go through one schedule_reconnect, and a worker waits for one reconnect at a time across the cooldown and the lookup queue, so it is not queued again while the lookup cap holds it back.
  • Tests that read or set PRIVATE_NETWORK hold a lock, since they share the process environment.

Vendor libp2p-stream instead of forking libp2p

  • The fork patched only protocols/stream. That crate is now copied into crates/transport/src/libp2p_stream (upstream aefbfbdc, MIT), and the rest of libp2p comes from crates.io 0.56.0, which drops the [patch.crates-io] for libp2p-identity.
  • The released crates differ from the fork's tree: core 0.43.2, swarm 0.47.1, quic 0.13.1 (fix for GHSA-5hq8-qhww-jm7q), kad 0.48.0, gossipsub 0.49.5. The gossipsub default subscription filter now caps topics at 2000; we use six.

Behaviour of the vendored crate, on top of the fork's two patches (unbounded dial channel, Control::accept_with_capacity)

  • Dial requests are de-duplicated per peer.
  • Every terminal dial error fails the queued open_stream calls, now including Aborted and LocalPeerId.
  • open_stream returns an error once its Behaviour is gone instead of hanging, and dropping the Behaviour cancels the requests still waiting for a dial.
  • A closed connection's sender is removed; upstream keeps one per closed connection.
  • A dial the swarm skips because the peer is already connected (DialPeerConditionFalse) hands its queued requests to that connection. The swarm builds a connection's handler before it reports the connection, and an open_stream in between queues behind a dial that is then skipped. With de-duplication that pending entry used to outlive the connection, and after the next disconnect requests to the peer waited on it instead of dialing.

Testing

  • cargo test --workspace --all-features --locked, cargo clippy --workspace --all-targets --all-features and nightly rustfmt pass on the branch rebased on main.
  • Each change has a regression test. The skipped-dial, dropped-behaviour, one-reconnect-per-worker and still-offline-worker tests hang or fail without their fixes.

🤖 Generated with Claude Code

@mo4islona
mo4islona removed this pull request from stack #218 October 15, 2026 09:15
@mo4islona mo4islona changed the title Vendor patched libp2p-stream instead of forking libp2p Look up workers after dial failures; vendor patched libp2p-stream Oct 15, 2026
@mo4islona
mo4islona changed the base branch from address-cache to main October 15, 2026 09:15
@mo4islona
mo4islona force-pushed the vendor-libp2p-stream branch 2 times, most recently from 21c35c1 to 3231c67 Compare October 15, 2026 10:43
kalabukdima and others added 3 commits October 15, 2026 14:49
The reconnect path fires only on ConnectionClosed, and at that point the
address cache still holds the dead address, so it is dialed instead of
looked up. That dial fails and evicts the entry, and nothing re-arms: a
failed dial establishes no connection, so no close event follows, and the
whitelist sweep is suppressed while the registered set is unchanged. The
worker stayed undialable until unrelated DHT traffic happened to supply
an address.

NoAddresses retries too, after the same cooldown. A lookup returns only
peers that answered, so it cannot find a worker that is still offline,
and the dial after it has no address; stopping there left the worker
unreachable after it came back.

A worker waits for one reconnect at a time, across the cooldown and the
lookup queue, so failures while the lookup cap is reached no longer queue
it again.

Tests that read or set PRIVATE_NETWORK now hold a lock: addr_is_reachable
reads it on every call and a binary's tests run as threads in one process.

Part of NET-384

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The kalabukdima/rust-libp2p fork patched nothing but protocols/stream: an
unbounded dial channel and Control::accept_with_capacity, both fixing streams
that upstream silently drops under concurrency. Carrying them as a fork of the
whole workspace also forced a [patch.crates-io] for libp2p-identity on every
downstream crate. Copy the one patched crate into the transport crate instead
and take the rest of libp2p from crates.io at the version the fork is based on.

- Add crates/transport/src/libp2p_stream: upstream rust-libp2p aefbfbdc (MIT)
  plus the two patches, and upstream's tests for them; unchanged apart from
  module paths, rand::rng() and this repo's rustfmt
- Point libp2p and friends at crates.io 0.56.0, drop the identity patch
- Drop the now-unreachable OpenStreamError arm and derive ClientBehaviour's
  Default: #[non_exhaustive] and clippy treat the types as local now

The fork branched 26 commits past the v0.56.0 release, so the released crates
are not identical to its tree: libp2p-core 0.43.1 -> 0.43.2, libp2p-swarm
0.47.0 -> 0.47.1, libp2p-quic 0.13.0 -> 0.13.1, and libp2p-kad and
libp2p-gossipsub back to the released 0.48.0 and 0.49.5. That picks up the
fixes for GHSA-5hq8-qhww-jm7q (quic panicking on certificate validation
failure) and for three gossipsub advisories published after the fork's base, at
the cost of the unreleased gossipsub 0.50.0 changes the fork carried. The one
behavioural delta that reaches us: pubsub builds gossipsub with the default
subscription filter, which is now capped at 2000 topics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dial errors beyond the four upstream handles, a Behaviour dropped with dials
pending, and a request that raced a new connection each left open_stream
waiting on a channel nothing would drain; the last became reachable once
dials were de-duplicated per peer. Closed connections also kept their senders
for the life of the process.

Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mo4islona
mo4islona force-pushed the vendor-libp2p-stream branch from 3231c67 to 76a6dff Compare October 15, 2026 11:49
@mo4islona
mo4islona merged commit e1460fd into main Oct 15, 2026
3 checks passed
@mo4islona
mo4islona deleted the vendor-libp2p-stream branch October 15, 2026 12:09
mo4islona added a commit to subsquid/sqd-portal that referenced this pull request Oct 15, 2026
f3564b4 never looks a worker up again after a failed redial evicts its cached
address, so requests to that worker fail with "no addresses for peer" until
unrelated DHT traffic or a restart brings the address back. e1460fd has
subsquid/sqd-network#217 and vendors libp2p-stream, so the libp2p-identity
patch for the fork goes away. sqd-assignments stays at f3564b4: the transport
does not use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mo4islona added a commit to subsquid/sqd-portal that referenced this pull request Oct 15, 2026
f3564b4 never looks a worker up again after a failed redial evicts its cached
address, so requests to that worker fail with "no addresses for peer" until
unrelated DHT traffic or a restart brings the address back. e1460fd has
subsquid/sqd-network#217 and vendors libp2p-stream, so the libp2p-identity
patch for the fork goes away. The tests and the harness now pass typed chunk
ids to the assignment builders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mo4islona added a commit to subsquid/sqd-portal that referenced this pull request Oct 15, 2026
f3564b4 never looks a worker up again after a failed redial evicts its cached
address, so requests to that worker fail with "no addresses for peer" until
unrelated DHT traffic or a restart brings the address back. e1460fd has
subsquid/sqd-network#217 and vendors libp2p-stream, so the libp2p-identity
patch for the fork goes away. The tests and the harness now pass typed chunk
ids to the assignment builders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants