[#815] Wait for the topology to settle before asserting RS side DS counts - #822
Merged
vharseko merged 1 commit intoAug 4, 2026
Conversation
…erting RS side DS counts ReplicationServerLoadBalancingTest sampled ReplicationServerDomain.getConnectedDSs() right after createReplicationDomain() returned. That return only means the directory server side of the handshake is done: the RS sends the TopologyMsg which ends it (DataServerHandler.sendTopoToRemoteDS()) before it registers the DS handler in its domain (ReplicationServerDomain.register()), so the RS side counter can still be one DS short. On a loaded CI runner this failed a whole job on a single assertion. Replace those one-shot samples with the checkForCorrectNumbersOfConnectedDSs() polling helper the rest of the class already uses, and drop the Thread.sleep(2000) of testSpreadLoad, which was both racy and too short for a rebalancing (2 monitoring publisher periods). The expected numbers themselves are unchanged: the load balancing input is read under the domain lock, so the layouts the test expects are the ones the algorithm produces - only the reading was racy. For testNoYoyo1/2/3 the RS to check is only known at runtime, hence the onlyCheckRS() helper building a layout where the other RSs are ignored.
maximthomas
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #815
ReplicationServerLoadBalancingTestread the replication server side DS counter right aftercreateReplicationDomain()returned, which only tells that the directory server side of thehandshake is over. The RS sends the
TopologyMsgthat ends the handshake(
DataServerHandler.sendTopoToRemoteDS(),:430) before it registers the DS handler in itsdomain (
ReplicationServerDomain.register(),:449), sogetConnectedDSs()can still be oneDS short. That is how a single assertion failed a whole 1 h 40 min CI job:
What this changes
Every remaining one-shot sample of the counter is replaced with the polling helper the rest of
the class already uses,
checkForCorrectNumbersOfConnectedDSs()(30 s timeout):testSpreadLoad, after the 20 DSs{{2, 4, 6, 8}}, replacingThread.sleep(2000)+ 4 assertionstestFailoversAndWeightChanges, after DS7…DS12{{4, 4, 4}}— a 3 long array, RS4 does not exist yet{{4,3,10,3},{3,4,10,3},{3,3,10,4}}testNoYoyo1/2/3, right after the lastcreateReplicationDomain()The restart of the 2 stopped DSs was not listed in the issue but has the same pattern, so it is
covered too. The
testNoYoyo*cases pick their RS at runtime, hence the smallonlyCheckRS()helper which builds a layout where every other RS is
-1and which sizes the array fromgetNbRSs(testCase), so RSs that were never started are not touched.The expected numbers themselves are unchanged. They are not affected by the race: the counter
that drives the weight based election is read under the domain lock
(
DataServerHandler.sendStartToRemote()at:496, called afterlockDomainNoTimeout()), andregister()happens under that same lock, so a DS starting its handshake always sees anup to date count. Only the lock-free sampling done by the test was racy.
Dropping the
Thread.sleep(2000)is a fix of its own: a rebalancing needs two monitoringpublisher periods (3 s each by default in
ReplServerFakeConfiguration), so 2 s could never beenough had the topology needed to settle.
Testing
Note
Closing the window on the production side (registering the handler before sending the
TopologyMsg) is deliberately not done here: the abort path of the handshake does notunregister the handler today, so that move would leak dead handlers. Filed separately as #821.