Skip to content

[#815] Wait for the topology to settle before asserting RS side DS counts - #822

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/815-lb-test-topology-wait
Aug 4, 2026
Merged

[#815] Wait for the topology to settle before asserting RS side DS counts#822
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/815-lb-test-topology-wait

Conversation

@vharseko

@vharseko vharseko commented Aug 3, 2026

Copy link
Copy Markdown
Member

Fixes #815

ReplicationServerLoadBalancingTest read the replication server side DS counter right after
createReplicationDomain() returned, which only tells that the directory server side of the
handshake is over. The RS sends the TopologyMsg that ends the handshake
(DataServerHandler.sendTopoToRemoteDS(), :430) before it registers the DS handler in its
domain (ReplicationServerDomain.register(), :449), so getConnectedDSs() can still be one
DS short. That is how a single assertion failed a whole 1 h 40 min CI job:

[ERROR] ReplicationServerLoadBalancingTest.testFailoversAndWeightChanges:512
        Wrong expected number of DSs connected to RS3 expected [4] but found [3]

What this changes

Every remaining one-shot sample of the counter is replaced with the polling helper the rest of
the class already uses, checkForCorrectNumbersOfConnectedDSs() (30 s timeout):

Site Expected layout
testSpreadLoad, after the 20 DSs {{2, 4, 6, 8}}, replacing Thread.sleep(2000) + 4 assertions
testFailoversAndWeightChanges, after DS7…DS12 {{4, 4, 4}} — a 3 long array, RS4 does not exist yet
same test, after DS13…DS20 and after the 2 stopped DSs are restarted {{4,3,10,3},{3,4,10,3},{3,3,10,4}}
testNoYoyo1/2/3, right after the last createReplicationDomain() only the RS the DS elected is examined, the others are ignored

The restart of the 2 stopped DSs was not listed in the issue but has the same pattern, so it is
covered too. The testNoYoyo* cases pick their RS at runtime, hence the small onlyCheckRS()
helper which builds a layout where every other RS is -1 and which sizes the array from
getNbRSs(testCase), so RSs that were never started are not touched.

The expected numbers themselves are unchanged. They are not affected by the race: the counter
that drives the weight based election is read under the domain lock
(DataServerHandler.sendStartToRemote() at :496, called after lockDomainNoTimeout()), and
register() happens under that same lock, so a DS starting its handshake always sees an
up to date count. Only the lock-free sampling done by the test was racy.

Dropping the Thread.sleep(2000) is a fix of its own: a rebalancing needs two monitoring
publisher periods (3 s each by default in ReplServerFakeConfiguration), so 2 s could never be
enough had the topology needed to settle.

Testing

mvn -o -pl opendj-server-legacy -P precommit verify -Dit.test=ReplicationServerLoadBalancingTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 172.6 s

Note

Closing the window on the production side (registering the handler before sending the
TopologyMsg) is deliberately not done here: the abort path of the handshake does not
unregister the handler today, so that move would leak dead handlers. Filed separately as #821.

…erting RS side DS counts

ReplicationServerLoadBalancingTest sampled ReplicationServerDomain.getConnectedDSs()
right after createReplicationDomain() returned. That return only means the directory
server side of the handshake is done: the RS sends the TopologyMsg which ends it
(DataServerHandler.sendTopoToRemoteDS()) before it registers the DS handler in its
domain (ReplicationServerDomain.register()), so the RS side counter can still be one
DS short. On a loaded CI runner this failed a whole job on a single assertion.

Replace those one-shot samples with the checkForCorrectNumbersOfConnectedDSs()
polling helper the rest of the class already uses, and drop the Thread.sleep(2000)
of testSpreadLoad, which was both racy and too short for a rebalancing (2 monitoring
publisher periods). The expected numbers themselves are unchanged: the load balancing
input is read under the domain lock, so the layouts the test expects are the ones the
algorithm produces - only the reading was racy.

For testNoYoyo1/2/3 the RS to check is only known at runtime, hence the onlyCheckRS()
helper building a layout where the other RSs are ignored.
@vharseko
vharseko requested a review from maximthomas August 3, 2026 11:24
@vharseko vharseko added tests Test suites: fixing, enabling, un-disabling replication concurrency Thread-safety / race-condition bugs CI labels Aug 3, 2026
@vharseko
vharseko merged commit 9cc45ab into OpenIdentityPlatform:master Aug 4, 2026
17 checks passed
@vharseko
vharseko deleted the issues/815-lb-test-topology-wait branch August 4, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI concurrency Thread-safety / race-condition bugs replication tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReplicationServerLoadBalancingTest asserts the RS side DS count before the RS registers the DS

2 participants