[vaidikgarg-bit]

Purpose of the PR

• close [Bug] In-memory union and intersection fail to combine equal scan rows #3396

RocksDBSession.BackendColumn compares name and value bytes in equals() but had no hashCode(), so it used identity hashing. A RocksDB scan allocates a new column per row, so the hash-based iterators in hg-store-core never matched byte-equal rows held in separate objects. With 100 keys read twice, the intersection with size = 2 returned 0 rows instead of 100 and the union below its 10,000-entry sort threshold returned 200. Those are the issue's numbers, and the new tests reproduce them on master.

Main Changes

• RocksDBSession.BackendColumn.hashCode() returns 31 * Arrays.hashCode(name) + Arrays.hashCode(value), the same arrays equals() compares (Bytes.equals is Arrays.equals, and both handle null arrays consistently).
• New FilterIteratorColumnEqualityTest in hg-store-test, registered in CoreSuiteTest. Every row is built from fresh arrays, as the issue asks. It covers:
  • equal hash codes for byte-equal columns, and distinct HashSet entries for the same name with different values;
  • IntersectionFilterIterator with size = 2;
  • UnionFilterIterator below its sort threshold;
  • MapLimitIterator.

Behaviour changes reviewers should know about

• MapLimitIterator (the LIMIT_DEDUP and PRECISE_DEDUP-with-limit paths in BusinessHandlerImpl) now drops byte-equal copies. Before, its ConcurrentHashSet kept all of them.
• The in-memory intersection now counts byte-equal rows together, as the sorted ReduceIterator path does. So it also has the limitation the class comments already describe: a row repeated inside one source counts toward size, so with size = 2 an element that appears twice in one source and not in the other is returned. This PR doesn't change that rule. Fixing it needs per-source counting, which I'd treat as a separate issue.
• No in-tree caller reaches these iterators today. StoreQueryParam fixes dedupOption to NONE, and the server's query.dedup_option setting is never passed to the store, so only a gRPC client that sets dedup_option gets here.

Out of scope

• Equality compares name and value, while the sorted union and intersection paths group by name only. With the in-tree INDEX_SCAN (lookupBack = true), scanIndexToElementId sets value to the owner id, which is the same in every source, so the two rules agree there. They can differ when lookup = false, where value is the raw index value. This PR keeps both rules and doesn't claim they're equivalent; the same-name/different-value test pins the equality rule.
• The SortShuffle path of the union (more than 10,000 entries) is [Bug] Precise union drops repeated keys in the SortShuffle path #3397.

Verifying these changes

• Trivial rework / code cleanup without any test coverage. (No Need)
• Already covered by existing tests, such as (please modify tests here).
• Need tests and can be verified as follows:
  • mvn test -pl hugegraph-store/hg-store-test -am -P store-core-test -Dtest=FilterIteratorColumnEqualityTest -Dsurefire.failIfNoSpecifiedTests=false: 4 of the 5 tests fail on master (hash codes differ, the intersection returns no keys, union and limit dedup return keys twice) and all 5 pass with this change. The iterator tests compare the exact set of returned keys.
  • mvn test -pl hugegraph-store/hg-store-test -am -P store-core-test: 139 tests, 0 failures (1 skipped, IpUtilTest.testConfiguredLinkLocalStays, an environment assumption).
  • mvn test -pl hugegraph-store/hg-store-test -am -P store-rocksdb-test: 19 tests, 0 failures.
  • mvn editorconfig:check -pl hugegraph-store/hg-store-test -am and mvn apache-rat:check pass.

Does this PR potentially affect the following parts?

• Dependencies (add/update license info & regenerate_known_dependencies.sh)
• Modify configurations
• The public API
• Other affects (typed here)
• Nope

Documentation Status

• Doc - TODO: required documentation is pending; complete it before merging.
• Doc - Done: documentation is included here or linked below.
• Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

Equality and hashing

Conceptual illustration of the hash-based paths; sorted name-only grouping, repeated rows within one source, and SortShuffle remain outside this change. Default dedup remains NONE. This diagram does not certify tests or CI.

[bitflicker64]

Blocking: no. Summary: The content hash fixes the in-memory intersection, the union below its sort threshold and limit dedup for unchanged rows, and the new tests catch the bug on the base. The PR's note that MapLimitIterator now drops byte-equal copies does not hold for id-only queries, because the null-property projection stage overwrites the emitted column that the iterator keeps as its set key; that mutation predates this PR, so it is not blocking. Evidence: read the full diff plus the BusinessHandlerImpl dedup paths, IntersectionFilterIterator, UnionFilterIterator, MapLimitIterator, QueryUtil.buildPlan, ProjectionStage and AggregativeQueryObserver.executePlainPipeline. At this head, mvn test -pl hugegraph-store/hg-store-test -am -P store-core-test passes (139 tests, 1 skipped); with the base RocksDBSession.java, 4 of the 5 new tests fail. The only CI run on this head (HugeGraph-Server CI) is waiting for maintainer approval.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.13%. Comparing base (c2ec570) to head (9f365e7).
⚠️ Report is 17 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3404      +/-   ##
============================================
+ Coverage     55.93%   56.13%   +0.20%     
- Complexity     2277     2386     +109     
============================================
  Files           774      775       +1     
  Lines         68534    69060     +526     
  Branches       9333     9507     +174     
============================================
+ Hits          38334    38770     +436     
- Misses        26634    26672      +38     
- Partials       3566     3618      +52     

Flag	Coverage Δ		*Carryforward flag
commons	96.37% <ø> (ø)		Carriedforward from c2ec570
hstore	36.08% <ø> (+0.27%)	⬆️
pd	49.27% <ø> (+0.09%)	⬆️	Carriedforward from c2ec570
server_hbase	41.96% <ø> (+0.07%)	⬆️	Carriedforward from c2ec570
server_memory	54.45% <ø> (+0.11%)	⬆️	Carriedforward from c2ec570
server_rocksdb	42.41% <ø> (+0.07%)	⬆️	Carriedforward from c2ec570
store	33.10% <100.00%> (+1.17%)	⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.