Take a sorted page's order from the index instead of every document

`find(ALL, orderBy("createdAt", Descending).limit(20))` asked for 20 rows and
cost what draining the whole collection costs. `SortedDocumentStream` collects
the entire result set before `BoundedStream` gets to drop 99% of it, and the cost
is the decode, not the comparison, so it scales with document *size* as much as
count. An index on the sort field bought nothing — the index was only ever used
to filter — and page 50 cost exactly what page 1 cost, because the work finished
before the skip applied.

An index on the sort field already stores that field's value for every document.
When the query has no filter, one sort field, a limit, and a simple unique or
non-unique index on exactly that field, the keys now come from the index
(`NitriteIndex.readSortKeys`) and `IndexedStream` fetches only the rows the page
returns. On MVStore over 2000 rows each carrying a 150-element list, a `limit(20)`
page went from ~125 ms to ~1 ms.

The index replaces the documents only when it is a faithful stand-in for them: a
multi-valued field is indexed once per element, so a duplicate-id check and an
entry-count check catch it and fall back to the blocking sort. Ordering is decided
by the same comparator either way — extracted as `DocumentSorter.compareValues` so
the two cannot drift — over keys read from the index rather than from documents,
which keeps nulls first and resolves ties in document-id order exactly as before.

Where documents are small the index walk replaces a decode that was nearly free,
so a sorted page over lean rows can cost a few hundred microseconds more than it
did. The trade is deliberate: that loss is bounded and sub-millisecond, the win
grows without bound with document size.

New API is additive. `NitriteIndex.readSortKeys(long)` and
`NitriteIndexer.readSortKeys(IndexDescriptor, NitriteConfig, long)` are `default`
methods returning `null`, so an existing indexer plugin is unaffected and simply
never takes the new path.

`CollectionSortedFindTest` pins every sorted query against the same query on an
unindexed collection — ascending, descending, deep pages, ties, strings, missing
fields, unique indexes, and the multi-valued fallback — so a divergence in
ordering fails outright. `CollectionSortedFindCostTest` runs on MVStore, where
documents are actually serialized, and compares a sorted page over lean documents
against one over fat documents at the same row count: only the decode differs
between them, so it is stable under load in a way a wall-clock threshold is not
(it reports 94x without this change and ~1x with it).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: anidotnet

CHANGELOG.md

@@ -1,3 +1,14 @@
+## Release 5.1.0 - Aug 14, 2026
+
+### Improvements
+
+- A sorted, limited `find` no longer fetches the whole collection when the sort field is indexed
+  - `find(ALL, orderBy("createdAt", Descending).limit(20))` asked for 20 rows and cost what draining every stored document costs. `SortedDocumentStream` collects the entire result set before `BoundedStream` gets to drop 99% of it — and the cost is the decode, not the comparison, so it scales with document *size* as well as count. An index on the sort field bought nothing: the index was only ever used to *filter*, never to order, and page 50 cost exactly what page 1 cost because the work finished before the skip applied.
+  - When the query has no filter, one sort field, a limit, and a simple unique or non-unique index on exactly that field, the sort keys are now read from that index — which already stores them — and only the documents actually returned are fetched. Measured over 2000 rows each carrying a 150-element list, a `limit(20)` page went from ~125 ms to ~1 ms, and stopped growing with the collection.
+  - The index is used only when it holds exactly one entry per stored document. A multi-valued field is indexed once per element, which is detected by a duplicate-id check and an entry-count check, and falls back to the blocking sort. Ordering — including where nulls sort and how ties break — is identical either way: the same comparator runs over keys taken from the index instead of from the documents.
+  - Where documents are small and cheap to read, the index walk replaces a decode that was nearly free, so a sorted page over lean rows can cost a few hundred microseconds more than before. The trade is deliberate: the loss is bounded and sub-millisecond, the win grows without bound with document size.
+  - New API, all additive: `FindPlan.getSortIndexDescriptor()`, `NitriteIndex.readSortKeys(long)` and `NitriteIndexer.readSortKeys(IndexDescriptor, NitriteConfig, long)` (both `default`-implemented to return `null`, so existing indexer plugins are unaffected), and `DocumentSorter.compareValues(Object, Object, Collator)`.
+
 ## Release 5.0.0 - Aug 7, 2026
 
 ### Breaking Changes

nitrite-bom/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-bom</artifactId>

nitrite-jackson-mapper/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-jackson-mapper</artifactId>

nitrite-mvstore-adapter/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-mvstore-adapter</artifactId>

nitrite-mvstore-adapter/src/test/java/org/dizitart/no2/integration/collection/CollectionSortedFindCostTest.java

@@ -0,0 +1,125 @@
+/*
+ * Copyright (c) 2017-2021 Nitrite author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.dizitart.no2.integration.collection;
+
+import org.dizitart.no2.Nitrite;
+import org.dizitart.no2.collection.Document;
+import org.dizitart.no2.collection.FindOptions;
+import org.dizitart.no2.collection.NitriteCollection;
+import org.dizitart.no2.common.SortOrder;
+import org.dizitart.no2.index.IndexOptions;
+import org.dizitart.no2.index.IndexType;
+import org.dizitart.no2.mvstore.MVStoreModule;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.dizitart.no2.filters.Filter.ALL;
+import static org.dizitart.no2.integration.TestUtil.deleteDb;
+import static org.dizitart.no2.integration.TestUtil.getRandomTempDbFile;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * The cost half of {@link CollectionSortedFindTest}, which needs a store that actually
+ * serializes documents.
+ * <p>
+ * A blocking sort deserializes every stored document to read one field, so
+ * {@code orderBy(indexed).limit(20)} cost what draining the whole collection cost - and the
+ * gap grows with document size, not just document count. Taking the sort keys from the index
+ * removes the decode: over 2000 rows carrying a 150-element array the sorted page went from
+ * ~37 ms to ~1 ms.
+ *
+ * @author Anindya Chatterjee
+ */
+public class CollectionSortedFindCostTest {
+    private static final int ROWS = 2000;
+
+    private final String fileName = getRandomTempDbFile();
+    private Nitrite db;
+
+    @Before
+    public void setUp() {
+        db = Nitrite.builder()
+            .loadModule(MVStoreModule.withConfig().filePath(fileName).build())
+            .openOrCreate();
+    }
+
+    @After
+    public void tearDown() {
+        if (db != null && !db.isClosed()) {
+            db.close();
+        }
+        deleteDb(fileName);
+    }
+
+    /**
+     * The same query, the same row count, the same index - only the size of the documents
+     * differs. A sorted page that decodes every row pays for the payload of every row, so
+     * the fat collection costs many times the lean one. A sorted page that decodes only the
+     * rows it returns costs the same either way.
+     * <p>
+     * Deliberately not "sorted page vs. full drain": both halves here are measured back to
+     * back under whatever load the machine is under, and the only variable between them is
+     * the thing that was broken.
+     */
+    @Test
+    public void testSortedPageCostDoesNotFollowDocumentSize() {
+        double lean = sortedPageCost("lean", 0);
+        double fat = sortedPageCost("fat", 150);
+
+        assertTrue("a sorted page over fat documents took " + fat + "ms against " + lean
+                + "ms over lean ones, same row count - it is still decoding rows it discards",
+            fat < lean * 4);
+    }
+
+    private double sortedPageCost(String name, int payloadSize) {
+        NitriteCollection collection = db.getCollection(name);
+        collection.createIndex(IndexOptions.indexOptions(IndexType.NON_UNIQUE), "seq");
+
+        for (int i = 0; i < ROWS; i++) {
+            Document document = Document.createDocument("seq", i);
+            if (payloadSize > 0) {
+                List<Document> payload = new ArrayList<>(payloadSize);
+                for (int w = 0; w < payloadSize; w++) {
+                    payload.add(Document.createDocument("text", "word" + w).put("start", w * 300));
+                }
+                document.put("payload", payload);
+            }
+            collection.insert(document);
+        }
+
+        FindOptions page = FindOptions.orderBy("seq", SortOrder.Descending).limit(20);
+        return timeOf(() -> {
+            for (Document ignored : collection.find(ALL, page)) {
+                // force the fetch
+            }
+        });
+    }
+
+    private static double timeOf(Runnable task) {
+        task.run(); // warm
+        long start = System.nanoTime();
+        for (int i = 0; i < 3; i++) {
+            task.run();
+        }
+        return (System.nanoTime() - start) / 3e6;
+    }
+}

nitrite-native-tests/pom.xml

@@ -7,7 +7,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-native-tests</artifactId>

nitrite-rocksdb-adapter/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-rocksdb-adapter</artifactId>

nitrite-spatial/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-spatial</artifactId>

nitrite-support/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite-support</artifactId>

nitrite/pom.xml

@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.dizitart</groupId>
         <artifactId>nitrite-java</artifactId>
-        <version>5.0.0</version>
+        <version>5.1.0</version>
     </parent>
 
     <artifactId>nitrite</artifactId>