Repository navigation
Conversation
A collection's secondary_indexes field was an AddressMap from index name to a BSON blob holding the definition and the index map root as a hex string. Chunk walks cannot see inside a blob, so dumboGC deleted every index map, and push, clone and backup left them out (#190). secondary_indexes is now a prolly map from index name to a value tuple of the definition (BSON, adaptive bytes) and the map root (an address field), which GC and the puller follow through ValueAddressOffsets. The per-index entry blob is gone. The definition keeps the user's key order and stores the partial filter and collation as embedded documents instead of hex-encoded BSON. Tables in the old layout are refused rather than read; DumboDB is pre-beta and keeps no backward compatibility. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Return an error, not a panic, when an index map root is not a hash. - Key the decoded-definition memo by definition alone, so indexed writes, which change only map roots, no longer add an entry each. - Refuse at startup a data dir whose admin database holds the old layout, instead of failing every login as "Authentication failed". - Compare partial filters regardless of key order in merge, as before. - Cover out-of-line definitions, multi-node secondary_indexes maps and backup/restore in the reachability tests. - Update the storage reference and stale comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c5f2be8 made indexSpecEqual sort partial filter keys at every level and called that main's behavior; main compared them in key order. Documents keep their key order in dumbo, so a reordered filter is a different definition. Restore the ordered comparison and test it at the top level and inside an embedded document. Drop a stale tracker reference from a merge comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DumboDB keeps documents in canonical form, keys sorted at every level, so the same logical state has the same bytes and hashes. Index definitions broke that: encodeIndexDefinition kept the user's key order for the partial filter and collation, so the same definition typed in a different order produced a different table hash, and after a merge the stored order depended on merge direction. da26211 made it worse by comparing partial filters in key order in merge, on the false premise that documents keep their key order. Two branches creating the same index with reordered filter fields then failed to merge with a spurious redefinition conflict. Encode the whole definition with sorted keys and compare partial filters in canonical form. Compound index key order, an array, is unaffected. listIndexes now echoes partial filters with sorted keys, as documents are returned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clustered collection's RecordId is KeyString(_id) without TypeBits, and find().showRecordId() returns it. TestKeyStringMatchesMongod inserts a corpus of edge cases plus random values of every _id type into a clustered collection on MONGOD_BIN and requires EncodeValue to produce the same bytes, for stored values and for values mongod treats as equal to a stored one. It skips without MONGOD_BIN. It fails today: 2085 of 3118 encodings differ and 796 stored values encode out of mongod's order. The clean-room encoder replaces it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EncodeValue now produces mongod's KeyString V1 bytes without TypeBits, the bytes a clustered collection uses as the RecordId for an _id, derived from mongod's observable output (fixes #206): - mongod's type bytes, and its numeric layout: integer parts shifted for a fraction flag, binary fractions, small and large magnitudes, so negative fractions in (-0.5, 0) no longer collide and doubles past the int64 range are exact; - Decimal128 as the equal double or integer, else the truncated double with flags and a continuation, or a 16-byte form beyond the double range; - documents field by field and arrays element by element, instead of one marker byte for every subdocument; - length-prefixed BinData, ordered by length, then subtype, then bytes. Every encoding delimits itself, so SplitKeyValues splits exactly one way. EncodeValue orders embedded document fields as storage does, sorted except within _id; EncodeID keeps an _id's order, as mongod does. With no value collapsing, the sort walk and distinct scan no longer regroup subdocuments, and only types without an encoding are lossy. TestKeyStringMatchesMongod passes against mongod 8.0.28 for 30k random values. Known deviation: an inexact Decimal128 below 2^-256 matches mongod in leading bytes and order but not in its trailing bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NumberSum folded doubles into a plain float64 before the first
Decimal128 arrived and converted only that rounded total, so
0.1 + 0.2 + NumberDecimal("0.3") gave ...0444 when the doubles came
first and ...0166 when the decimal did. MongoDB carries a double-double
sum into the decimal; NumberSum now tracks the float64 sum's rounding
error with TwoSum and converts it too. The double-only result is
unchanged.
The decimal total itself collapsed to zero when its exact value needed
more than 34 digits (1E20 + 1E-20); it now rounds as Decimal128
arithmetic does.
Natural order following _id order exposed the first bug in parity's
AggGroupNum_DecimalFloatMixSum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macneale4
marked this pull request as ready for review
November 6, 2026 23:03
Collaborator
Author
|
Security Review Feedback: https://claude.ai/artifact/AheFrTmWkqZQFPvQ7aaLAA |
This was referenced Nov 7, 2026
Collaborator
Author
|
Code Review Feedback: https://claude.ai/artifact/QGNENCGkH3L4rdZZvXKa3k |
The primary map was keyed by hashID(_id), 20 bytes of SHA-512 over a normalized _id, so nothing could serve an _id range or sort and natural order was hash order (#207). It is now keyed by index.EncodeID(_id), the bytes mongod uses as a clustered collection's RecordId: point lookups are an exact seek, the map is in _id order, and equal _ids (1, 1.0, NumberLong(1)) are one key. Secondary index entries end with that key instead of the hash. Keys vary in length, so entries split from the front (index.SplitEntry): values delimit themselves and the 0x04 end marker sorts below every value type. The catalog keys collections by name the same way, document locks use the key's hash as their identity, and RecordIDs hash the whole key, since keys share prefixes. The table schema declares _id as varbinary. hashID and its normalizing helpers are gone. Collated string _ids still rely on the uniqueness scan; the key is the uncollated _id. Pre-beta: existing data directories must be recreated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The primary key column now holds KeyString(_id), a varbinary(16383), not the 20-byte hash it replaced, so the storage test's expected schema changes with it (review N1, approved by the owner). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain's executionStats reported zeros for update, delete, findAndModify, count and distinct, and for find copied docs examined into keys examined (#145). Explain now runs the command's own read under per-operation counters and reports what it read. - backends.ExaminedCounts counts the keys and documents read under a context; every serverStatus count site also adds to it. - count and distinct run their handler's execution, now execCount and execDistinct; writes run the query and filter the write would, stopping at the first match for a single-document write. - executionStages carry the measured counts down a single-input chain: keysExamined on index scans, docsExamined and matches on FETCH and COLLSCAN. - A multi-document delete explains as BATCHED_DELETE, as on MongoDB. - When a sort walk wins its trial, the losing filter plan's keys are left out of the operation's counts, as MongoDB leaves out a losing plan's trial work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain chose its index with its own planner, so it named indexes the runtime declined and missed plans the runtime ran: an _id point read was a COLLSCAN (#143), a range over half the collection an IXSCAN (#139), and a compound or nested $or union a COLLSCAN (#132). - query, behind Query, returns the stage tree of the scan that served the query: the index stream, sort walk, key sort, union branches, _id point read or collection scan. Explain runs it and wraps the scan in the SORT, SKIP, LIMIT, projection and write stages; count and distinct keep their own shapes. - An _id equality alone explains as EXPRESS_IXSCAN, EXPRESS_UPDATE or EXPRESS_DELETE; with other fields as FETCH of an _id_ IXSCAN. A lone $or union is under SUBPLAN, and findAndModify under UPDATE or DELETE. - The plan cache state is read before planning fills it. Making explain honest exposed runtime plans that differed from MongoDB's: - An _id equality beside an indexed field read the other index; the primary-key read is now tried first when there is no hint. - A hinted index the filter does not bound fell back to a collection scan; it is now read whole, as on MongoDB. - A range of at most narrowRangeEntries entries is always read through the index: in a small collection the half-collection cap declined ranges too cheap to matter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Below an OR stage executionStages reported zero (#132). A union reads each branch whole while planning, so Explain returns each branch's keys and matches as ExplainResult.UnionBranches, and executionStages give every branch IXSCAN its own keysExamined and nReturned and the OR the documents it fetched, as on MongoDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An aggregate's explain showed only the local collection's plan, though the $lookup read its foreign collection through an index (#144). - An equality $lookup directly after the stages pushed into the query explains as an EQ_LOOKUP stage over the query plan, as MongoDB 8.0's SBE engine reports it: IndexedLoopJoin with the index, else HashJoin or, past 10,000 foreign documents, NestedLoopJoin. - Any other stage moves the explain into a stages array, the query under $cursor; a $lookup there lists, at executionStats verbosity, the foreign indexes it and its nested $lookups read. - The index is the one the runtime probes: stages.LookupJoinField names the joined field, and the fetcher's indexOn the index IndexedField finds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
find({bucket: {$in: [1, 3]}}).sort({k: 1}).limit(5) on {bucket: 1, k: 1}
read both ranges whole and sorted them (#146). Each $in value's range is
already in k order, so MongoDB merges them (SORT_MERGE) and stops at the
limit, reading about 6 keys.
- tryStreamMergedRanges serves an $in of scalars on the leading field,
sorted either way on the second key of an ascending, non-multikey index,
when the leading field is the only one filtered: one range iterator per
value, merged by the second key, fetching lazily. The merge advances a
range only when the next id is asked for, so a limit reads no further
than MongoDB does.
- RangeIDIter.NextEntry also returns the entry's encoded index values.
- Explain shows LIMIT > FETCH > SORT_MERGE > IXSCAN per value.
- ExaminedCounts records each branch's keys and entries under an OR or
SORT_MERGE as the query runs, replacing the planning-time union counts,
so executionStages report every branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An index on "a.b" over {a: [{b: 5}, {c: 1}]} held only 5, where mongod
also holds null for the element without b, so {"a.b": null} could not be
served from the index and scanned the collection (#181).
- A dotted path without positional segments now takes its keys from
commonpath.IndexKeyValues, MongoDB's rules: null for a missing field, an
array element it does not reach, or a scalar or empty array before the
path ends; undefined for an empty array at its end. Paths like a.0.b keep
FindValues.
- With those keys, null equality on such a dotted field is served from a
non-sparse index, multikey or not.
Indexes on dotted paths built before this change lack the new null keys
and must be rebuilt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
find({k: {$gte: 5000, $lt: 5050}}, {k: 1, _id: 0}) on {k: 1} fetched
every document, where mongod reads only the index (#142). DumboDB could
encode values into index keys but not decode them.
- Each entry's value tuple now holds one type tag per indexed value, as
MongoDB's TypeBits: int32, int64, double, -0.0, decodable from the key,
or fetch for what the key cannot restore (subdocument and array markers,
collation sort keys, lossy values, and doubles whose encoding does not
give back the same bits).
- index.DecodeValue restores a value from its encoding and tag;
RangeIDIter.NextEntry returns the tags.
- Index streams yield entries. When every field a read needs is a
top-level key of a non-multikey index, the range, sorted-range and
merged-range iterators build each document from its entry and fetch only
when an entry cannot restore a field, or holds null, which cannot tell a
null from a missing field.
- A find whose projection drops _id no longer reads _id unless it filters
or sorts on it, and explain's execution reads the same fields.
Entries written before this change hold no tags and keep fetching.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
distinct("grp") read every index key and fetched a document per value,
and a filtered distinct did the same over its range (#141). MongoDB's
DISTINCT_SCAN reads one key per value and no document.
- scanDistinctFromIndex restores each value from its entry's encoding and
type tag, checks a filter against it, and seeks past the rest of the
value's entries. A subdocument or array marker group still fetches every
member, and an entry that cannot restore its value fetches one document.
- index.SeekKeyRange starts a key range only when an entry lies at or after
its start: a prolly iterator started past the last entry reads beyond the
end of the map, returning an empty key or panicking. RangeIDIter uses it
too, and both treat an empty key as the end of the range.
- index.DecodeEntry splits a raw entry; index.SkipValues names the key past
a value's entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
$match {grp: {$gte: 0}} then $group by $grp, with an index on grp, scanned
the collection: the range holds the whole collection, past the half where
one point lookup per entry loses to a sequential scan (#139). A read that
needs only the index's keys looks nothing up, so it now always streams
the index, as mongod answers it without fetching a document.
Explain plans and runs with the fields the explained find or pipeline
reads, ExplainParams.Fields, so its plan and counts are those of the
covered read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
$geoNear and find with $near or $nearSphere computed every document's distance and sorted them in memory, so a large result failed at the 100MB sort limit where mongod streams it from its geo index (#124), and a limit or $maxDistance still read the whole collection. - A 2dsphere or 2d index now keys each point by its cell in a 2^24 grid per axis (longitude and latitude, or both coordinates in [-180, 180]) as a Morton code, so every quadtree cell is a range of keys. A non-point geometry or a point outside the grid shares one reserved key. - backends.GeoNearScanner reads nearest first. The dolt backend reads rings of doubling radius: it fetches every document in the cells covering a ring's bounds (split at the antimeridian, widened at the poles) and returns, by distance, those within the ring, since nothing unread is nearer. The caller supplies the distance, so the order is the one the handler measures. A $maxDistance ends the read. - A pipeline starting with $geoNear, and a find with $near or $nearSphere, read through it when a geo index serves the field, and stream instead of sorting. Without one they still sort in memory. - A geo index's keys are cells, so range reads and covered reads no longer use it. Geo indexes built before this change hold the old keys and must be rebuilt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-ups to the query planning work:
- Among indexes sharing a leading field, prefer the one bounding more of
the filter's fields, then one covering the read's fields, then a
single-field one. A compound index now serves {grp, k} branches and
covered {grp: x} projections of grp and k.
- Merge an $or's single-predicate branches on one index into one scan of
their intervals, and order the OR's scans as MongoDB does: by index,
a branch of several predicates as soon as it is reached.
An $or of equalities on one field explains as an $in, without SUBPLAN.
A branch its index does not bound fetches and filters its documents
under the OR.
- Count the keys a multi-interval IXSCAN lands on outside its intervals,
as MongoDB's classic engine does. $in scans now report mongod's counts.
- The planner considers indexes in name order. MongoDB uses creation
order, which the index definition does not record.
- Explain a $match then a count ($count, or countDocuments' $group) as
MongoDB's SBE plan: explainVersion 2, queryPlan GROUP over a COUNT_SCAN,
IXSCAN, FETCH or COLLSCAN, nReturned the group's output, and no keys
counted between intervals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mongod encodes a magnitude below 2^-255, not 2^-256, as the double scaled by 2^768; the two encodings meet at 2^-255. Values in [2^-256, 2^-255) encoded out of mongod's order. The oracle corpus now covers that band. A decimal in the scaled band clears its leading double's lowest mantissa bit, which shares bit 1 with the decimal flags, so it cannot encode the same as a double there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A find projection such as {r: {$meta: "recordId"}} is neither inclusion nor exclusion in MongoDB: it keeps every field and adds r. DumboDB treated $meta as an inclusion and dropped the other fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Error messages such as E11000's "dup key: { _id: ... }" printed doubles
above 1000 as %.15e, other fractions to two places, and Decimal128 as
"Decimal128(high, low)". mongod prints a double as appendDoubleNice does
(%.16g, with ".0" unless the text holds '.', 'E' or 'N') and a decimal as
its string, e.g. "1e+20.0", "0.1", "1E-400".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mongod rejects explain in a multi-document transaction with OperationNotSupportedInTransaction (263) and leaves the transaction open. DumboDB ran the explain. It now refuses it, and the refusal does not abort the transaction (review N4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The maxTimeMS beside an explained command was accepted unchecked and ignored. It is now validated with mongod's errors (BadValue for a non-number, a fraction, or a value outside [0, 2^31-1]) and bounds the explain, which fails with MaxTimeMSExpired when it runs out (review N4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestKeyStringMatchesMongod skipped in every CI job because none set MONGOD_BIN, so the byte-for-byte check of the primary key format ran only by hand. The parity job already installs mongod 8.0.28; it now runs the test there and fails if the test skips (code review C8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macneale4
force-pushed
the
macneale4/query-planning-explain
branch
from
November 9, 2026 23:33
58488ed to
7d15429
Compare
Since the RecordId became the _id KeyString, $project, $set and $addFields carried it onto documents they rebuilt, so find on a view that hides _id returned the base _id through showRecordId or $meta "recordId". Reshaped documents now carry no record key, as on mongod, and find on a view refuses showRecordId with InvalidPipelineOperator, as mongod does (security review O10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A covered read reported a null key as null unless the filter compared with null, so $exists, $type, $not and $all ran against an invented null: {a: {$exists: true}} returned a document without a. The read now reports null only when every predicate treats null and missing alike (comparisons, $in and $nin with non-null values, joined by $and or $or), as mongod does, and fetches the document otherwise (code review C28).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mongod reads a legacy point stored as a document from its first two fields, whatever their names. DumboDB stores documents with sorted keys, so geo.Point read {lng: 10, lat: 1} as (1, 10), and $geoWithin and 2d indexes matched points that are not there. Without the written order the point cannot be read correctly, so a stored embedded document is no longer a point. Arrays and GeoJSON points are unchanged. The README lists the deviation (code review C33).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MinKey and MaxKey bounds span every type except on a clustered collection's top-level _id. $elemMatch filtered each array element as a top-level document, so {arr: {$elemMatch: {_id: {$gte: MinKey}}}} kept the clustered _id bracketing and matched only MinKey. Elements are now matched with their _id renamed to a key BSON cannot hold (review N3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three write commands declared hint as an ignored string, so {$natural: -1} never reached the planner and a single-document write acted on the first match instead of the last. They now pass the hint to the query, as find does, and refuse a hint that names no index: BadValue as a command error for findAndModify and as the statement's write error for update and delete, as mongod does (review N3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
showRecordId returned a long on reads that walk a sort index or fetch an $or union, which set no record key (the union path set no RecordId at all). In a pipeline, $meta "recordId" was always missing. Both read paths now set the record key, a pipeline drops it after every stage except $match, and $meta "recordId" returns it while it is there; $set and $addFields never see it. This matches mongod (code review C30). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion mongod accepts create on an existing clustered collection only when the clusteredIndex spec matches, and returns NamespaceExists (48) when the name differs. DumboDB accepted any spec. Every DumboDB collection is clustered with the default spec, so a clusteredIndex naming another index now fails with 48 and mongod's message (code review C20). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mongod treats v: null as unset, truncates any number (including a Decimal128) to an integer, and refuses other types with TypeMismatch (14). DumboDB refused null and Decimal128 values, gave 5979704 for a string, and echoed v: 1.5 in the version error (code review C35). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unsorted find hinted to an index with a descending key returns documents in that index's order on mongod. DumboDB stores every key ascending, so find({}).hint({b: -1}) came back in ascending b. Such a find now sorts by the index's key pattern in memory, stably over the ascending read, so equal keys keep _id order as mongod's index keeps them in RecordId order. This covers mixed-direction compound indexes too (code review C32).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Array elements with equal keys, such as 2 and 2.0, share one index entry. Writing every element left the last one's type tags, so distinct on {a: [2, 2.0]} returned 2.0 where mongod returns 2. Equal rows now collapse to the first before they are written (code review C40).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A panic outside Apply only set the runtime phase to "failed": the member kept advertising its state and stopped replicating unnoticed. Each pass of the replication loop now recovers a panic, records it in the failure history, marks continuity lost (the member goes RECOVERING) and retries, as other failures do. Recovered panics, including Apply's, now log their stack instead of putting it into the failure message, which replSetGetStatus reports (code review C34, security note). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README said only "its indexed values plus the _id". The encoded size can be larger: string terminators and NUL escapes, collation sort keys, and hashed and text indexes, which store values and words rather than hashes. The README lists those cases, and dolt-storage.md now describes the cap and why it sits below Dolt's limit (code review C31). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mongodb-keystring.md said 3 of 3145 small decimals differ from mongod; the review's sweep found 2,290 of about 21,400 continuations differ, with order and uniqueness intact. The merge's tie comment said its order is MongoDB's; mongod's follows its heap and is unspecified. buildBSONFieldPredicate's comment still said NaN and infinities get no predicate (code review C39). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review asked to share one maxTimeMS helper with getMore and aggregate. mongod's errors differ by command (a string is BadValue for explain, TypeMismatch for getMore; a negative fraction is truncated for explain, "must be >= 0" for aggregate), and parity checks each, so the comment records why the check stays separate (code review C49). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Encoding a Decimal128 at an extreme exponent still takes up to a few hundred microseconds, and the loop over an _id $in's values never checked its context, so maxTimeMS could not bound a request carrying hundreds of thousands of them. The loop now checks every 256 values, and the clustered read returns the context's error. The per-value cost itself is unchanged (security review O3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain planned with the whole sort, which no index walk serves, and showed SORT over a COLLSCAN, while the measured run, like find, passed only a one-field sort and read the filter's index, so the plan contradicted its own counts. The planner now passes the sort under the same rule and shows SORT over the filter's index scan (code review C36). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain stopped its measured read at the limit, so find {u: {$gt: 0}} skip 2 limit 1 reported one key where mongod reads three, and a SORT under skip 1 limit 1 reported one document kept where mongod keeps two. The read now stops at skip plus limit, SORT reports what it keeps, and SKIP what passes it. DumboDB still fetches the skipped documents, which the skip_limit XFail records (code review C25).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry walk The sort walk counted index entries before dropping repeats of a document, so a multikey walk reported each document once per array element in its IXSCAN. It now counts what it yields. A lone $or answered by a walk was explained under SUBPLAN only on the trial-won path; the plan-cache-hit and fallback walks dropped it (code review C38). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A count that fetches documents reports, in explain, the matches its FETCH
passed up, skipped ones included: count {a: 3} hint b_1 skip 2 limit 3
returns 5 from FETCH on mongod. DumboDB reported the final count, 3. The
count now also returns how many matches it read before the skip (code
review C12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A count over a multikey index range counted every entry's document, so
count {m: {$gt: 5}} limit 5 read 189 keys where mongod reads 8. It now
streams the range and stops once it has found skip plus limit distinct
documents (code review C12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hint made count skip the index shortcut and fetch every document, so a count hinted to the multikey index it would count from anyway reported 50 documents examined where mongod reports 0. A filtered count now passes its hint to the backend, which counts from the index only when the hint names the index it would choose (code review C12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A count with an $in filter fetched every document and explained a COUNT_SCAN; mongod counts from an IXSCAN of the $in's ranges with no FETCH, on multikey indexes and others. The count now streams the ranges, each document once and stopping at skip plus limit, and explain shows the IXSCAN (code review C12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain ANDed the user's filter onto a view's leading $match even when the view reshapes documents, so find {z: 3} on a view projecting z: "$a" matched nothing, and an aggregate on a view dropped its own $match. A find, count or aggregate on a view that is not only $match stages is now explained as an aggregate of the view's stages and the command's, with leading $match stages merged as mongod merges them; a count on a view reports one document from its GROUP (code review C13).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does Three gaps in $lookup explain. A foreign view reported no indexesUsed at executionStats; it now reports its base collection's join index. A classic $lookup listed the following $unwind of its join array as a stage; it now absorbs it as its unwinding field. And a $match between a $lookup and that $unwind kept the join in the query plan; a $match that does not read the join's output now moves ahead of the $lookup, as mongod's optimizer moves it, so the $unwind follows its join (code review C14). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clustered storage. Fixes #190, fixes #206, fixes #207, fixes #210. Includes the #190 index layout and the query planning and explain work from the closed #191.
Collections are stored the way a MongoDB clustered collection is:
TestKeyStringMatchesMongod, run withMONGOD_BIN). The one known deviation: an inexact Decimal128 below 2^-255 keeps mongod's leading 9 bytes, order and uniqueness, but its continuation can differ (3 of 3145 corpus values). Seedocs/reference/mongodb-keystring.md. This fixes KeyString index encoding: negative doubles collide, BinData misordered #206.KeyString(_id)instead of a hash. Natural order is_idorder, and_idequality, ranges,$inand sorts read the map directly, explained asEXPRESS_CLUSTERED_IXSCAN/CLUSTERED_IXSCANwith mongod's counts.showRecordId()and$meta: "recordId"return the KeyString. This fixes _id ranges and sorts scan the whole collection: add an implicit ordered _id index #207.KeyTooLong(17280). Dolt's own limit is about 32KB; the constant's comment records what was tested.listIndexesandlistCollectionsreport every collection as clustered (time-series excepted);createvalidatesclusteredIndexas mongod does.Review fixes, one commit each with parity cases: the scan prefilter with Decimal128, NaN and large longs (#210);
$insort merges (deduplicated, heap-merged, at most 200 scans);_idregex; merge-conflict resolution with unique indexes;$sum/$avgacross value orders; MinKey/MaxKey ranges;$natural: -1hints; distinct over equal numbers and with a hint;$groupcollation; counts with a hint; geo distances, 2dsphere key errors and stored point forms; write, count, view and$lookupexplains; explainmaxTimeMSand explain inside a transaction; mongod's number formatting in error messages; replication panic recovery.Round 3 fixes: no RecordId through view stages that reshape documents; covered reads fetch when the filter tells null from missing; hints honored on update, delete and findAndModify;
$elemMatchon a nested_idwith MinKey/MaxKey; RecordIds on sort-walk and$orreads and through$match; hinted descending indexes read in index order; multikey indexes keep the first of equal elements;clusteredIndexvalidation; replication panics recorded and retried; count, view,$lookup, skip and sort-walk explain counts.Known differences added this round: a geo point stored as an embedded document is not read as a point, because stored keys are sorted; and a large
_id$inof tiny decimals is now bounded bymaxTimeMS, though each value still costs up to a few hundred microseconds to encode. Bugs found on main are filed as #216 to #222.Known differences, in the README's limitations:
$orindex scans run in index name order (mongod uses creation order), a custom clustered index name is not kept, and geo points stored as embedded documents are skipped.Validation:
go test ./..., replication, and the full parity suite pass locally against mongod 8.0.28. The KeyString oracle (TestKeyStringMatchesMongod) runs in the parity CI job against mongod 8.0.28.Companion parity PR: dolthub/dumbodb-parity-testing#66.
Pre-beta: existing data directories must be recreated; older tables are refused with that message.
🤖 Generated with Claude Code