A run that indexes zero documents reports success

[ddeboer]

A run that indexes nothing reports success: ✔ Completed, exit 0, and a stage line claiming the full item count.

What happens

A SearchField declared required: true becomes optional: false in the Typesense collection. Give the pipeline documents that lack that field, and:

   - Stage CreativeWork
   ✔ Stage CreativeWork 413 items, 413 quads, took 7.3s
   ✔ Stage Person 157 items, 157 quads, took 1.2s
   …
  ✔ Completed in 13.7s

$ curl localhost:8108/collections
creative_works | docs: 0
people         | docs: 157
places         | docs: 34
terms          | docs: 571

Exit code 0. Nothing in the log at any level mentions a rejected document, and the stage line reports 413 — the count of items extracted, which reads as the count indexed.

How to reproduce

1. Declare a required: true field on a root type.
2. Index a dataset where that field is absent from every document (a derive returning undefined is the easiest way).
3. The stage reports the full item count, the run exits 0, and the collection is empty.

Control: restore the field so it populates, re-run the identical job, and the same collection gets its 413 documents. Only the required field differs.

Why this matters more than a wrong exit code

The failure is silent, total, and type-scoped. Six collections filled correctly in the same run, so nothing looks wrong from outside: the pipeline is green, the other types are populated, and only the one type that lost its required field is empty. In an unattended or scheduled run there is no signal at all — and if the collection previously held documents, an operator sees a search index that has quietly gone blank for one entity type.

It also defeats the provenance skip in the worst direction: the run is recorded as successful, so a re-run against an unchanged dataset skips, and the empty collection persists.

We reached this while confirming a claim we had only read from the code — that a missing required field would fail the batch loudly. It does not, and we would not have found out from the log.

Suggested

• Fail the run. A batch whose documents are all rejected should throw, exit non-zero, and not record provenance.
• Report per-document rejections, at least in aggregate: 413 items → 0 indexed, 413 rejected (missing required field: sdDatePublished). Typesense's import response carries the per-document error; today it appears to be discarded.
• Distinguish the two counts in the stage line. 413 items currently means extracted, not indexed, and those differ precisely when something has gone wrong.

Observed with @lde/search-indexer 0.6.3, Typesense 27.1, REBUILD_MODE=in-place.