Skip to content

[vector] Expose IVF search tuning options - #9367

Merged
JingsongLi merged 1 commit into
apache:masterfrom
shyjsarah:codex/expose-vector-query-tuning
Aug 24, 2026
Merged

[vector] Expose IVF search tuning options#9367
JingsongLi merged 1 commit into
apache:masterfrom
shyjsarah:codex/expose-vector-query-tuning

Conversation

@shyjsarah

Copy link
Copy Markdown
Contributor

Summary

Expose the IVF and IVF-PQ query tuning options added by paimon-vector-index 0.4.0 through Paimon vector search requests.

This lets callers bound the initial filtered IVF probe expansion and control IVF-PQ batch distance-table reuse without changing the existing defaults.

Changes

  • propagate ivf.max_initial_filter_expansion_factor to scalar and batch vector searches
  • propagate ivf_pq.batch_table_reuse and ivf_pq.batch_table_reuse.max_bytes to batch searches
  • parse the reuse memory budget as a long value
  • reuse one resolved parameter object for filtered and unfiltered batch native calls
  • document the new options, their defaults, scope, and Recall@K trade-off
  • add coverage for validation, reuse mode propagation, memory budget propagation, and values larger than 32-bit integers

Compatibility

  • no default behavior change
  • no storage format change or index rebuild required
  • IVF-PQ batch table reuse remains auto with a 512 MiB default budget
  • the initial filter expansion cap remains unset by default

Paimon now consumes the released paimon-vector-index 0.4.0 through #9341. This restores and extends the Paimon-side option wiring previously introduced in #8925.

Testing

  • mvn -nsu -pl paimon-vector test (44 run, 0 failures, 0 errors, 2 skipped)
  • Checkstyle and Spotless checks passed as part of the Maven lifecycle
  • git diff --check

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit c4308ca into apache:master Aug 24, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants