Skip to content

fix: reject empty special token strings - #586

Open
chavalasantosh wants to merge 1 commit into
openai:mainfrom
chavalasantosh:fix/reject-empty-special-token
Open

fix: reject empty special token strings#586
chavalasantosh wants to merge 1 commit into
openai:mainfrom
chavalasantosh:fix/reject-empty-special-token

Conversation

@chavalasantosh

@chavalasantosh chavalasantosh commented Jul 29, 2026

Copy link
Copy Markdown

Summary

Reject empty special-token strings when constructing CoreBPE.

An empty special token produces a zero-width match. If that token is allowed
while encoding non-empty text, the match does not advance the encoding cursor,
so the operation does not complete. The configuration currently constructs
without an error.

Changes

  • Validate special-token strings in CoreBPE::new_internal.
  • Return a clear ValueError through the existing Python constructor wrapper.
  • Add a native Rust regression test.
  • Add a Python API regression test.

Validation

  • cargo test --lib — 3 passed
  • pytest tests -q — 34 passed
  • cargo fmt --check — passed
  • git diff --check — passed

Disclosure

I used AI assistance during investigation and implementation. I reproduced the
behavior, reviewed the complete change, ran the validation above, and take
responsibility for the contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant