You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6281fcb
Browse filesBrowse the repository at this point in the historyBrowse files
Migrate StringSimilarityEncoder to narwhals, add polars support
fit() rebuilds encoder_dict_ with narwhals cast(nw.String)/value_counts,
matching the CountEncoder/RareLabelEncoder convention. cast() preserves
nulls as null on both pandas and polars (verified empirically), unlike
pandas' own astype(str) which stringifies NaN to "nan" - this lets
"impute" mode fill_null("") directly and "ignore" mode drop_nulls()
before casting, replacing the old "nan"/"<NA>" text-sentinel workaround
with a real null check (col.is_null()) that can't collide with a
genuine category literally named "nan" or "<NA>" (both edge cases stay
covered by test_string_dtype_with_literal_nan_strings).
transform()'s per-row difflib.SequenceMatcher similarity has no
vectorised narwhals equivalent, so it's computed once per unique value
via numpy broadcasting (np.unique's inverse index fans the small
per-unique-value matrix back out to all rows) and reassembled with
nw.new_series()/with_columns(), same pattern DecisionTreeFeatures uses
for externally-computed new columns.
Benchmarked a pandas-specific fast path (X.join(dict-of-columns), as
DecisionTreeFeatures uses) against the unified narwhals with_columns()
here across 10k-100k rows x 1-10 columns x 5-50 categories: assembly
overhead ranges 0.9x-6.25x depending on shape, but the difflib
computation itself dominates wall time by 1-3 orders of magnitude in
every realistic scenario (e.g. 30ms difflib vs <1ms assembly overhead
at 100k rows/20 categories) - even the worst synthetic case (500 output
columns) only costs ~10ms extra out of an already tens-of-ms-to-seconds
transform. Went with the unified/merged implementation: no is_pandas
split, one code path for both backends.
Rewrote tests as single parametrized cases over
@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]),
keeping only the pandas-NA-sentinel tests (np.nan/pd.NA/None,
StringDtype) pandas-only since polars has no equivalent multi-sentinel
behavior to exercise. All doc examples (including the Titanic worked
example) re-verified against actual output; added a "With polars"
section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
0 commit comments