fix(generic): use Welford's algorithm for numerically stable rolling std - #864
Merged
polakowo merged 2 commits intoJul 14, 2026
Merged
Conversation
…std (polakowo#714) rolling_std_1d_nb computed variance via a naive two-pass formula (sum(x**2) - 2*mean*sum(x) + n*mean**2) reconstructed from cumulative sums. When values have a large offset relative to their variance (e.g. large price levels with tiny fluctuations), this subtracts two nearly equal large numbers and loses most significant digits. Replaced it with a single-pass Welford update (with a matching remove-point update for values leaving the sliding window), which tracks the mean and sum of squared deviations directly instead of squaring raw values. Function signature, minp/ddof semantics, and NaN handling are unchanged. Verified against pd.Series.rolling().std(): with offset=1e8 and window=4000, max abs error drops from ~9.15 to ~1e-9. Typical price data (offset ~100) is unaffected (~1e-9 either way).
polakowo
requested changes
Jul 14, 2026
polakowo
left a comment
Owner
There was a problem hiding this comment.
Thanks, the approach makes sense and fixes the reported case. Before merging, please:
- Change the validity check from
window_len == ddoftowindow_len <= ddof; otherwiseddof > window_lencan return zero instead of NaN. - Do not clamp every negative M2 to zero. Reverse Welford can drift materially negative on long/high-offset streams. Please only clamp tiny round-off and handle or recompute an unstable state. Please add tests for both cases.
- Shorten the docstring and comments. A brief note that this uses Welford for numerical stability is enough.
…drift, shorten docs - window_len == ddof -> window_len <= ddof, so ddof > window returns NaN instead of dividing by a negative window_len - ddof. - Only clamp tiny round-off in M2 to zero; when the remove-point Welford update drifts M2 materially negative (long/high-offset streams), recompute mean/M2 from scratch over the current window instead of silently reporting zero variance. Drift tolerance is scaled by a decayed residual-magnitude estimate, not by `mean` itself (mean is dominated by any large offset and is useless as a drift scale). - Shortened the rolling_std_1d_nb docstring/comments per review. - Added tests: ddof > window returns NaN; a high-offset adversarial stream (offset=1e14) where blind-clamping would wrongly report exact zero for 1181/49971 points, verifying the recompute path does not.
Contributor
Author
|
Pushed. All three:
Added tests for the ddof>window case and the drift case (checked against a fresh per-window np.std, not just pandas, since at that offset pandas' own rolling std isn't a reliable reference either). |
Owner
|
Thanks, merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #714
Problem
rolling_std_1d_nb(vectorbt/generic/nb.py) computes the rolling variance fromcumulative sums using the naive two-pass formula:
This subtracts two large, nearly equal quantities whenever the input values have a
large offset relative to their variance (e.g. large price levels with small
fluctuations, as reported in #714). The subtraction discards most of the significant
digits and produces results that diverge sharply from
pd.Series.rolling().std().Fix
Replaced the cumsum-based formula with a single-pass Welford update, extended with a
matching "remove-point" update for the value leaving the sliding window (the standard
reverse of Welford's update). Welford's algorithm accumulates the mean and the sum of
squared deviations from the running mean directly, instead of accumulating and later
subtracting squared raw values, so it does not suffer from this cancellation.
The function signature,
minp/ddofsemantics, and NaN handling (values are skippedwhen computing the mean/variance but still occupy their position in the fixed-size
sliding window) are all unchanged.
rolling_std_nb(the 2-dim wrapper) is untouchedsince it only calls
rolling_std_1d_nbper column.Numerical verification
Compared against
pd.Series.rolling(window, min_periods=window).std(ddof=ddof)as thereference, using the old and new implementations side by side:
For ordinary price-scale data the two implementations agree to within floating-point
noise (both around 1e-9 to 1e-12), so no existing golden values change. For the
large-offset case from the issue, the max absolute error drops from ~9.15 to ~1e-9.
Tests
Added
test_rolling_std_numerical_stabilitytotests/test_generic.py, parametrizedover
ddof in {0, 1}, covering:pandas.Series.rolling().std()(the scenario from For the same precision data, there is an accuracy error in the results. #714), and
minp/NaN-count bookkeepingin the sliding window.
Test results on this branch (
uv run --python .venv --no-sync pytest tests/):tests/test_generic.py -k rolling_std: 22 passed (20 pre-existing + 2 new)tests/test_generic.py(full file): 170 passedtests/test_indicators.py tests/test_engine.py tests/test_labels.py: 77 passed, 86skipped (skips are pre-existing, due to optional dependencies such as TA-Lib not
being installed, unrelated to this change)
tests/: 938 passed, 99 skipped, 0 failedNo existing test assertions or golden values needed to change.