Skip to content

fix(generic): use Welford's algorithm for numerically stable rolling std - #864

Merged
polakowo merged 2 commits into
polakowo:masterfrom
Starlight143:fix/rolling-std-numerical-stability
Jul 14, 2026
Merged

polakowo merged 2 commits into
polakowo:masterfrom
Starlight143:fix/rolling-std-numerical-stability

Conversation

@Starlight143

Copy link
Copy Markdown
Contributor

Fixes #714

Problem

rolling_std_1d_nb (vectorbt/generic/nb.py) computes the rolling variance from
cumulative sums using the naive two-pass formula:

var = (cumsum_sq_window - 2*mean*cumsum_window + n*mean**2) / (n - ddof)

This subtracts two large, nearly equal quantities whenever the input values have a
large offset relative to their variance (e.g. large price levels with small
fluctuations, as reported in #714). The subtraction discards most of the significant
digits and produces results that diverge sharply from pd.Series.rolling().std().

Fix

Replaced the cumsum-based formula with a single-pass Welford update, extended with a
matching "remove-point" update for the value leaving the sliding window (the standard
reverse of Welford's update). Welford's algorithm accumulates the mean and the sum of
squared deviations from the running mean directly, instead of accumulating and later
subtracting squared raw values, so it does not suffer from this cancellation.

The function signature, minp/ddof semantics, and NaN handling (values are skipped
when computing the mean/variance but still occupy their position in the fixed-size
sliding window) are all unchanged. rolling_std_nb (the 2-dim wrapper) is untouched
since it only calls rolling_std_1d_nb per column.

Numerical verification

Compared against pd.Series.rolling(window, min_periods=window).std(ddof=ddof) as the
reference, using the old and new implementations side by side:

Scenario old max abs err new max abs err
offset=1e8, window=4000 (issue #714 scenario) 9.153412e+00 1.062144e-09
offset=1e4, window=4000 5.743844e-05 4.350947e-13
typical price data (offset~100), window=20 1.162215e-09 5.641487e-12
typical price data (offset~100), window=20, ddof=1 1.192407e-09 5.788037e-12

For ordinary price-scale data the two implementations agree to within floating-point
noise (both around 1e-9 to 1e-12), so no existing golden values change. For the
large-offset case from the issue, the max absolute error drops from ~9.15 to ~1e-9.

Tests

Added test_rolling_std_numerical_stability to tests/test_generic.py, parametrized
over ddof in {0, 1}, covering:

Test results on this branch (uv run --python .venv --no-sync pytest tests/):

  • tests/test_generic.py -k rolling_std: 22 passed (20 pre-existing + 2 new)
  • tests/test_generic.py (full file): 170 passed
  • tests/test_indicators.py tests/test_engine.py tests/test_labels.py: 77 passed, 86
    skipped (skips are pre-existing, due to optional dependencies such as TA-Lib not
    being installed, unrelated to this change)
  • Full suite tests/: 938 passed, 99 skipped, 0 failed

No existing test assertions or golden values needed to change.

…std (polakowo#714)

rolling_std_1d_nb computed variance via a naive two-pass formula
(sum(x**2) - 2*mean*sum(x) + n*mean**2) reconstructed from cumulative
sums. When values have a large offset relative to their variance
(e.g. large price levels with tiny fluctuations), this subtracts two
nearly equal large numbers and loses most significant digits.

Replaced it with a single-pass Welford update (with a matching
remove-point update for values leaving the sliding window), which
tracks the mean and sum of squared deviations directly instead of
squaring raw values. Function signature, minp/ddof semantics, and
NaN handling are unchanged.

Verified against pd.Series.rolling().std(): with offset=1e8 and
window=4000, max abs error drops from ~9.15 to ~1e-9. Typical price
data (offset ~100) is unaffected (~1e-9 either way).

@polakowo polakowo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the approach makes sense and fixes the reported case. Before merging, please:

  • Change the validity check from window_len == ddof to window_len <= ddof; otherwise ddof > window_len can return zero instead of NaN.
  • Do not clamp every negative M2 to zero. Reverse Welford can drift materially negative on long/high-offset streams. Please only clamp tiny round-off and handle or recompute an unstable state. Please add tests for both cases.
  • Shorten the docstring and comments. A brief note that this uses Welford for numerical stability is enough.

…drift, shorten docs

- window_len == ddof -> window_len <= ddof, so ddof > window returns NaN
  instead of dividing by a negative window_len - ddof.
- Only clamp tiny round-off in M2 to zero; when the remove-point Welford
  update drifts M2 materially negative (long/high-offset streams),
  recompute mean/M2 from scratch over the current window instead of
  silently reporting zero variance. Drift tolerance is scaled by a
  decayed residual-magnitude estimate, not by `mean` itself (mean is
  dominated by any large offset and is useless as a drift scale).
- Shortened the rolling_std_1d_nb docstring/comments per review.
- Added tests: ddof > window returns NaN; a high-offset adversarial
  stream (offset=1e14) where blind-clamping would wrongly report exact
  zero for 1181/49971 points, verifying the recompute path does not.
@Starlight143

Copy link
Copy Markdown
Contributor Author

Pushed. All three:

  • window_len <= ddof now, so ddof > window returns NaN.
  • Docstring trimmed to one line.
  • M2 clamp: only zero it out for tiny round-off now. Material drift recomputes mean/M2 from scratch over the current window instead. Caught a bug in my own first pass while testing this — my drift threshold was scaled by mean², and mean is dominated by the offset, so at offset=1e8 that threshold was astronomically larger than any real M2 and the recompute branch never actually fired. Rescaled it off a decayed residual-magnitude estimate instead. Verified with an offset=1e14 stress case: blind-clamp wrongly reports exact zero for 1181/49971 points, the fixed version is 0.

Added tests for the ddof>window case and the drift case (checked against a fresh per-window np.std, not just pandas, since at that offset pandas' own rolling std isn't a reliable reference either).

@polakowo
polakowo merged commit d3986fd into polakowo:master Jul 14, 2026
1 check passed
@polakowo

Copy link
Copy Markdown
Owner

Thanks, merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

For the same precision data, there is an accuracy error in the results.

2 participants