[gravesee]

Opt-in tiled, row-wise histograms (stacked on #123)

Stacked PR: based on feat/model-identical-perf (#123). Merge that first; this PR's diff only shows the tiled histogram work. "Base" below means #123.

Summary

Adds an opt-in tiled_histograms setting (set_tiled_histograms in Rust, tiled_histograms=True in Python). When it's on, large nodes build histograms row by row from a tiled copy of the binned data, instead of a column at a time.

• Speed vs Feat/model identical perf #123: fits with 1M+ rows are 16–38% faster. Wide data gains the most.
• Speed vs main: with Feat/model identical perf #123 underneath, most cases are 38–51% faster than main.
• Off by default. With it off, models are byte-identical to Feat/model identical perf #123.
• Results with it on: sums are added in a different order, so trees can differ in the last bits. They don't depend on the thread count. Evaluation log loss matched tiling off to 5 decimals in every benchmark.

Where the gains come from

1. The column-wise fill is limited by scattered reads

The default fill (fill_feature_histogram) loops over features. For each feature it walks the node's rows and reads each row's bin from that feature's column.

• Scattered rows: below the root, a node's rows are scattered across the data, so each (row, feature) read is mostly its own cache miss on a 2-byte value.
• Gradient copies: each node also first copies its gradients and hessians into new vectors.
• Cost: a node costs about rows × features random reads.

2. A tile puts a row's bins in one cache line

TiledBins keeps a second copy of the bins, split into tiles of up to 32 consecutive columns and stored row by row within each tile. 32 u16 bins are 64 bytes, one cache line.

• What a task does: it sums one tile over one segment of the node's rows. For each row it reads one line of bins plus the row's gradient and hessian, then updates up to 32 columns' histograms.
• Effect: up to 32× fewer random reads per row.
• Precedent: XGBoost's hist method blocks columns by 32 the same way (kColBlockSize in src/common/hist_util.cc) when a full-row histogram doesn't fit in cache.

3. Each task's histogram fits in cache

Tiles are capped at 32 columns and 8,192 bins. A task's scratch histogram (f64 gradient and hessian sums) is then at most about 130 KB and stays in L2.

• Why not plain row-wise: a row-wise fill over all columns doesn't fit. At 800 columns each thread's histogram is about 3.3 MB.

• Kernel probe at 250k × 800 (histogram time per tree):

  Fill	ms per tree
  Column-wise	216–221
  Plain row-wise, all columns	209–216
  32-column tiles	114–123

• Larger tiles were slower at every width tested. Histogram time per tree (ms) on main, summed over a tree's nodes (main builds nodes one after another):

  Tile size	1M × 50	4M × 50	1M × 200	250k × 800
  Column-wise	49–52	183–198	147–156	167–172
  32 columns / 8,192 bins	31.0	83.8	75.1	81.0
  64 / 16,384	34.6	95.4	81.5	110.3
  128 / 32,768	35.1	95.8	98.9	116.0

4. Enough parallel work for any node, with fixed results

A task is one tile × one row segment.

• Segments: at least 2,048 rows each and at most 16 per node. Even a lone node (the root, or a LossGuide split) has work for 16 threads.
• Fixed results: each tile's segment sums are added in a fixed order, so results don't depend on the thread count.
• Fallback: nodes with fewer than 8 tasks use the column-wise fill.

5. Kernel details that made the difference

The first version barely helped: it saved 0–22 ms of tree time per iteration on the cases below. The final kernel saves 14–83 ms. The changes:

• Gradients are read by row, not copied first. Each row's gradient and hessian is read once per tile. When rows are scattered, they're prefetched 16 rows ahead along with the row's bins.
• No bounds check per value. Each column gets one extra slot for values past the last cut (+inf). Those slots are dropped at the end, as the column-wise fill drops those values.
• A plain loop over the whole tile when all of its columns are used.
• Smaller segments (2,048 rows, at least 8 tasks). Nodes of 10k–100k rows now qualify. With 8,192-row segments and at least 16 tasks, almost none did.

Measured: the saving is all in building trees

Tree time per iteration, base vs. this PR (perf_phases --mode phases, depth 6, 15 iterations):

Case	Base	Tiled	Saved per iteration
1M × 50	57.1 ms	42.7 ms	14.4 ms (−25%)
4M × 50	216.6 ms	139.9 ms	76.7 ms (−35%)
1M × 200	178.6 ms	95.8 ms	82.8 ms (−46%)
250k × 800	171.4 ms	103.2 ms	68.2 ms (−40%)

Over 30 iterations, this accounts for nearly all of the end-to-end savings below (for example, 2.3 s of the 2.5 s at 4M × 50). Other phases run the same code, with two exceptions:

• a one-time tile copy after binning (about 20 ms at 1M × 50);
• when sampling, sampled rows are read in place instead of being copied into a contiguous subset.

The saving doesn't depend on #123

The same change on main saves about the same time per fit:

Case	Saved on main	Saved on #123
1M × 50	0.55 s	0.54 s
4M × 50	2.68 s	2.51 s
4M × 50, GOSS	1.56 s	1.76 s
1M × 200	2.54 s	2.57 s
250k × 800	2.09 s	2.34 s

• Why the percentage is larger here: Feat/model identical perf #123's iterations are shorter, so the same saving is a bigger share.
• The two stack: they speed up different parts of the iteration.
• Caveat: the two columns come from separate benchmark sessions on the same machine and data.

Results

Median fit time over 3 interleaved rounds:

Case	main	#123	This PR	vs #123	vs main	Share of saving vs main from this PR
100k × 50	1.78 s	0.97 s	0.96 s	−1%	−46%	1%
1M × 50	3.74 s	2.36 s	1.82 s	−23%	−51%	28%
1M × 50, LossGuide 64 leaves	2.79 s	2.47 s	2.04 s	−18%	−27%	58%
4M × 50	13.42 s	9.36 s	6.85 s	−27%	−49%	38%
4M × 50, GOSS	11.66 s	8.52 s	6.76 s	−21%	−42%	36%
1M × 200	8.71 s	6.81 s	4.24 s	−38%	−51%	58%
1M × 200, colsample_bytree 0.5	6.13 s	4.48 s	3.78 s	−16%	−38%	30%
250k × 800	7.92 s	6.91 s	4.57 s	−34%	−42%	70%
250k × 800, colsample_bytree 0.1	2.87 s	2.27 s	2.40 s	+6%	−16%	—

With the missing branch (1M × 50, 30 iterations, one run), tiling cuts the fit from 2.57 to 1.88 s with the same log loss.

Where it doesn't help

• Small data. At 100k rows few nodes have enough work for tiles, so most histograms stay column-wise.

• Few sampled columns on wide data. A tile row is read whole, so with few sampled columns most of each read is wasted. Histograms fall back to column-wise when they use under a quarter of the columns in the tiles they touch (second commit). Crossover at 250k × 800, without the fallback:

  colsample_bytree	0.1	0.2	0.3	0.5
  Tiled vs. off	+18%	+2%	−7%	−17%

  At 0.1 the fallback means tiles are never used, but the tile copy is still built, which costs 6% vs. Feat/model identical perf #123.

Costs

• Memory: a second copy of the binned data, rows × columns × 2 bytes. That's 100 MB at 1M × 50; benchmark peak memory went from 1.06 to 1.15 GB.
• Not byte-identical to the default fill when on, because the sums are added in a different order. Results are the same for any thread count.

Commits

Commit	Change
18a0c61	Tiled histograms. TiledBins (in binning), the tiled fill and HistogramMatrix::build, which chooses between the tiled and column-wise fills. Tiles are passed through Tree::fit_with_tiles to depth-wise levels and LossGuide splits. Adds the tiled_histograms parameter in Rust and Python, and --tiled for perf_phases.
3874616	Column-wise fill when a tree uses few of a tile's columns (the colsample_bytree fallback above).

API notes

• Rust:
  • New GradientBooster::tiled_histograms field. It defaults to off when loading older models.
  • New set_tiled_histograms, Tree::fit_with_tiles and binning::TiledBins. Tree::fit is unchanged.
  • Breaking for custom splitters: Splitter::split_node and handle_split_info take an extra tiles: Option<&TiledBins> argument.
• Python: new tiled_histograms argument. It's included in get_params() and survives save and load.

Testing

• Rust: cargo test passes 71 tests, 7 of them new:

  • the tile layout;
  • the tiled fill matching the column-wise fill within f32 rounding (all and sampled columns, in-order and scattered rows, several segment sizes, values past the last cut);
  • identical results for 1, 3 and 8 threads;
  • falling back to column-wise for unordered columns, sparse column sets and small nodes;
  • booster fits close to the default;
  • identical trees across thread counts, with and without GOSS;
  • saving and loading the setting.

  cargo fmt --check passes; clippy reports no new warnings.

• Python: 122 tests pass, including the new test_tiled_histograms. ruff check and ruff format --check pass.

• Golden checks: scripts/perf_golden.py with tiling off matches Feat/model identical perf #123 on all 15 configurations.

Not done

• Skipping the tile copy when colsample_bytree is below 0.25, which removes the 6% cost above.
• Storing 1-byte bins when nbins <= 256, which would halve the tile memory and bandwidth.
• Release notes.

Benchmark setup

• Machine: Xeon Platinum 8272CL, 8 cores / 16 threads, run with 16 threads.
• Runs: examples/perf_phases.rs --mode fit, depth 6, learning rate 0.1, 256 bins, one 50k-row evaluation set scored every iteration. 30 iterations (100 at 100k rows, 40 with GOSS); median of 3 interleaved rounds.
• Data: from scripts/make_perf_data.py: continuous, discrete and zero-inflated columns, with missing values in about 30% of columns.
• main column: main with tiling off, plus a --colsample flag for perf_phases. Its models are byte-identical to main.

To reproduce:

python scripts/make_perf_data.py --out /tmp/forust-perf/w50_4m --rows 4000000 --eval-rows 50000 --cols 50
cargo run --release --example perf_phases -- --mode fit --data /tmp/forust-perf/w50_4m \
    --iterations 30 --max-depth 6 --tiled true

[gravesee]

Timings on dedicated compute comparing this PR to 0.7.0

n_rows	n_cols	max_depth	colsample_bytree	version	fit_s_median
1000000	200	5	0.3	forust	76.719
1000000	200	5	0.3	forust_pre-release	58.595
1000000	200	5	0.8	forust	98.473
1000000	200	5	0.8	forust_pre-release	77.133
1000000	200	8	0.3	forust	152.699
1000000	200	8	0.3	forust_pre-release	67.891
1000000	200	8	0.8	forust	203.502
1000000	200	8	0.8	forust_pre-release	89.527
1000000	800	5	0.3	forust	202.823
1000000	800	5	0.3	forust_pre-release	146.06
1000000	800	5	0.8	forust	297.515
1000000	800	5	0.8	forust_pre-release	174.186
1000000	800	8	0.3	forust	406.025
1000000	800	8	0.3	forust_pre-release	224.198
1000000	800	8	0.8	forust	695.727
1000000	800	8	0.8	forust_pre-release	291.276
4000000	200	5	0.3	forust	319.007
4000000	200	5	0.3	forust_pre-release	217.919
4000000	200	5	0.8	forust	395.884
4000000	200	5	0.8	forust_pre-release	268.882
4000000	200	8	0.3	forust	520.056
4000000	200	8	0.3	forust_pre-release	263.515
4000000	200	8	0.8	forust	831.746
4000000	200	8	0.8	forust_pre-release	429.859
4000000	800	5	0.3	forust	829.985
4000000	800	5	0.3	forust_pre-release	717.778
4000000	800	5	0.8	forust	1250.418
4000000	800	5	0.8	forust_pre-release	813.053
4000000	800	8	0.3	forust	1517.233
4000000	800	8	0.3	forust_pre-release	942.8
4000000	800	8	0.8	forust	2556.06
4000000	800	8	0.8	forust_pre-release	1040.026

[jinlow]

Very cool, and makes sense why this would lead to speed ups. Also agree with having it opt-in because of the extra memory is uses over the default.