Skip to content

vello_gpu: Fix methods for bilinear and bicubic image sampling - #1963

Merged
LaurenzV merged 6 commits into
mainfrom
laurenz/fix_bilinear_image
Oct 9, 2026
Merged

LaurenzV merged 6 commits into
mainfrom
laurenz/fix_bilinear_image

Conversation

@laurenz-canva

@laurenz-canva laurenz-canva commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

This PR makes the following changes:

  • It adds a shim in the test suite so that we can run most external texture tests (the once that sample the full texture) with Vello CPU as well. By doing so, Vello CPU can serve as the reference, and we increase the coverage of tests that compare against Vello CPU and GPU.
  • It fixes the issue where, instead of extending each tap of bilinear/bicubic samples, we only extended the original sample location. This is wrong, and is the reason why a couple of image tests were inconsistent between CPU and GPU. In addition to that, it also ports some changes to the extend logic that we recently applied to Vello CPU, to make the results more numerically stable and the code more comparable to Vello CPU.

This does unfortunately regress performance: Rendering a full-sized image on my Android phone goes from around 73FPS to 66FPS. See the below videos.

Before:

IMG_1336.MOV

After:

IMG_1337.MOV

But the previous approach was just fundamentally wrong. However, the good news is that with #1964, I will introduce a fast path that uses GPU-native bilinear sampling for images that are sampled from the whole external texture, which will not only undo this slowdown, but in fact make image rendering even faster compared to current main, when using external textures and sampling the whole texture! See that PR for more information. Since rendering whole images is the most common operation (except for glyph caching, but this is experimental right now, anyway. And nearest-neighbor sampling is less affected than bilinear sampling), in my opinion this is a trade-off worth taking.

Using the image atlas will unfortunately stay slower for now, even with #1964, but I think that's something we have to accept for now. Once we revisit glyph caching, we can figure out how to best fix this. We could also improve this in the future by restricting the possible image sampling modes for atlas images (for example, only allowing extend mode Pad). But the problem is simply that when we sample from an arbitrary subregion of an atlas, we have to emulate correct extension ourselves, which is much slower than letting the hardware do it, so this should be avoided.

@laurenz-canva
laurenz-canva marked this pull request as draft September 29, 2026 14:58
@laurenz-canva
laurenz-canva added this pull request to stack #1965 September 29, 2026 14:59
@laurenz-canva
laurenz-canva force-pushed the laurenz/fix_bilinear_image branch from 8ae8442 to 690d88d Compare September 29, 2026 15:08
Base automatically changed from laurenz/size-impr to main September 30, 2026 06:18
@LaurenzV
LaurenzV force-pushed the laurenz/fix_bilinear_image branch from 690d88d to 93e02c4 Compare September 30, 2026 06:18
@@ -1,5 +1,6 @@
{
"timeouts": {
"pageLoad": 1200000,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to add this to address another timeout failure resulting from our newly added tests: https://github.com/linebender/vello/actions/runs/36677557384/job/109765766086

@laurenz-canva
laurenz-canva force-pushed the laurenz/fix_bilinear_image branch from f50d0ce to 4ee3473 Compare September 30, 2026 07:36

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The border in the new snapshot is expected. The test uses bilinear sampling with repeat. This means that for example, for the top-right part, as we approach the right border of the yellow pixel, it should slowly start to fade into the red of the top-left pixel, hence why it becomes orange.

@LaurenzV
LaurenzV marked this pull request as ready for review September 30, 2026 08:02
@LaurenzV
LaurenzV requested a review from grebmeg September 30, 2026 08:58
@LaurenzV LaurenzV changed the title vello_gpu: Implement proper bilinear and bicubic image sampling vello_gpu: Fix methods for bilinear and bicubic image sampling Sep 30, 2026
@LaurenzV
LaurenzV removed the request for review from grebmeg October 1, 2026 09:29
@LaurenzV
LaurenzV marked this pull request as draft October 1, 2026 09:29
@laurenz-canva
laurenz-canva force-pushed the laurenz/fix_bilinear_image branch from 4ee3473 to 0fa8fe9 Compare October 1, 2026 10:58
@laurenz-canva
laurenz-canva removed this pull request from stack #1965 October 1, 2026 10:59
@laurenz-canva
laurenz-canva changed the base branch from main to laurenz/reduce_slots October 1, 2026 11:00
@laurenz-canva
laurenz-canva added this pull request to stack #1969 October 1, 2026 11:00
Base automatically changed from laurenz/reduce_slots to main October 2, 2026 06:21
@LaurenzV
LaurenzV force-pushed the laurenz/fix_bilinear_image branch from 0fa8fe9 to 76aed20 Compare October 2, 2026 06:21
@laurenz-canva
laurenz-canva force-pushed the laurenz/fix_bilinear_image branch 5 times, most recently from 217da5b to d129a87 Compare October 8, 2026 08:08
@LaurenzV
LaurenzV marked this pull request as ready for review October 8, 2026 08:31
@LaurenzV
LaurenzV requested a review from grebmeg October 8, 2026 08:49
Comment thread vello_tests/src/renderer.rs Outdated
Comment thread vello_gpu_shaders/shaders/helpers/external_texture.wesl Outdated
Comment on lines +76 to +83
let x = vec2<i32>(
external_axis_index(base_coords.x - 1.0, image_extend_modes.x, image_size.x),
external_axis_index(base_coords.x, image_extend_modes.x, image_size.x),
);
let y = vec2<i32>(
external_axis_index(base_coords.y - 1.0, image_extend_modes.y, image_size.y),
external_axis_index(base_coords.y, image_extend_modes.y, image_size.y),
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try a dedicated fast path for Pad/Pad? I'm wondering if some of the 73 → 66 FPS comes from the switch in extend_mode now running once per tap (4× for bilinear, 8× for bicubic) instead of twice per pixel. If the mobile compiler flattens it, Pad images would also pay for the repeat/reflect divisions. A single if all(image_extend_modes == vec2(EXTEND_PAD)) that just clamps the tap indices would sidestep that without changing the output, and it could also be where the hardware-sampling path from #1964 goes later.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah seems to help a bit! I'll add this for now then, but unsure if it's worth the additional complexity, especially if we are going to lean in fully on the native sampling path.

However, not sure I fully understand the second comment, I don't think the native sampling path can go in there, because it has other simplifications as well, so it will probably have to stay a separate path.

Comment on lines +108 to +119
let x = vec4<i32>(
external_axis_index(base_coords.x - 2.0, image_extend_modes.x, image_size.x),
external_axis_index(base_coords.x - 1.0, image_extend_modes.x, image_size.x),
external_axis_index(base_coords.x, image_extend_modes.x, image_size.x),
external_axis_index(base_coords.x + 1.0, image_extend_modes.x, image_size.x),
);
let y = vec4<i32>(
external_axis_index(base_coords.y - 2.0, image_extend_modes.y, image_size.y),
external_axis_index(base_coords.y - 1.0, image_extend_modes.y, image_size.y),
external_axis_index(base_coords.y, image_extend_modes.y, image_size.y),
external_axis_index(base_coords.y + 1.0, image_extend_modes.y, image_size.y),
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try extending only the first tap on each axis and stepping to the others? The taps are consecutive integers, so after one euclid_mod each neighbour is just +1 with a single conditional wrap. That would cut the divisions from 8 to 2 per bicubic pixel and from 4 to 2 for bilinear. It should be bit-identical too, since everything stays exact integer math. The only catch is tiny images, where one wrap isn't enough (repeat with width < 3, reflect with width 1), but those could fall back to the per-tap path.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might work, but I think special-casing smaller images adds to much complexity to the shader, so I will keep it as is for now if it's fine with you!

@LaurenzV
LaurenzV force-pushed the laurenz/fix_bilinear_image branch from e56f0be to 93c6347 Compare October 9, 2026 06:11
@LaurenzV
LaurenzV added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 235739c Oct 9, 2026
17 checks passed
@LaurenzV
LaurenzV deleted the laurenz/fix_bilinear_image branch October 9, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants