Skip to content

pick a SAM device the GPU can actually run in objects_3d - #483

Open
salmanmkc wants to merge 8 commits into
google:mainfrom
salmanmkc:fix/objects3d-sam-webgpu-fallback
Open

pick a SAM device the GPU can actually run in objects_3d#483
salmanmkc wants to merge 8 commits into
google:mainfrom
salmanmkc:fix/objects3d-sam-webgpu-fallback

Conversation

@salmanmkc

Copy link
Copy Markdown
Contributor

on a GPU that reports the WebGPU default maxStorageBufferBindingSize of 128 MB, Detect spams uncaught validation errors during "fitting boxes…" and takes a very long time to produce anything:

Binding size (402653184) of [Buffer (unlabeled)] is larger than the maximum
storage buffer binding size (134217728)
... While calling [Device].CreateBindGroup([BindGroupDescriptor "MatMul"])

that 402653184 is SlimSAM's attention buffer: 4096 tokens over 12 heads at fp16 is exactly 384 MB, 3x over the cap. onnxruntime already requests the adapter maximum, so it can't be raised — the device genuinely won't go higher. machines with a big limit (mine reports 4 GB) never see it, which is why it reproduces for some people and not others.

the device pick was 'gpu' in navigator && navigator.gpu ? 'webgpu' : 'wasm' with fp16 whenever webgpu was chosen, and the comment claimed transformers.js downgrades transparently. it doesn't when device is passed explicitly — it throws, and separately fp16 needs the shader-f16 feature that plenty of mobile/XR GPUs don't expose.

so probe the adapter and check both things before choosing, then try candidates in order: webgpu/fp16webgpu/fp32wasm/fp32. same model every time, so masks don't change quality depending on the device — only where it runs. cpu stays last so a driver failure the capability check didn't predict still lands somewhere that works.

also stops caching a rejected _samPromise. getSam() hands back the cached promise, so one failed load used to make every later Detect press fail until reload.

simulating the 128 MB limit locally: before, uncaught WebGPU errors and no clean completion; after, it routes to cpu and fits boxes with an empty console. an unmodified adapter is unaffected and still runs webgpu/fp16.

?mask=segmenter is untouched — still an explicit choice, never automatic.

same one-liner exists in the objects3d addon on #417, fixed separately there.

SlimSAM's attention buffer needs 384 MB at fp16, but a GPU reporting the
WebGPU default 128 MB storage buffer binding limit cannot bind it, and
onnxruntime already requests the adapter maximum. Probe the adapter for
shader-f16 and the binding limit before choosing a backend, then try
webgpu fp16, webgpu fp32 and cpu in turn so the same model still loads.

Also stop caching a rejected load promise, which previously made every
later Detect press fail until the page was reloaded.
@salmanmkc

Copy link
Copy Markdown
Contributor Author

@dli7319 does this fix the GPU errors in the console for you?

@dli7319 dli7319 self-assigned this Aug 4, 2026
@dli7319

dli7319 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

The WebGPU errors are gone but the whole app completely freezes up now during detection when running on device 😅.

Screenshot 2026-08-04 at 10 57 05 AM

@salmanmkc

Copy link
Copy Markdown
Contributor Author

The WebGPU errors are gone but the whole app completely freezes up now during detection when running on device 😅.

Screenshot 2026-08-04 at 10 57 05 AM

ya hard to test since my device doesn't hit this memory limit, seeing as it works but is just slower, I believe it has it's own fallback and forcing CPU blocks the thread, so I think best thing to do is leave the error message for now since it works

@salmanmkc salmanmkc closed this Aug 5, 2026
@salmanmkc salmanmkc reopened this Aug 5, 2026
Routing those devices to wasm made things worse: onnxruntime's wasm
backend runs synchronously on the main thread and stalled the render
loop for over a minute, which reads as a frozen app on a headset. The
oversized binding only produces recoverable WebGPU validation warnings,
and detection still completes on the GPU, so warn about the limit and
carry on instead of switching backend.

Shrinking the input to fit the limit is not an option either: the
exported SlimSAM graph fixes pixel_values at 1024x1024.
@dli7319

dli7319 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

I tested this PR on Quest 3 and the same WebGPU errors appear there

Screenshot 2026-08-09 at 5 30 44 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants