VAE is not offloaded from VRAM when "Model Offload" is selected, is it fixable?

[alex-ie]

I have small VRAM and I use "Model Offload". Main model weights are released, however, VAE weights seems not offloaded after image generation completion. While VAE is often fast on CPU, on "old" CPU it is very slow in my experience.

Will it be easy to offload VAE weights too if "Model Offload" is selected? TIA

[wbruna]

"Model Offload" (or --sdoffloadcpu) should already work exactly the same for CLIP, diffusion and VAE weights. On which model are you seeing unreleased weights?

[henk717]

This ia probably what we already discussed wbruna. Keep in mind your fixes didn't land in the release yet.
Let us know if it also happens on https://koboldai.org/rolling

[wbruna]

Sorry, you are still not providing enough information. Are you using image to image, or LoRAs? On which Koboldcpp version?

[alex-ie]

im-2-im for now, I mostly do not use LORAs but start to learn them - how does it matter as VAE is not part of LORA?
1.120 linux.

[wbruna]

It matters because LoRAs are currently not released from VRAM with --sdoffloadcpu, and some could have sizes similar to VAE weights.

The VAE unload issue henk717 mentioned was fixed on version 1.119, so it's not that, either.

I do observe a VRAM bump after the first generation, though. But:

• it's still there on a plain text 2 image (so no VAE encoding) aborted in the middle of the generation
• it seems a bit smaller when using TAE (--sdvaeauto), but too large to be simply TAE weights (around 300M)

[alex-ie]

I do observe a VRAM bump after the first generation, though.

I too observe much larger bump when VAE are on GPU vs. if I select CPU for VAE at kcpp start - 50 MB vs. 600 MB. And that bump reduces max resolution of next images.

[alex-ie]

Today I have tried to run using Vulkan on GPU. VRAM looks completely released after each image generation in kcpp (as opposed to CUDA). But the duration with Vulkan is 2x longer.

[henk717]

Baaically very recently we fixed a vae unload issue but it wasn't fixed for audio vae's. Its because sdcpp itself doeen't do this. Rolling has a fix for it.

[alex-ie]

but it wasn't fixed for audio vae

But I experienced an issue with image vae, your note got me confused.

[henk717]

Still, its all so recent I do not take this issue as credible until you can reproduce it on the rolling build. Because that would proof its not fixed yet.

[alex-ie]

Today I have tried to run using Vulkan on GPU. VRAM looks completely released after each image generation in kcpp, flux 1 VAE too (as opposed to CUDA). But the duration with Vulkan is 2x longer.

[wbruna]

@LostRuins , @henk717 : the leak is not from stable-diffusion.cpp itself: I've added debug code to ggml_backend_cuda_buffer_type_alloc_buffer / ggml_backend_cuda_buffer_free_buffer to track VRAM usage, and everything is released after each gen. So it's either a driver, or a ggml backend issue.