Can InstantID be faster?

[duongna21]

Thank you for your great work!
I'm curious if we can make InstantID faster? It's 3x slower than the standard SDXL pipeline on my 4090 (2.73it/s vs 8it/s). What slows it down so much?

[haofanwang]

I think LCM-LoRA and SDXL-turbo would be good choices. Deserve a shot.

[haofanwang]

You can also turn on xformer for optimization.

[dm33tri]

You can also turn on xformer for optimization.

I found inference to be faster when using AttnProcessor2_0 (from the InstantID repo) than using xFormers. Perhaps code is unfinished, as there is a is_torch2_available function but it's not used?

[haofanwang]

Yes. Your gain benefits from torch2.0

[Syndulla]

You can also turn on xformer for optimization.

I found inference to be faster when using AttnProcessor2_0 (from the InstantID repo) than using xFormers. Perhaps code is unfinished, as there is a is_torch2_available function but it's not used?

Would you mind sharing how you implemented AttnProcessor2_0? I'd like to test it.

[wangqixun]

The simplest solution:
set num_inference_steps=20

The image quality is good enough. Speed up

[dm33tri]

You can also turn on xformer for optimization.

I found inference to be faster when using AttnProcessor2_0 (from the InstantID repo) than using xFormers. Perhaps code is unfinished, as there is a is_torch2_available function but it's not used?

Would you mind sharing how you implemented AttnProcessor2_0? I'd like to test it.

It's already implemented, you just swap AttnProcessor to AttnProcessor2_0 in pipeline_stable_diffusion_xl_instantid.py

[ResearcherXman]

We will update soon

[ResearcherXman]

Besides of Pytorch 2.0. We have already added an example for LCM-LoRA, which allows for fewer steps and can improve speed quite a bit.

[zvit]

@haofanwang Where do you turn on xformer?

[inck86]

Besides of Pytorch 2.0. We have already added an example for LCM-LoRA, which allows for fewer steps and can improve speed quite a bit.

"To use it, you just need to load it and infer with a small num_inference_steps"

I add in app.py and it was downloaded:
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="latent-consistency/lcm-lora-sdxl", filename="pytorch_lora_weights.safetensors", local_dir="./checkpoints")

but where should I insert this to make it work? :

from diffusers import LCMScheduler

lcm_lora_path = "./checkpoints/pytorch_lora_weights.safetensors"

pipe.load_lora_weights(lcm_lora_path)
pipe.fuse_lora()
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

num_inference_steps = 10
guidance_scale = 0

[inck86]

Thank you for your great work! I'm curious if we can make InstantID faster? It's 3x slower than the standard SDXL pipeline on my 4090 (2.73it/s vs 8it/s). What slows it down so much?

One guy from my channel wrote: Installed locally on ubuntu. The 3090 card. On average, an image takes about 22 seconds..
Maybe it's about Ubuntu?

[johndpope]

I'm on 3090 - 24gb VRAM - got 100gb of RAM (DDR5) / nvme samsung

I asked chatgpt to help profile - it suggested cprofile (comes out of the box)
seeing 16.289 seconds

am running this faceswap.py code (lcm)
https://gist.github.com/johndpope/71b70e876a28128228db8bcee2355b0a

#89

import cProfile
import faceswap  # Replace with your script/module name
import pstats


cProfile.run('faceswap.main()', 'profile_output')
p = pstats.Stats('profile_output')
p.sort_stats('cumulative').print_stats(10)  # Adjust as needed

Mon Jan 29 17:34:06 2024    profile_output

         16026416 function calls (11379666 primitive calls) i**n 16.289 seconds**

   Ordered by: cumulative time
   List reduced from 4085 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    334/1    0.006    0.000   16.295   16.295 {built-in method builtins.exec}
        1    0.036    0.036   16.295   16.295 <string>:1(<module>)
        1    0.004    0.004   16.260   16.260 /media/2TB/InstantID/faceswap.py:98(main)
        1    0.000    0.000    4.307    4.307 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/torch/utils/_contextlib.py:112(decorate_context)
        1    0.002    0.002    4.307    4.307 /media/2TB/InstantID/pipeline_stable_diffusion_xl_instantid_inpaint.py:154(__call__)
        1    0.000    0.000    3.390    3.390 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/diffusers/loaders/lora.py:1407(load_lora_weights)
     33/4    0.099    0.003    3.333    0.833 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py:102(_inner_fn)
        1    0.000    0.000    2.889    2.889 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/diffusers/loaders/lora.py:374(load_lora_into_unet)
        1    0.000    0.000    2.742    2.742 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/peft/mapping.py:136(inject_adapter_in_model)
        1    0.000    0.000    2.742    2.742 /home/oem/miniconda3/envs/comfyui/lib/python3.11/site-packages/peft/tuners/lora/model.py:110(__init__)

[haofanwang]

dreamshaperXL_turboDpmppSDE is good turbo model and can generate good results with fewer steps.

[inck86]

on Forge UI it works more, more faster!
About 1.5min to 1280x1024 for my 3060/12!