Llamacpp really slow on Termux (Android)

[Darin755]

With the recent changes to ollama I'm working on migrating to pure llamacpp. However, for whatever reason Ollama on my phone is much faster than llamacpp. As a test model I've been using Granite4.1 3b and it runs reasonably fast on ollama. However, on llamacpp I only get around 0.05 tokens/sec which is painfully slow. When I lowered the thread count to 4 (my device has 8 CPU cores) I managed to get it up to 1-2 tokens a second. This is still really slow compared to ollama where I get around 12 tokens/sec.

Is there something I'm missing? It would be nice to switch to llamacpp but right now it is unusable.

[taronaeo]

A few things to check.

1. What quantization are you using? IIRC, Ollama defaults to Q4_K_M.
2. Do you have SMT turned on? If yes, you only have 4 physical cores, and should adjust the threads parameter respectively.
3. Are you building llama.cpp with any supported backend, or just CPU only?

[Darin755]

I'm using the Termux llamacpp package at the moment. I suspect that the problem might have to do with memory management but I'm not entirely sure. By default llamacpp also seems to use way to many threads so setting it manually to 4 does give a little boost to performance. I also confirmed that llamacpp is using Q4_K_M.

Honestly I probably need to just play around with it a bit.

[taronaeo]

Try to compile it from source, and also look for backends that your device supports. It is very likely that you are currently using a CPU-only build and lack the acceleration support that Ollama already shipped for you.

I don't work on Android but maybe these guides can help you.

Android guide: https://github.com/ggml-org/llama.cpp/blob/master/docs/android.md
OpenCL guide: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#opencl

[ZYZ666-RGB]

@Darin755 since you confirmed the Termux package + Q4_K_M and that capping threads to 4 already helps, a focused next step:

1. Add --no-mmap and keep -t at the big-core count (often 4 on phones that advertise 8 logical CPUs).
2. Run llama-bench -m … -t 2 -t 3 -t 4 -p 128 -n 64 and look at tg (tokens generated / s), not just interactive feel.
3. If tg stays ~1–2, try a source build with -DGGML_NATIVE=ON -DGGML_OPENBLAS=OFF — the Termux package can oversubscribe threads / miss i8mm vs Ollama’s tuned build, which is why Ollama looks fine on the same SoC.

If that (or the earlier checklist) gets you into a usable range, please mark an answer so other Termux users can find it.