HTTP Caching of downloaded Models

[luckylinux]

I'd like to know how to cache downloaded Models (from Huggingface, I presume).

I don't currently have any experience with llama.cpp but I have been using ollama for a while.

Apparently, for ollama, it requires a bit of a special Caching Registry Proxy:
https://github.com/simonfrey/ollama-registry-pull-through-proxy

Ollama, supposedly, can be configured to use that using HTTPS_PROXY and HTTP_PROXY Environment Variable. Alternatively, the full URL to the Model can be specified when downloading it.

I wonder why, both for ollama and llama.cpp, it's not possible to "simply" use a squid Caching Proxy though 😕.

Mirroring all Models to a local Registry is probably going to use WAY too much Disk Space, so I think that's not a realistic Option.

Does llama.cpp have an equivalent Environment Variable to set the HTTP/HTTPS Proxy when downloading Models ?

And how could the Models be downloaded through a Caching System ?

[kokleong98]

If diskspace storage for all the cached models was your limitation, I think disregard what approach you will still short of disk spaces, otherwise you create some NFS sharing with cache and mount a directory whereby your llama cpp can read from with following environment variable.

Example for Linux/macOS

export LLAMA_CACHE="/path/to/your/custom/model/folder"

Example for Windows (Command Prompt)

set LLAMA_CACHE="C:\path\to\your\custom\model\folder"