Swapping LLMs in Under Two Seconds: A Sleep Mode for llama.cpp

Abstract#

Different tasks want different language models, and with a single GPU a proxy such as llama-swap has to swap them, often several times an hour. A swap between two GPU-filling models took 10.3 seconds and threw away the cached prompts of the model it replaced. One configuration change and a small wrapper, llama-sleep, which parks a model’s GPU memory in host RAM instead of stopping it, bring it down to 2.1 seconds with cached prompts intact. What remains is the copy between host RAM and GPU, held back mainly by 4 KB pages rather than by PCIe; with multi-size transparent huge pages, a kernel setting, the swap takes 1.7 seconds, about six times faster than at the start.

1 Introduction#

On my home server, a Gemma 4 12B model answers a voice assistant, and a Gemma 4 26B model handles long inputs, many of them from automations. Each fills the GPU alone, so every automation job costs two swaps: to the 26B and back. llama-swap [1], a proxy in front of llama.cpp’s llama-server [2], does the swapping: it stops the running model and starts the requested one. That costs a full model load, and the voice model loses its cached system prompt of about 8,000 tokens; recomputing it delayed the first spoken word by another 6–7 seconds.

vLLM has a sleep mode for this [3], and llama-swap ships a vllm-wrapper that uses it [4, 5]. llama-server’s --sleep-idle-seconds unloads a model after idle time [6], but it frees the KV cache with it and reloads the weights on the next request. The question of this note: how fast can a swap behind llama-swap get, with llama-swap unchanged and as little extra software as possible?

2 Setup and method#

• Hardware: NVIDIA RTX 3060 12 GB on PCIe 3.0 x16 (the Ryzen 5 PRO 5650G limits the slot to gen 3), 125 GB DDR4, Fedora 44, driver 615.71.
• Software: llama-swap v263 and llama.cpp b11552 (CUDA 13) in one container (ghcr.io/mostlygeek/llama-swap:v263-cuda13-b11552-non-root). All model files are in the page cache.
• Models: Gemma 4 12B QAT and Gemma 4 26B A4B QAT, both UD-Q4_K_XL. The 12B uses 7.9 GiB of GPU memory, the 26B 11.1–11.3 GiB with part of its expert weights on the CPU. For a worst case, two separate 26B processes are swapped against each other.
• Metric: time from sending a request for the other model until its first token arrives (max_tokens 1), measured at the client. Each value is the mean of two steady-state rounds of the cycle 26B → 26B → 12B.
• Breakdowns: llama-swap logged with microsecond timestamps and the wrapper logged every step; client, proxy and wrapper events were merged into one timeline.

3 Where the ten seconds go#

A baseline swap between the two 26B models takes 10.3 seconds: stopping the old process, starting the new one, loading the weights and waiting until llama-swap’s health check succeeds. Two parts of that are pure overhead.

• Probing memory. llama.cpp’s --fit finds out how many layers fit on the GPU by creating trial contexts, again at every start. A fixed split (--n-cpu-moe 14 for the 26B) halves the swap to 5.5 seconds. The catch: the split is tuned by hand for one model on one GPU. llama-sleep’s wake does not depend on it, but every cold start does.
• Polling. llama-swap checks the new model’s health 250 ms after starting it and then once a second [1], so a model that is ready just after a check waits up to a second.

Skipping llama.cpp’s warmup run (--no-warmup) changes nothing measurable. What remains is loading weights that the machine already holds in RAM, and that is what parking avoids.

4 Parking instead of stopping#

4.1 What cuda-checkpoint can do#

Since driver 550, NVIDIA’s cuda-checkpoint tool can suspend the CUDA state of a running process [7]. It copies the process’s device memory into host memory and frees the GPU; a second call copies it back to the same addresses. On the running 12B this took 3.8 seconds out and 1.5 seconds back in, and the model answered correctly afterwards.

Two observations shaped the design:

• A request to a parked model crashed it. A suspended llama-server still answers /health with 200, so llama-swap considers it ready and forwards the request while the restore is still running. The process then segfaulted inside libcuda. NVIDIA’s documentation says such calls may block until CUDA is resumed [7]; with llama-server they did not.
• A frozen process cannot be woken. Freezing the container would keep requests away, but the restore then hangs: it runs on a CUDA restore thread of the target process (cuda-checkpoint --get-restore-tid reports its ID) [7, 8], and the cgroup freezer stops that thread too.

4.2 llama-sleep#

llama-sleep [9], MIT-licensed, follows the pattern of llama-swap’s vllm-wrapper [4]: a small program that is the model’s cmd and cmdStop. Figure 1 shows how it swaps two models.

macros:
  "serve": llama-sleep serve --name ${MODEL_ID} --listen 127.0.0.1:${PORT} --idle 5s --
  "sleep": llama-sleep sleep --name ${MODEL_ID} --stop-pid ${PID}
models:
  "gemma-26b": { cmd: "${serve} llama-server ...", cmdStop: "${sleep}" }

• It holds requests. llama-server runs as a detached daemon that only llama-sleep can reach, through a Unix socket or an internal container port. llama-sleep is the reverse proxy on llama-swap’s port and holds every request, the health check included, until the model is awake and healthy. This prevents the crashes and replaces llama-swap’s polling: the health check returns the moment the model is ready.
• It parks in the background. With --idle 5s, a model that has been idle for five seconds is parked while nobody waits. A later swap then only has to wake the new model.
• It wakes on start. llama-swap starts serve because a request is waiting, so the wake begins at once instead of at the first health check.
• It is one file. llama-sleep is a single Go binary of about 7 MB, built without cgo. It calls the checkpoint functions of the CUDA driver API (cuCheckpointProcess*) [7, 8] through dlopen [10] and needs neither cuda-checkpoint nor the CUDA toolkit.
• It fails safe. A changed command line replaces a parked daemon instead of resuming it; a failed restore or a crashed daemon ends serve, so llama-swap notices and starts the model fresh on the next request.

5 Results#

The steps are cumulative: the fixed layer split from section 3; holding requests, parking in the background and waking on start from section 4.2; and huge pages from section 6. Each removes a visible part of the swap, and parking brings the largest gain (Figures 2 and 3). The direction not shown, from the 12B to the 26B, behaves like the 26B pair: 10.1 seconds at the start, 1.6 at the end.

Cached prompts survive. A 9,589-token prompt sent to the 12B again after a swap to the 26B and back was answered with one token processed and 9,588 taken from the cache; after a cold start, all 9,589 had to be processed.

Parking pays off only in the background. Parked during a swap, the slower copy direction sits on the critical path: with the fixed split, 26B → 12B then takes 6.8 seconds against 3.5 seconds for stopping the old model.

With parking in the background and the wake on start, everything around the copy shrinks to 82 ms: llama-swap, process start, unlock, health check, prompt and first token. The other 1,784 ms are the copy back to the GPU (Figure 4).

6 The copy: page faults, not the bus#

Both copies, out of the GPU when parking and back in when waking, are far slower than the link. A small benchmark with the CUDA driver API (1 GiB copies, best of five) shows what the link can do: 13.4 GB/s from pinned host memory to the GPU and 13.2 GB/s back, and about 12–13 GB/s with ordinary memory that has been touched before. Copied from the GPU into a freshly allocated buffer, the same data moves at only 2.2 GB/s, which is exactly llama-sleep’s park speed. Parking is that case: the driver allocates the host buffer when it parks a model, so every 4 KB page has to be faulted in and zeroed during the copy, about two million of them for 8 GB. The faults, not PCIe, set the pace.

Larger pages mean fewer faults [11]. Classic 2 MB transparent huge pages helped, but only while the kernel had free 2 MB blocks: on this host, which had been running for a while, the first two models got them and the third did not, and forcing them with synchronous compaction made parking slower. Multi-size transparent huge pages for anonymous memory (mTHP, Linux 6.8 and later) add sizes between 4 KB and 2 MB, and the kernel kept finding those in the same fragmented memory:

# as root; not persistent, so run it at every boot (for example from tmpfiles.d)
cd /sys/kernel/mm/transparent_hugepage
for s in 16 32 64 128 256 512 1024; do
  echo always > hugepages-${s}kB/enabled
done

With mTHP for 16 KB to 1 MB, llama-sleep parks about twice as fast and wakes about 1.5 times as fast, for both models and in every round:

Cold starts are not affected: the 26B took 9.7–11.6 seconds from container start to first token with mTHP and 10.4–11.8 seconds without. End to end, the swap between the two 26B models drops from 2.1 to 1.7 seconds and the swap back to the 12B from 1.9 to 1.1 seconds, the last step in Figure 2.

The wake still averages only about 8.5 GB/s, but not because the copy is slow. A trace of every copy during a wake of the 12B (8.3 GB) shows the driver staging the data through two alternating 2 MiB buffers: it fills one while the other crosses the bus, and once running, this loop moves 12.8 GB/s, 96 % of the link. Of the wake’s 0.98 seconds, 0.62 are the transfer at bus speed and 0.22 the driver’s setup and teardown around the copy. About 0.11 go to the first 1.3 GiB, which cross at half speed while the idle GPU brings its clocks and PCIe link up from their power-saving state; the last 0.02 are the handoffs between the two buffers.

A faster or multi-threaded memcpy, locked GPU clocks and a CPU performance policy changed nothing. Keeping the GPU awake saves less than 0.1 seconds per wake but costs 25–30 W around the clock. On this hardware the wake is at its practical limit; PCIe 4.0 would halve the transfer.

7 Limitations#

• A parked model occupies as much host RAM as it used on the GPU.
• A swap that arrives while a model is still being parked waits for it; with 8-second gaps between requests, swaps took 3.2–5.3 seconds instead of 2.1 (measured with 4 KB pages). The idle time has to fit the workload.
• NVIDIA GPUs and Linux only, driver 570 or later. One machine; differences below about 0.15 seconds are noise.

8 Conclusion#

Most of the time a swap took was overhead around the model, not the model itself. The copies that are left were held back by 4 KB pages rather than by the bus; a kernel setting removes most of that, and the wake now spends most of its time on the bus itself.

References#

1. mostlygeek. llama-swap. github.com/mostlygeek/llama-swap
2. ggml-org. llama.cpp. github.com/ggml-org/llama.cpp
3. vLLM. Sleep Mode. docs.vllm.ai
4. llama-swap. vllm-wrapper. github.com/mostlygeek/llama-swap/tree/main/cmd/vllm-wrapper
5. llama-swap issue #1267. Feature request: Support vllm level 2 sleep in vllm-wraper. github.com/mostlygeek/llama-swap/issues/1267
6. llama.cpp. llama-server: Sleeping on Idle. github.com/ggml-org/llama.cpp
7. NVIDIA. cuda-checkpoint. github.com/NVIDIA/cuda-checkpoint
8. NVIDIA. CUDA Driver API: CUDA Checkpointing. docs.nvidia.com
9. Robert Zaage. llama-sleep. git.zaage.it/robert/llama-sleep
10. Ebitengine. purego. github.com/ebitengine/purego
11. The Linux Kernel. Transparent Hugepage Support. docs.kernel.org