Bizarre adventures in computer screen

[Run Your Own] Running llama-swap as a proxy for llama.cpp on Guix

Sun, 26 Jul 2026 00:00:00 +0000

So, we are not finished with locally running LLMs yet.

After some time spent using llama-cpp-server from the previous blog post, I faced a couple of nice-to-haves:

Ability to run different models by specifying desired one during the request
Scale down models when they are not used to save resources

So, I decided to refactor earlier created llama-cpp-server to be able to run multiple models (not in parallel, but by choice).

Idea

I started by trying to find any tool available for these nice-to-haves - and found a project called llama-swap. It is licensed under the MIT license and allows all the things I wanted to:

   Run multiple generative AI models on your machine and hot-swap between them on demand. llama-swap works with any OpenAI and Anthropic API compatible server and is used by thousands of people to power their local AI workflows.

Built in Go for performance and simplicity, llama-swap has zero dependencies and is incredibly easy to set up. Get started in minutes - just one binary and one configuration file.

So, it is written in Go (so I can package it in Guix relatively easily) and can swap between models. I already had a service config for running llama-cpp-server, so why not modify it just a little bit (hopefully!) for llama-swap?

A little bit

I decided to reuse previously written config as much as possible - so llama-cpp-configuration was simplified and now used inside a new llama-swap-configuration:

 (define-configuration/no-serialization llama-swap-configuration
  (host
    (string "127.0.0.1")
    "Host to run server on.")
  (port
   (string "8080")
   "Port to run server on.")
  (llamas
   (list '())
   "List of llama-cpp-configuration to run")
  (with-nvidia?
   (boolean #f)
   "Enable hack for running on Nvidia with Vulkan")
  (nvidia-driver-package
   (package nvidia-driver)
   "Nvidia driver package to use")
  (nvidia-modprobe-package
   (package nvidia-modprobe)
   "Nvidia modprobe package to use")
  (user
   (string "llama")
   "User to run as.")
  (group
   (string "llama")
   "Group to run as.")
  (timeout
   (string "2400")
   "Model startup timeout in seconds.")
  (log-to-stdout
   (string "both")
   "Logging: proxy, upstream, both or none. Defaults to both")
  (requirements
   (list '())
   "List of additional service requirements."))

(define-configuration/no-serialization llama-cpp-configuration
  (name
   (string)
   "Model name to be used in API")
  (huggingface-name
   (string)
   "Huggingface model to run.")
  (parameters
   (list '())
   "Model parameters."))

Most of the settings (except nvidia-related ones) are for the llama-swap itself. Nvidia settings actually apply to all specified llama-cpp-configurations - I thought it would be enough configurability for now (:

Of course, I had to modify the Shepherd service a little bit - it runs a different binary now and I also need to make sure the llama-server binary is in the PATH, so llama-swap could run it when needed.

Here is the result.

llama-cpp-server is gone, now llama-swap-server could be used in place. Even if you have only one model to run, it could be useful with lazy loading of models and scaling down models when they are unused.

There are a lot of changes, so I will try to comment on the caveats I faced.

Caveats

Packaging

llama-swap was not packaged for Guix, but with the Go importer it was relatively easy to package it for my personal channel:

 (define-public llama-swap
  (package
    (name "llama-swap")
    (version "228")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
              (url "https://github.com/mostlygeek/llama-swap")
              (commit (string-append "v" version))))
       (file-name (git-file-name name version))
       (sha256
        (base32 "1yxml6n4lyc8galm0q4jfxhlnhjkl2sbvsy2qzddc71rm90d7khh"))))
    (build-system go-build-system)
    (arguments
     (list
      #:tests? #f ;; TODO: Skip failing tests
      #:import-path "github.com/mostlygeek/llama-swap"
      #:phases
      #~(modify-phases %standard-phases
          ;; Force disable legacy ServeMux behaviour, with which routing
          ;; doesn't work.  It is enabled automatically because we use
          ;; GO111MODULE=off in go-build-system.
          (add-after 'unpack 'fix-httpmuxgo121
            (lambda _
              (substitute* "src/github.com/mostlygeek/llama-swap/llama-swap.go"
                (("^package main")
                 "//go:debug httpmuxgo121=0\n\npackage main")))))))
    (inputs (list go-github-com-stretchr-testify
                  go-github-com-shirou-gopsutil-v4
                  go-github-com-billziss-gh-golib
                  go-github-com-tidwall-gjson
                  go-github-com-tidwall-sjson
                  go-github-com-klauspost-compress
                  go-github-com-fxamacker-cbor-v2
                  go-gopkg-in-yaml-v3

                  go-github-com-gin-gonic-gin
                  go-github-com-charmbracelet-bubbles
                  go-github-com-charmbracelet-lipgloss
                  go-github-com-google-jsonschema-go))
    (home-page "https://github.com/mostlygeek/llama-swap")
    (synopsis "llama-swap")
    (description
     "llama-swap is an @code{OpenAI} API compatible server that gives you complete
control over how you use your hardware.  It automatically swaps to the
configuration of your choice for serving a model.  Since
@@url{https://github.com/ggerganov/llama.cpp/tree/master/examples/server,llama.cpp's
server} can't swap models, let's swap the server instead!")
    (license license:expat)))

There are a couple interesting things to mention about this package.

The first one is that I had to force disable legacy ServeMux behaviour. Because of the way Guix build Go applications (with Go modules disabled), some apps end up using the wrong ServeMux implementation and get 404s on correct paths. To overcome this issue a go:debug entry was added to the main file.

And the other one was UI - actually llama-swap has some fancy frontend, but as it is nearly impossible to build frontend in Guix and I don't use it (I am mostly working with LLM from Emacs), I decided just not to build it. It doesn't break anything from the API point of view, so it is fine.

Creation of YAML configuration file for llama-swap

I created the YAML file by using strings with spaces and newlines, not a lot of magic here:

 (define (serialize-llama-cpp-configuration config)
  (match-record config 
                (name huggingface-name parameters)
    (let ((llama-server-args (append (list "--port ${PORT}"
                                           (string-append "-hf " huggingface-name))
                                     parameters)))
      (format #f "  ~a:\n   cmd: llama-server ~a\n" name (string-join llama-server-args " ")))))

(define (llama-swap-config-file config)
  (match-record config 
                (llamas timeout log-to-stdout)
    (mixed-text-file "llama-swap-conf.yaml" "
healthCheckTimeout: " timeout "
logToStdout: " log-to-stdout "
models:
" (string-join (map serialize-llama-cpp-configuration llamas) ""))))

In the end I have a YAML configuration file with llama-server command lines:

 rodion@bumblebee-mighty ~$ cat /gnu/store/m3hpzgjyl66m01j1q8la2ddi0514q0l0-llama-swap-conf.yaml

healthCheckTimeout: 2400
logToStdout: both
models:
  qwen-3.5:
   cmd: llama-server --port ${PORT} -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL -c 64000 --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.00
  qwen-3.6:
   cmd: llama-server --port ${PORT} -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL -c 64000 --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.00

Not a big problem, but it required some time to make it more or less readable.

Make it run with Vulkan on Mesa

The previous version of llama-cpp-service failed to run on Mesa, so I had to also fix it.

Devices

As I run llama-swap using least-authority-wrapper, I have to create a filesystem mapping for all the necessary devices. And it turned out I had missed a couple of them before:

 (device-mappings
 (append
  (list (file-system-mapping
          (source "/dev/dri")
          (target source)
          (writable? #t))
        (file-system-mapping
          (source "/sys/dev")
          (target source)
          (writable? #t))
        (file-system-mapping
          (source "/sys/devices")
          (target source)
          (writable? #t)))
  (if with-nvidia?
      (map (lambda (dev)
             (file-system-mapping
               (source dev)
               (target source)
               (writable? #t)))
           '("/dev/nvidiactl"
             "/dev/nvidia0"
             "/dev/nvidia-modeset"
             "/dev/nvidia-uvm"
             "/dev/nvidia-uvm-tools"))
      '())))

The important part here is the addition of /sys/dev and /sys/devices, as they appeared to be necessary for Vulkan to work in Mesa. Yes, probably these mappings are too broad (I could try mounting only specific devices), but I am OK with this for now.

Vulkan drivers

The next problem was Vulkan ICDs for Mesa. I thought that setting the environment variable VK_ICD_FILENAMES to Mesa's icd.d directory would be enough, but it actually wasn't, because VK_ICD_FILENAMES doesn't support directories, only files. I tried VK_DRIVER_FILES, but it didn't work with llama.cpp. So I decided to go hacky and construct the list of all Mesa's Vulkan ICDs myself - now make-forkexec-constructor call looks like this:

 (make-forkexec-constructor
 (append (list #$llama-swap
               "-listen" (string-append #$host ":" #$port)
               "-config" #$config-file))
 #:environment-variables
 (list (string-append "HOME=" #$%llama-home)
       "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
       (string-append "VK_ICD_FILENAMES=" (if #$with-nvidia?
                                              #$icd
                                              (string-join (map (lambda (file) (string-append #$icd "/" file))
                                                                (scandir #$icd
                                                                         (lambda (dir) (string-suffix? ".json" dir))))
                                                           ":")))
       (string-append "PATH=" #$llama-cpp "/bin")))

icd here is created like this - it is a file for Nvidia case and directory for Mesa:

 (if with-nvidia?
    (mixed-text-file "nvidia_icd.x86_64.json" "{
    \"file_format_version\" : \"1.0.1\",
    \"ICD\": {
        \"library_path\": \"" nvidia-driver-package "/lib/libEGL_nvidia.so.0\",
        \"api_version\" : \"1.4.312\"
    }
}")
    (file-append mesa "/share/vulkan/icd.d"))

Making service work with Nvidia on headless device

While working on llama-swap-service, I finally removed all desktop-related software from the machine configuration (it was running KDE before) and moved it to a separate room. So it became headless. And this moment I found out that llama-swap-service stopped working after reboot.

The problem was that for the Nvidia device to work in headless mode, I had to load the kernel module and initialize all necessary devices before I actually ran anything.

So I added some more commands to the Shepherd service startup when the service was configured to be run on Nvidia:

 (lambda args
  (when #$with-nvidia?
    (format #t "llama-cpp: running nvidia-modprobe~%")
    (system* #$(file-append nvidia-modprobe-package "/bin/nvidia-modprobe") "-c0" "-m")
    (format #t "llama-cpp: running nvidia-smi~%")
    (system* #$(file-append nvidia-driver-package "/bin/nvidia-smi"))
    (format #t "llama-cpp: waiting for nvidia devices~%")
    (let loop ((remaining 300))
      (cond
       ((every file-exists? nvidia-devices)
        (format #t "llama-cpp: nvidia devices ready~%"))
       ((zero? remaining)
        (error "llama-cpp: timed out waiting for nvidia devices"))
       (else
        (sleep 1)
        (loop (- remaining 1))))))
  (apply forkexec args))

Before starting the actual llama-swap under least-authority-wrapper I now make sure all necessary setup commands are run and all necessary devices are present.

Example configuration

Currently, I have two configurations which I use daily.

Here is the one for a beefy Nvidia-powered machine:

 (use-modules (nongnu packages nvidia)
             (rodion services llama))

(service llama-swap-service-type
         (llama-swap-configuration
          (host "0.0.0.0")
          (with-nvidia? #t)
          (nvidia-driver-package nvidia-driver-new-feature)
          (nvidia-modprobe-package nvidia-modprobe-new-feature)
          (llamas (list (llama-cpp-configuration
                         (name "qwen-3.5")
                         (huggingface-name "unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL")
                         (parameters (list "-c" "64000"
                                           "--temp" "0.6"
                                           "--top-k" "20"
                                           "--top-p" "0.95"
                                           "--min-p" "0.00")))
                        (llama-cpp-configuration
                         (name "qwen-3.6")
                         (huggingface-name "unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL")
                         (parameters (list "-c" "64000"
                                           "--temp" "0.6"
                                           "--top-k" "20"
                                           "--top-p" "0.95"
                                           "--min-p" "0.00")))))))

And here is the configuration for the laptop:

 (use-modules (rodion services llama))

(service llama-swap-service-type
         (llama-swap-configuration
          (host "127.0.0.1")
          (with-nvidia? #f)
          (llamas (list (llama-cpp-configuration
                         (name "apertus-8b")
                         (huggingface-name "unsloth/Apertus-8B-Instruct-2509-GGUF:UD-Q4_K_XL")
                         (parameters (list "-c" "9600")))
                        (llama-cpp-configuration
                         (name "qwen-7b")
                         (huggingface-name "Qwen/Qwen2.5-7B-Instruct-GGUF")
                         (parameters '("-c" "9600")))))))

No additional problems found with this setup, so will keep using it for some time, hopefully (: Have no plans of any breaking changes for this, so feel free to use it if you want.

It is available in my channel.

Happy hacking!

[Run Your Own] Running llama.cpp server on Guix

Wed, 27 May 2026 00:00:00 +0000

This is a start to a (hopefully) series of posts about running different kinds of things on Guix distro.

Idea

Even with all the ethical problems with LLMs (and, especially, companies providing them), technology could be useful enough for at least giving it a try.

Luckily, a couple of years ago I thought that I wanted to play some AAA games. I had a decent sum of money in hand, so I decided to buy a beefy configuration with RTX 4090 GPU. I haven't actually played anything seriously, except some hopeless crashes in Microsoft Flight Simulator to overcome my fear of flying, so most of the time the computer was gathering dust.

Until one not-so-sunny Berlin Saturday morning, when I finally decided to load it with local LLM inference.

How to run?

Question was answered pretty fast - Guix has llama.cpp packaged and it has server mode, which suits nicely. However, two problems appeared:

To use RTX 4090 fully I need to run proprietary drivers
There is no Guix system service for running llama.cpp as a server

On drivers

Thanks to Nonguix channel, running proprietary Nvidia drivers is not a big problem. You just have to carefully follow instructions from the README. Nowadays I am running nvda-595 driver and it is working fine (as fine as NVIDIA could on Linux).

On llama.cpp system service

This thing is a little bit more complicated. We have to provide correct Vulkan environment variables to make use of Nvidia proprietary drivers in Guix. To simplify these things I decided to pack all the complexity inside the Guix system service.

As this service uses proprietary software, there is no way it could be merged in the main Guix. But nobody can stop me from implementing it in my own channel.

So, here is llama-cpp-service-type. For now it only supports running GGUF models from HuggingFace via Vulkan backend. Following configuration parameters are available:

 (define-configuration/no-serialization llama-cpp-configuration
  (host
    (string "127.0.0.1")
    "Host to run server on.")
  (port
   (string "8080")
   "Port to run server on.")
  (huggingface-name
   (string)
   "Huggingface model to run.")
  (parameters
   (list '())
   "Model parameters.")
  (with-nvidia?
   (boolean #f)
   "Enable hack for running on Nvidia with Vulkan")
  (nvidia-driver-package
   (package nvidia-driver)
   "Nvidia driver package to use")
  (user
   (string "llama")
   "User to run as.")
  (group
   (string "llama")
   "Group to run as.")
  (requirements
   (list '())
   "List of additional service requirements."))

So, the only caveat here is providing correct Vulkan ICD path. Let's check it in detail:

 (define (llama-cpp-shepherd-service config)
  (match-record config 
                (huggingface-name parameters with-nvidia?
                                  nvidia-driver-package user
                                  group requirements
                                  host port)
    (let* ((icd (if with-nvidia?
                    (mixed-text-file "nvidia_icd.x86_64.json" "{
    \"file_format_version\" : \"1.0.1\",
    \"ICD\": {
        \"library_path\": \"" nvidia-driver-package "/lib/libEGL_nvidia.so.0\",
        \"api_version\" : \"1.4.312\"
    }
}")
                    (file-append mesa "/share/vulkan/icd.d")))
           (device-mappings
            (append
             (list (file-system-mapping
                     (source "/dev/dri")
                     (target source)
                     (writable? #t)))
             (if with-nvidia?
                 (map (lambda (dev)
                        (file-system-mapping
                          (source dev)
                          (target source)
                          (writable? #t)))
                      '("/dev/nvidiactl"
                        "/dev/nvidia0"
                        "/dev/nvidia-modeset"
                        "/dev/nvidia-uvm"
                        "/dev/nvidia-uvm-tools"))
                 '())))
           (llama-server
            (least-authority-wrapper
             (file-append llama-cpp "/bin/llama-server")
             #:name "llama-server"
             #:user user
             #:group group
             #:preserved-environment-variables
             (append %default-preserved-environment-variables
                     '("SSL_CERT_FILE" "VK_ICD_FILENAMES" "HOME"))
             #:mappings
             (append
              (list (file-system-mapping
                      (source "/etc/ssl/certs/ca-certificates.crt")
                      (target source)
                      (writable? #f))
                    (file-system-mapping
                      (source %llama-home)
                      (target source)
                      (writable? #t))
                    (file-system-mapping
                      (source (if with-nvidia? nvidia-driver-package mesa))
                      (target source)
                      (writable? #f))
                    (file-system-mapping
                      (source icd)
                      (target source)
                      (writable? #f)))
              device-mappings)
             #:namespaces
             (fold delq spaces '(net user)))))
      (list (shepherd-service
              (documentation "Llama-cpp server")
              (provision '(llama-cpp-server))
              (requirement (append '(networking) requirements))
              (start #~(make-forkexec-constructor
                        (append (list #$llama-server
                                      "--host" #$host
                                      "--port" #$port
                                      "-hf" #$huggingface-name
                                      #$@parameters))
                        #:environment-variables
                        (list (string-append "HOME=" #$%llama-home)
                              "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
                              (string-append "VK_ICD_FILENAMES=" #$icd))))
              (stop #~(make-kill-destructor)))))))

There are basically three parts:

Determining Vulkan ICD path depending on passed with-nvidia? flag and Nvidia driver version.
```
 (if with-nvidia?
    (mixed-text-file "nvidia_icd.x86_64.json" "{
    \"file_format_version\" : \"1.0.1\",
    \"ICD\": {
        \"library_path\": \"" nvidia-driver-package "/lib/libEGL_nvidia.so.0\",
        \"api_version\" : \"1.4.312\"
    }
}")
    (file-append mesa "/share/vulkan/icd.d"))
```
If we have with-nvidia? flag passed, we create an ICD in Guix store with a path to nvidia-driver-package. It is necessary, because ICD provided in driver package doesn't work for headless setup (and in my case beefy PC is headless, so it is necessary). In case we run without Nvidia proprietary driver, we fallback to Mesa ICDs.

As we are using least-authority-wrapper for an enhanced security, it is necessary to specify all the devices we will need access to:

 (device-mappings
 (append
  (list (file-system-mapping
          (source "/dev/dri")
          (target source)
          (writable? #t)))
  (if with-nvidia?
      (map (lambda (dev)
             (file-system-mapping
               (source dev)
               (target source)
               (writable? #t)))
           '("/dev/nvidiactl"
             "/dev/nvidia0"
             "/dev/nvidia-modeset"
             "/dev/nvidia-uvm"
             "/dev/nvidia-uvm-tools"))
      '())))

We always mount /dev/dri and, in case of Nvidia, specify additional Nvidia devices.

Finally we create least-authority-wrapper itself. It is pretty straightforward: just getting all file mappings together and specifying environment variables we want to save for the running service:

 (llama-server
 (least-authority-wrapper
  (file-append llama-cpp "/bin/llama-server")
  #:name "llama-server"
  #:user user
  #:group group
  #:preserved-environment-variables
  (append %default-preserved-environment-variables
          '("SSL_CERT_FILE" "VK_ICD_FILENAMES" "HOME"))
  #:mappings
  (append
   (list (file-system-mapping
           (source "/etc/ssl/certs/ca-certificates.crt")
           (target source)
           (writable? #f))
         (file-system-mapping
           (source %llama-home)
           (target source)
           (writable? #t))
         (file-system-mapping
           (source (if with-nvidia? nvidia-driver-package mesa))
           (target source)
           (writable? #f))
         (file-system-mapping
           (source icd)
           (target source)
           (writable? #f)))
   device-mappings)
  #:namespaces
  (fold delq spaces '(net user))))

SSL certificates are needed for llama.cpp to be able to access HuggingFace via HTTPS. Also, ICD path and the whole Mesa or Nvidia driver path in Guix store are mapped, so all the Vulkan drivers stuff could be loaded.

What to run?

As we have the service in place, we now can use it in OS configuration. Here is an example:

 (use-modules ...
             (rodion services llama))

(operating-system
  (services (list ...
                  (service llama-cpp-service-type
                           (llama-cpp-configuration
                            (huggingface-name "unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL")
                            (parameters (list "-c" "64000"
                                              "--temp" "0.6"
                                              "--top-k" "20"
                                              "--top-p" "0.95"
                                              "--min-p" "0.00"))
                            (host "0.0.0.0")
                            (with-nvidia? #t)
                            (nvidia-driver-package nvidia-driver-595))))))

In my case machine cannot be accessed from the Internet (only via the VPN), so 0.0.0.0 host seems to be a rather safe choice.

When the model had been downloaded, you can access UI via the http://:8080/ to give your newly deployed model a try.

Of course, OpenAI-compatible API is also available on the same host and port and could be used with a wide range of tools, i.e. with Claude Code, gpt.el in Emacs or my favorite editor-agnostic ECA.

Model I am currently running is Qwen3.5-27B with 64000 context. It can be used for light coding and asking questions, as well as tool calling with speed ~32 t/s on RTX 4090.

For now I am delighted with results, stay tuned and happy hacking.

P.S. If you have any questions/patches on the little-guix-channel or this llama-cpp-service-type in particular, don't hesitate to write an email on rodion [at] goritskov.com