Building DLLs
This document provides instructions to replicate our Windows DLL builds.
Requirements:
Windows 11 (x64)
Build Tools for Visual Studio 2022
AMD HIP SDK 7.1.1
Vulkan SDK 1.4.341.1
In the MSYS shell
As our Makefile makes massive use of unix shell applications, it's much easier to just replicate that environment. We'll need it just to setup the repo (with make setup), which initializes all the submodules and applies our patches to the original code. In theory after the setup you should also be able to run make -j8 to build llamafile with cosmocc on Windows (but honestly, why?)
install the required tools (vim is not really required)
pacman -S git patch unzip wget make vimcreate a build workspace
mkdir /c/Users/Your_Username/workspace
cd /c/Users/Your_Username/workspaceclone the repo
setup
In the Windows terminal
After the repo is set up, you can build the cuda / rocm / vulkan DLLs as follows. The .bat files to run the builds are in the llamafile directory and accept the following parameters:
--cleanto restart a build from scratch--outputto provide a custom output filename for the dll (default is ggml-xxxx.dll in the current directory for xxxx in (cuda, rocm, vulkan)only for the cuda libraries, you also have the
--cublasoption to link the library against NVIDIA's cublas instead of tinyblas
Also note that for cuda and rocm libraries there are *_parallel.bat scripts that should work faster by parallelizing compilation and taking advantage of your compute. Here's how you call the build scripts:
cd to the llamafile dir and start CUDA parallel build (this will run for a while...)
run ROCm parallel build as follows:
run Vulkan build as follows (parallel is not needed, this is usually much faster than the other two):
At the end of this process, you should have the following libraries available in your llamafile directory (note that sizes might differ):
To run llamafile with these libraries, add them in your home directory or bundle them in your llamafile (see Creating a llamafile).
Verifying numerical consistency (release gate)
Before shipping a GPU library, verify that it computes the same results as the CPU reference. The backend_ops_test harness wires upstream ggml's test-backend-ops into llamafile's real runtime loading path (cosmo_dlopen
ms_abi thunks), so it validates the actual DSO artifact and ABI boundary.
Build it once (it is not part of the default test suite):
Copy o//tests/backend_ops_test next to the DSO(s) on the target machine (rename to backend_ops_test.exe on Windows) and run the fast subset:
A full backend_ops_test test run covers every op and takes much longer. Every backend DSO present is loaded and tested; the summary at the end must report all backends passing.
If a mismatch appears, attribute it before blaming the GPU: the CPU reference itself goes through llamafile's tinyBLAS/iqk fast path. Rerun with LLAMAFILE_DISABLE_SGEMM=1 to compare against vanilla ggml — if the failures disappear, the bug is in llamafile's CPU kernels, not the GPU library (this is how the iq4_xs, bf16, and permuted-stride CPU bugs were found).
Last updated