PantheonSim / Ollama on a simulated GPU

Run Ollama on a simulated GPU in the playground

PantheonSim, the GPU simulator for CUDA and ROCm, lets you run Ollama in the browser playground with the model loaded onto a simulated NVIDIA GPU, with nothing to install and no real GPU anywhere. This guide covers what the playground sets up, the commands to run, how to confirm the GPU was used, and the limits of a test like this.

What the playground sets up

Ollama's installer writes to /usr/local as root, and a playground session is an unprivileged user on a read-only root, so Ollama cannot be installed inside one. The playground image therefore carries Ollama (version 0.21.0) already. Its bundled CUDA backend would normally talk to an NVIDIA driver, so the image replaces that backend's libcudart, libcublas and libcublasLt with links to the simulator's. Ollama finds them on its own, and nothing is preloaded.

One small model is baked in, smollm2:135m, so ollama run works even on a machine with no network. On Linux, nothing starts the Ollama server for you, so the playground's ollama command does it: the first client command (run, pull, list, ps, show, rm, cp, create or stop) that finds no server starts one in the background and logs to ~/.ollama/serve.log.

Run a model

Open the playground, pick an NVIDIA card (the tests use a T4) and an operating system, and in the terminal:

ollama list
ollama run smollm2:135m "The capital of France is"

The first command starts the server and lists the baked-in model. Ollama's REST API works too, on the usual local address, which is how you script it:

curl -s --noproxy '*' http://127.0.0.1:11434/api/generate \
  -d '{"model":"smollm2:135m","prompt":"The capital of France is","stream":false,
       "options":{"num_predict":3,"temperature":0,"seed":1}}'

A simulated GPU loads a model slowly, so the playground's wrapper raises Ollama's load timeout to 60 minutes unless you set OLLAMA_LOAD_TIMEOUT yourself. Expect the first request to wait.

Confirm the GPU was used

Two places say so. Ollama's server log records the GPU it found, with its compute capability and name, and ollama ps shows how much of the loaded model sits on the GPU:

grep 'library=' ~/.ollama/serve.log | head -1
ollama ps

The playground's own checks do this. On a simulated T4, the log line names a Tesla T4, and the whole model is resident on the GPU. To compare against Ollama's CPU backend, stop the model and ask again with "num_gpu":0 in options:

ollama stop smollm2:135m
curl -s --noproxy '*' http://127.0.0.1:11434/api/generate \
  -d '{"model":"smollm2:135m","prompt":"The capital of France is","stream":false,
       "options":{"num_predict":3,"temperature":0,"seed":1,"num_gpu":0}}'

For that short greedy generation, the answer from the simulated GPU matches the CPU backend's. The playground image is built with this check as a gate: a model generated on a simulated GPU with every layer on it, matching Ollama's CPU backend, or the image does not build.

Other models

The playground's network reaches Ollama's registry, so ollama pull works for other models. A larger model takes longer to load and to generate on a simulated GPU, and a session lasts up to an hour, with ten idle minutes ending it, so a tiny model is the right test. You can also choose an AMD card: the ROCm image carries Ollama's ROCm backend with its HIP and HSA runtimes linked to the simulator, and keeps gfx12 matrix products in rocBLAS because the hipBLASLt kernel library is not part of Ollama's package.

Limits

It is a functional test. Tokens per second from a simulated GPU mean nothing. The simulator has no timing model and the kernels run on the CPU.
One model is verified. The checks above cover smollm2:135m and a short generation. They do not prove every model and every sampler setting.
The card must be one Ollama's backend knows. The simulator has profiles for architectures Ollama does not ship code for. For example, Ollama 0.21.0's CUDA backend has no build for compute capability 10.7, so Ollama itself rejects that simulated Rubin card.
It is the playground's setup. This guide describes the playground image. On your own machine you would link Ollama's bundled CUDA libraries to the simulator's in the same way.

Questions

Can I run Ollama without a GPU on a simulated one?

Yes, in the PantheonSim playground. Ollama is preinstalled, a small model is baked in, and the model loads onto the simulated GPU. It checks that things work, not how fast they are.

Why is the first Ollama request so slow?

A simulated GPU loads a model slowly, and the server starts on the first client command. The playground raises the load timeout to 60 minutes for that reason.

Next steps

Open the playground, see PyTorch on a simulated GPU, read the docs, or browse the GPU index. For programs you build yourself, see how to run CUDA without a GPU.