PantheonSim / nvidia-smi simulator

A simulated nvidia-smi for testing monitoring tools and scripts

PantheonSim, the GPU simulator for CUDA and ROCm, includes drop-in nvidia-smi, rocm-smi and amd-smi commands that report simulated GPUs, along with lspci, dmesg, uname and hostname for the same machine. You can test a dashboard, a health check or a provisioning script against an eight-GPU server on a laptop that has no GPU at all.

What you get

The tools answer from a device profile, so a machine of eight H100s or eight MI300X cards looks like one to every program that asks. A program that talks to the NVIDIA Management Library finds it too: the simulator serves NVML as libnvidia-ml.so.1, so pynvml, gpustat and nvitop work against it. nvidia-smi supports the table, -L, -q (with -d sections and -x for XML), --query-gpu with --format=csv, -i, -l, topo -m and --version, with the column widths scripts cut on. On an AMD machine, rocm-smi (the concise table, -a, --json, --csv and many show options), amd-smi (list, static, metric, monitor and more) and rocm_agent_enumerator answer the same way.

Start a simulated machine with vgpu shell

vgpu shell opens a shell on the machine you describe. Everything that reports on GPUs inside it reports the machine you chose, and any CUDA or HIP program you build runs against it. Build the simulator first (make build in the repository), or use the playground, which runs the same tools in a browser terminal.

build/vgpu shell -y --gpu nvidia/h100 --count 8 --cuda 13.0 --driver 580.126.20 \
  --os ubuntu:24.04 --hostname dgx-test --load 0.65

Without -y it asks for each value. The options you will use:

The -c form is the one to use in a script or a test. This prints one line per simulated GPU and exits:

build/vgpu shell -y --gpu nvidia/h100 --count 8 -c \
  'nvidia-smi --query-gpu=index,name,memory.total --format=csv'

Inside the session the host's own filesystem, packages and CPU are still there; only the GPU-facing surfaces are simulated. Where the machine can create an unprivileged mount namespace, /proc/driver/nvidia/version and /sys/class/drm show the simulated machine as well, and uname and hostname report the kernel and name you set.

Describe the machine once, in a settings file

If you test against the same machine every time, write it down. vgpu config init writes a commented pantheonsim.yaml in the current directory, and every command reads it:

vgpu config init
cat pantheonsim.yaml      # edit it, for example:
#   gpu: amd/mi300x
#   count: 8
#   rocm_version: 7.2.0
vgpu config show          # every setting in effect, and where each came from

The order of precedence is a command-line flag, then an environment variable, then the settings file, then the default. gpu, count, vram_mb, cuda_version, driver_version, rocm_version and os are the keys that describe the machine.

Give a monitoring agent something to watch

vgpu serve presents a rack without running any kernels, which is what an exporter or a dashboard needs to scrape. Start it in the background and point the tool at it:

build/vgpu serve --gpu nvidia/h100 --count 4 --load 0.9 --alloc-mb 8192 &
build/bin/nvidia-smi
LD_LIBRARY_PATH=build/shim python -c "import pynvml; pynvml.nvmlInit(); print(pynvml.nvmlDeviceGetCount())"

Any CUDA program running against the simulator publishes the same telemetry, so you can also watch a real run from another terminal.

Provoke the errors a health check exists to catch

An uncorrectable ECC error cannot be triggered on demand on real hardware. Here it is one command, and every surface that reports it agrees, so you can test the alert, not just the happy path:

build/vgpu fault inject --gpu 0 --ecc uncorrected
build/vgpu fault inject --pcie replay --count 3
build/vgpu fault throttle --gpu 0 --reason sw_thermal_slowdown --seconds 5
build/vgpu fault show
build/vgpu fault reset --volatile          # a driver reload

nvidia-smi's ECC fields, NVML's ECC and PCIe counters, rocm-smi's RAS blocks and amd-smi's ECC metrics all read these counts. A card without ECC refuses ECC injection, as it has nowhere to count one. vgpu fault can also lose a GPU, degrade a PCIe link or arm a hang.

What is real and what is modelled

This matters if a script makes decisions from the numbers. Memory used and total, utilization (the fraction of time inside kernel launches), kernels launched and bytes moved are exact state of the simulator. Power, temperature, voltage, clocks, fan and performance state are a deterministic model driven by that utilization: power tracks load quickly, temperature lags behind it, and both decay when idle. They make dashboards behave plausibly. They are not predictions of any physical card, and vgpu smi --explain prints this table at the terminal. ECC mode, and the PCIe link, come from the profile, and replay and error counters are zero until you inject some.

Limits

• The stock nvidia-smi will not run against this NVML. NVIDIA's binary gates its startup on an undocumented internal export table, so the simulator ships its own nvidia-smi that renders the same output. A tool that shells out to nvidia-smi or links NVML works; code that requires NVIDIA's own binary does not.
• Not a container or a VM. The filesystem, packages and CPU are the host's. The session simulates the GPU-facing surfaces of a machine.
• dmesg is generated. It is a synthetic kernel log for the session (boot, PCI enumeration, driver bring-up), because there is no kernel driver to log anything.
• No device nodes on AMD. There is no real /dev/kfd or /dev/dri backing the GPUs; programs reach them through the session's HIP and HSA runtimes. Fields with no value, such as VBIOS and serial numbers, are reported as N/A.
• NVML answers what it can. Queries the simulator has no data for return NVML_ERROR_NOT_SUPPORTED, which tools show as N/A. DCGM's own diagnostics have not been run against it: only the NVML half is covered.

Several mock-hardware projects target the same testing need from a different angle, for example in Kubernetes. The comparison page says what each is for.

Questions

Next steps

See the docs on nvidia-smi, NVML and rocm-smi, pick a card from the GPU index, try the commands in the playground, or run the same machine in CI. To run programs rather than tools, see running CUDA without a GPU.