We poked NVIDIA's GSP with eBPF to change gpu scheduling, here's what the hardware actually did

We started with a simple question, can a kernel-level eBPF policy change how NVIDIA’s gpu schedules competing workloads? The machine was a dgx spark with a GB10 chip & driver 580.173.02, with GSP firmware.

On modern nvidia gpus (Turing and later), most of the Resource Manager runs on the gpu itself instead of the host cpu. It runs as firmware on the GSP, the GPU System Processor, a small RISC-V core built into the chip. The host driver packages each request as a remote procedure call (RPC), sends it to the GSP, and waits for a reply. The GSP then talks to the GPU's scheduling hardware.

It handles privileged gpu management tasks such as resource management, context switching, power, memory, and scheduling commands.

A TSG, or Timeslice Group, is the nvidia driver’s scheduling object for a group of cuda channels. It is the unit that the driver and GSP present to the gpu scheduler to move around or rotate.

What gets scheduled?#

A gpu runs work from many processes at once. Each process submits work through channels. A channel is a queue of commands that the cpu fills and the gpu drains. Channels that belong to the same cuda context are bundled into a TSG. When the gpu switches from one process's work to another's, it is switching TSGs.

The hardware keeps a runlist, a list of TSGs waiting for an engine, and cycles through it.

We started with two scheduling controls:

• timeslice: the requested duration for which a TSG may hold an engine.
• interleave: how frequently that TSG appears in the firmware’s scheduling rotation, its basically a weight.

How the driver names its commands?#

Every command we sent had a name like this:

NVA06C_CTRL_CMD_SET_TIMESLICE

The Resource Manager organises gpu objects into classes, and each class has a hexadecimal number. A06C is the channel-group class, the object behind a TSG. It dates from the Kepler generation and is still in use. Every command for that class shares the prefix, and the final digits select the operation. For example, 0105 preempts and 0104 reads the timeslice back.

First experiment: changing the timeslice#

We patched nvidia.ko to find the target graphics TSG and send SET_TIMESLICE through the Resource Manager to the GSP. The GSP replied with success. When we queried the driver afterward, it reported the value we had requested.

To see what the gpu actually did, we used NVIDIA Nsight Systems. It traces gpu execution directly, independent of anything the driver reports. The number we watched was context residency: how long a TSG stayed on the engine before being switched out.

We compared two requests 32 times apart:

As you can see residency barely moved, and switching between low and high interleave levels had no effect either: the two workloads kept splitting GPU time almost exactly evenly, a sharing ratio of about 1.0.

A success reply from the GSP means the firmware received and processed the command but it carries no information about whether a runlist entry, a scheduling quantum or a hardware register changed. Reading the value back proves even less, since the driver may just return the number it saved when the request came in.

So we dove into the driver source, and it explained our case. On our path, the timeslice request reached this function:

The _56cd7a suffix marks one of the driver's auto-generated stub implementations. This one returns NV_OK, the driver's success code, and writes no scheduler state. So the callers see a normal, working interface, but the hardware never hears about the request.

Second experiment: preemption#

We moved to a more direct control, preemption. It forces a running TSG off the engine immediately, without waiting for its timeslice to end. NVIDIA already defines the command:

NVA06C_CTRL_CMD_PREEMPT

eBPF programs can only call kernel functions that are explicitly exposed to them, known as kfuncs so we added one:

hClient and hTsg are Resource Manager handles for the calling client and the target TSG. The call travels this route:

Most eBPF code must run to completion without pausing. This function has to wait, both for Resource Manager locks and for the GSP's reply. So we registered it as sleepable, which limits it to eBPF program types that are allowed to block.

So what did we learn from the preemption run?#

Our eBPF program counted every preemption call it made and checked the result each one returned. Across both the short mechanism run and the longer performance runs, every call succeeded as below

Each "arm" is one version of the experiment: one run with preemption switched on, and one baseline run with it switched off. To see whether those calls actually changed anything on the GPU, we turned to Nsight again. It counts context switches, the moments when the GPU stops one TSG's work and starts another's. With preemption active, it recorded noticeably more of them:

That works out to 46 extra switches, i.e an increase of about 19%. This was the result the timeslice experiment never produced. The preemption request made it all the way through the driver and firmware, reached the scheduler, and changed how the GPU moved between workloads.

The change came at a price which was expected. The median latency climbed from about 171 µs to about 3012 µs, roughly 18x slower. Every forced preemption interrupts work mid-flight, and the GPU spends that time saving one context and restoring another.

This experiment turned one question into three interesting questions:

• Did the eBPF program run?
• Did the Resource Manager and GSP accept the command?
• Did the gpu hardware do what we asked?

Kernel logs will tell you whether the eBPF program ran, and the firmware's response will tell you whether the command was accepted. For the third question, you have to stop asking the software and watch the gpu directly, with something like Nsight.

Where this leaves us?#

eBPF policies running in the kernel can call into NVIDIA's driver and trigger real gpu scheduling commands but whether a given command actually changes anything is another matter. We observed that preemption did while timeslice came back with a success code and the gpu carried on exactly as before. Interesting part is that, looking from the kernel, both calls seemed to work fine.

More to come! follow us @latentone_