Why can't two gpu profilers watch the same process?

Most people who profile gpu code expect to stack their tools, one for timing and one for checking memory, the same way you can pile several kinds of logging onto a normal program. On NVIDIA gpus there is one small seat inside the driver that makes this harder than it sounds. CUPTI is NVIDIA's profiling interface, the layer that tells you when each kernel started and ended, and Compute Sanitizer is NVIDIA's memory-checking tool, which can also patch a kernel so it reports which blocks ran where. Both tools ask for that same seat, and the driver hands it to only one of them at a time.

Starting with the why#

Each tool on its own only gives you half of what's going on. CUPTI tells you when every kernel started and ended, but it can't see inside a kernel to tell you which thread blocks ran or in what order. Our Compute Sanitizer probe gives you exactly that block-by-block view, but only for the one kernel we patch, and it doesn't give you the timing of everything around it.

We wanted both views on the same request. Timing alone tells you a step is slow. It doesn't tell you why. If a cache kernel takes twice as long on one step, the kernel times can't show whether fewer blocks did the work, the blocks ran in a different order, or one block held everything up. The block data can show that, but only if it comes from the same launch you timed. Without that, you're matching two separate stories and guessing they line up. So we started by trying to run both tools in one process, since that's the only setup where they're guaranteed to see the same launch.

Short version#

• Four small tests tried to run CUPTI and Compute Sanitizer together in one process, in every order we could think of, and each one lost data from one tool or the other.
• The error that kept coming back was CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED, and NVIDIA's docs say that's what you get when something else already holds the CUPTI seat.
• Giving the seat back with sanitizerUnsubscribe() or cuptiFinalize() didn't free it up, even when cuptiFinalize itself returned CUPTI_SUCCESS.
• So the live setup now runs two vLLM processes, one with CUPTI and one with Compute Sanitizer, and every chat request is sent to both.
• The CUPTI side alone followed 8 requests through 35 decode steps and 1,400 kernel intervals with 0 drops, though the two processes only line up roughly in time.

What we wanted#

Two earlier posts each looked at one side of a live vLLM request on our DGX Spark. The "where the time goes" piece used CUPTI to follow a request from the client down to real kernel start and end times, and the Compute Sanitizer post patched production kernels so each thread block reported itself, which gave us which block ran and in what order.

Each view tells you something different, because CUPTI times every kind of kernel while our sanitizer probe only watches the one kernel we patch it for, picked by matching one piece of its name. So the next step was to put both on the same request in the same process, and that's what this article is about.

What we tried#

Before risking the Qwen3-14B server, we ran four small coexistence tests on tiny CUDA test programs. Each one changed which tool signed up first, or tried to hand the seat over partway through, because the order seemed like the thing most likely to matter.

What failed and how#

The first test ran Compute Sanitizer next to Nsight, which uses CUPTI underneath, and the block events from our patched kernel did show up, but Nsight wouldn't start its own collection and printed CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED.

Starting the sanitizer first and then registering a CUPTI agent that asked only for activity records got result 39 when it signed up, which is just the number for that same CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED error, so no CUPTI activity came back at all. Flipping the order just moved the problem to the other side, CUPTI caught the kernels and the sanitizer subscriber gave us no events.

The last idea was a handoff, where one tool would finish, give the seat back, and let the other take it. In the run with cuptiFinalize(), the call returned CUPTI_SUCCESS and the very next CUPTI registration still got CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED, and sanitizerUnsubscribe() didn't free the seat either. On top of that, we couldn't read the sanitizer's device buffers back with a normal cuMemcpyDtoH, it just failed with "invalid argument".

NVIDIA's CUPTI documentation describes this rule for cuptiSubscribe: "Only a single subscriber can be registered at a time." It also says the same error is returned when an application is launched under NVIDIA tools such as Nsight Systems, Nsight Compute, cuda-gdb and cuda-memcheck. So our notes treat this as a limit of the stack we tested, and we took the handoff code out of the production source.

What we did instead#

Giving each tool its own process turned out to be the simplest way through, so the live demo now runs two vLLM containers side by side on the same gpu. The primary runs CUPTI, times every kernel launch and sends the reply the user sees, while the shadow runs Compute Sanitizer and reports per-block data for reshape_and_cache_flash_kernel, and we throw its reply away.

The browser sends every chat message to both servers at the same moment, so the two processes see mirrored copies of the same prompt. Fitting both on one Spark needed --gpu-memory-utilization 0.4 on the shadow, because by default vLLM sizes its memory grab against the whole device, so it just failed while the first container was already loaded.

On the CUPTI side, the EXP-0020 run followed 8 concurrent requests through 35 selected decode steps, joining each authoritative packed row to a runtime correlation ID and then to the matching kernel's real start and end time. That produced 1,400 intervals for the cache kernel, 40 per step, with 0 correlation misses and 0 buffer drops, and the run collected no sanitizer block events, since that's exactly why we split the processes.

What we can't say yet#

The two servers are matched by time, not by launch. That's enough to say a block event and a kernel came from the same prompt, but not that they came from the same kernel launch. Getting an exact match needs something both processes share, like a launch ID, and two separate processes don't have one. That's the next thing we want to build.

Each server also schedules and batches on its own, so the shadow's packing can drift from the primary's, and we haven't measured how far. We also don't know yet how much running two 14B servers on one gpu slows either reply down.

All of this comes from one DGX Spark, CUDA 13.2.1, one patched kernel family and a handful of small tests, which leaves a few questions open. A newer CUPTI or sanitizer release might allow a real handoff, a different way of releasing the seat might work where ours did not, and the mirrored processes might diverge enough under heavy load to blur the picture.

So on this stack, if you want CUPTI timing and Compute Sanitizer block data for the same request, the way that worked for us was to run each tool in its own process and send both of them the same request.