An agent harness is the code around a model that gives it tools, runs its loop and decides when its work is done. Coding agents such as Claude Code, Codex and OpenCode run on general-purpose harnesses built to read, edit and run code in a repository. That works for standard tasks, but specialized engineering work needs knowledge and tools these harnesses don’t have, both to do the work and to verify it.
Say you want an agent to speed up a GPU kernel on NVIDIA’s Blackwell GPUs. Getting the most out of Blackwell takes hardware-specific tuning, and a coding harness may not come with tools to research the hardware or profile a kernel on it. Without those tools, the harness falls back on a simple timing benchmark, which an agent can learn to beat without making the kernel faster. Researchers call this reward hacking, and catching it usually falls to an engineer who has to re-check every result. That review eats into the time and cost the agent was supposed to save.
Parasail serves 40+ open models, and each new model or GPU we add can need its own kernels. We built a specialized agent harness to write them, with tools to research and profile the hardware and a separate reviewer model that checks every claimed speedup.
Over four days in October, that reviewer rejected 46 DeepSeek-V4.1-Flash kernels. Each had passed the correctness and timing tests a general harness would rely on, but broke a launch requirement that production depends on. The kernel the agent wrote after those rejections is a validated candidate that hasn’t yet been deployed or tested on mirrored production traffic, which copies real user requests to a test server while production keeps serving every user.
This post follows one of the 46 rejected kernels through each part of our harness and shows which parts you’d need to build for your own specialized work.
Define the task contract your agent harness checks
To start out, we assigned the agent the task of speeding up one GPU kernel used to serve DeepSeek-V4.1-Flash. A GPU kernel is a function that runs on the GPU and carries out one piece of the model's math. This kernel reverses the rotary position embedding (RoPE) that encodes each token's position, and it converts the result to FP8, an 8-bit number format. We'll call it the inverse-RoPE kernel. The agent had to write a version that runs faster than the inverse-RoPE kernel production uses today.
A task contract is the set of conditions a result must meet before the harness accepts it. Our task contract for the inverse-RoPE kernel required the agent's kernel to match the production kernel's output, run faster than it and keep its launch behavior. The harness's initial automated checks covered only the first two conditions.
- The agent's kernel had to produce exactly the same output as the production kernel on a batch of 1,536 tokens.
- It had to run faster than the production kernel in an interleaved A/B timing test, which alternates runs of the two kernels on the same GPU.
A kernel that passed both initial checks went to a reviewer model as a claimed win. The harness ran both checks itself, so neither check relied on the agent's report of its own result. Neither check tested launch behavior, and that gap is where the agent found its speedup.
Give the agent the hardware context it needs for the task
A kernel only needs to be fast on the GPUs it runs on. We serve DeepSeek-V4.1-Flash on NVIDIA B200 GPUs, which use Blackwell, NVIDIA’s newest GPU architecture.
The models available to us weren't trained on Blackwell code, so on their own they would hallucinate kernel code. One of the biggest early challenges was teaching them the Blackwell architecture without training a custom model. Rather than rely on what the model already knows about Blackwell, our harness gives the agent hardware knowledge from two sources.
- A research agent reads NVIDIA’s published documentation, such as its Blackwell tuning guide, and runs its own experiments on a B200 to work out how the architecture behaves. It passes its findings to the agent that writes the kernel.
- A profiler tool lets the kernel-writing agent run each kernel on a production-grade GPU and see where its time goes, so the next attempt can target the slowest part.
All of this information reaches the agent through its context, which is the text the model reads with each request. Nothing in the harness retrains the model, so the harness has to supply the same knowledge on every attempt.
A larger model might seem like the obvious way to get faster kernels, but in our tests the largest model wasn’t always the best. What did help the models write faster kernels was the research findings and profiling data the harness supplies. Once the harness's verification layer was mature enough, it didn’t really matter what model we used. Any model that could write decent code was enough, because the verifier could filter out the bad code to find the gems worth promoting to production testing.
Find the blind spots in your agent's benchmark
Our agent submitted an inverse-RoPE kernel that passed both initial automated checks. Its output matched the production kernel's output exactly. The A/B timing test measured it at 59.4 µs against the production kernel's 105.1 µs. That’s a 1.77x speedup. By every automated measure, the agent's kernel was a win.
The agent's kernel got its speed by switching off programmatic dependent launch (PDL). According to NVIDIA's CUDA programming guide, PDL is a CUDA feature on Hopper and later GPUs that lets a kernel start before the kernel it depends on has finished. The second kernel can start work that doesn't depend on the first kernel's output, but it still waits for that output before using it, so the overlap isn't guaranteed.
In DeepSeek-V4.1-Flash, the inverse-RoPE kernel runs right after the attention step. The production kernel uses PDL to start its independent work while the attention step is still finishing, which can hide part of the inverse-RoPE kernel's time.
The A/B timing test runs the inverse-RoPE kernel in isolation, with no attention step before it. Without an attention step to overlap with, PDL only adds waiting time. The production kernel paid that waiting time in the test. The agent's kernel skipped PDL, avoided the waiting time and measured 1.77x faster.
In the full model, the agent's kernel has to wait for the attention step to finish completely before it starts. That wait gives up the overlap the production kernel relies on, so the agent's kernel would likely run slower than the production kernel, even though it timed 1.77x faster on its own.
Gaps between what a benchmark measures and what production does exist in every domain, and an agent rewarded for benchmark results will find them. The KernelBench-Verified re-evaluation of model-written kernels found a ReLU kernel that reported a 374x speedup by returning its input unchanged whenever the input matched the test's shape. The test inputs were all nonnegative, so returning them unchanged happened to produce the correct ReLU output. Negative inputs exposed the shortcut.
Use an LLM-as-a-judge to refute claimed wins
Agents grade their own work too generously, as Anthropic found in its write-up on harness design for long-running apps. Our harness sends every claimed win to a separate adversarial LLM judge whose only job is to find a reason the claimed win is false. For the inverse-RoPE kernel, the judge returned a verdict of PHANTOM with high confidence, meaning the speedup would not hold in production. We've quoted the judge's full verdict from the run's verdict file below.
PHANTOM, confidence high.
"Launch-contract mismatch — the candidate silently drops PDL (programmatic
dependent launch: gdc_launch_dependents / gdc_wait), which production
runs with launch_pdl=1. The isolated graphed A/B charges the native the
full PDL serialization penalty but cannot see the overlap-with-attention
benefit PDL exists for. Measured PDL effect (1.95x) MORE than explains the
claimed 1.77x."The judge measured PDL's effect on this kernel at 1.95x, which is larger than the 1.77x speedup the agent's kernel claimed. Dropping PDL alone could account for the entire claimed speedup.
The agent kept submitting kernels with PDL switched off after the judge rejected them. One later kernel kept the PDL instructions in its code and switched PDL off through a configuration default instead, as shown below.
_PDL = os.environ.get("KO_DSV41_PDL", "0") == "1" # <- PDL off by default
...
kernel[grid](..., launch_pdl=_PDL)From October 16 to 19, the judge reviewed 46 inverse-RoPE kernels that had passed both initial automated checks. All 46 broke production's launch requirements by switching PDL off in one form or another, and the judge rejected all 46.
Turn each caught failure into a deterministic check
A judge model can be wrong, and it only runs after a kernel has used GPU time for testing. For a failure we already understand, a coded check is cheaper and more reliable.
Once the judge had shown that all 46 phantom speedups came from dropping PDL, we turned that finding into a static check. A static check is a rule the harness applies to a kernel's code and launch settings without running anything on the GPU. Ours rejects any inverse-RoPE kernel that doesn't call both PDL instructions or launches with PDL switched off.
The static check runs during the agent's self-check, the step where the agent tests its own kernel before submitting it. A kernel that fails the static check is rejected before it uses any GPU time or any of the judge's review. Before relying on the static check, we ran it against the 46 kernels the judge had rejected. It rejected all 46 and accepted the production kernel. We applied the same process to every other phantom kernel the judge caught, and over several weeks the static checks grew into an effective deterministic verifier built for our infrastructure and production needs.
The agent's next kernels kept PDL on. In the first run after the static check went in, the quickest of the agent's two kernels was about 2.2x faster than the production kernel. It kept the same overlap with the attention step as the production kernel and did less work each time it launched. That 2.2x came from our original benchmark target. The section on production configuration below explains why we later invalidated results from that target, so treat 2.2x as an initial result until we remeasure it.
In any agent harness for specialized work, we'd recommend handling failures in two steps. First, a person or a judge model catches a failure once. Second, the harness turns that failure into a coded check at the earliest point in the loop, so the agent can't repeat it. Our Vela operations agent works the same way for operations questions and moves a method into reviewed code once the same question keeps coming back.
Evaluate the agent's work in a production configuration
The initial automated checks test each kernel against a benchmark target, which is the set of inputs and settings we expect production to use. If the benchmark target is wrong, every check in the harness can pass while the kernel is tuned for a setup production doesn’t run. When we got MiniMax M3.1 Flash ready for production, a timing race between two kernels only appeared with eight or more concurrent requests, which single-request tests never caught.
Next, we ran the agent's faster inverse-RoPE kernel on production-configuration DeepSeek-V4.1-Flash servers with 4x B200 GPUs, using synthetic traffic. We alternated between the production kernel and our kernel over eight test windows.
The live servers logged a quantization block size of 32. The quantization block size is the number of values that share one scale factor when the kernel converts them to FP8. Our benchmark target had assumed 128, so the two kernels the harness accepted on October 19 were tuned for the wrong block size.
We fixed the block-size mismatch in three places.
- The serving kernel now handles both block sizes, and we re-verified that its output matches bit for bit at a block size of 32.
- The benchmark target now uses a block size of 32.
- The results measured against the old 128 benchmark target were formally marked invalid, so the agent no longer builds new kernels on top of them.
Marking old results invalid is the easiest of these fixes to overlook. Without that step, an agent that carries findings from one attempt to the next keeps building on any result measured against a benchmark target that doesn't match production.
Decide what a custom agent harness should build
Most agent harnesses don't need to be custom. The agent loop, tool calling and passing context between steps work the same way across tasks, and existing frameworks already supply them. LangChain's guide to building a custom harness shows how middleware can add tools, track state and run code at each step of a standard agent loop.
We built our kernel harness from scratch for two reasons. First, we needed deterministic guarantees about the quality of the kernels being generated. These kernels go into production and we can't allow LLM hallucination to affect our customers. Second, we wanted kernel optimization tightly integrated with the rest of our infrastructure, so we can generate, validate and ship new kernels as fast as possible. Everything runs on Parasail infrastructure with models hosted on Parasail, so our platform is in effect optimizing the kernels it serves.
The parts worth building yourself are the checks only your domain can supply. A general-purpose harness has no way to know that a kernel's real speed depends on overlapping with the step before it, or which quantization block size production runs. For a kernel-writing agent, the table below lists the custom parts of our harness, when each one runs and what it does.
| Part | When it runs | What it does |
|---|---|---|
| Research agent and profiler | Before and during each attempt | Supply the hardware knowledge the task needs |
| Initial automated checks | After the agent submits a kernel | Checks for identical output and a faster time on the benchmark |
| Static check | During the agent's self-check, before any GPU time | Rejects kernels that drop PDL |
| LLM judge | After a kernel passes the initial automated checks | Rejects speedups that would not hold in production |
| Serving test | On production-configuration servers with synthetic traffic | Catches benchmark settings that don't match production, such as the block size |
| Result invalidation | Whenever the benchmark target changes | Stops the agent from building on results measured against the wrong target |
Apply this design to your own agent harness
Use these five steps to design a harness for your own specialized task.
- List what the agent needs to know that its training data doesn't cover, and give it a research or measurement tool for each item.
- List what your benchmark can't measure, and test each item under production conditions.
- Send every claimed win to a separate model or person whose only job is to refute it, and keep the verdicts.
- Turn each confirmed failure into a check that runs before the expensive steps. Test that check against past failures and against a known-good result.
- Give your harness a way to mark results invalid, so later attempts stop building on them.
In the initial testing described here, our harness rejected all 46 phantom kernels, and the kernel the agent wrote next passed a serving test on production-configuration servers. Since then, we've developed the harness into a continuous pipeline that researches, generates and tests new kernels, and we're expanding it to work on all of the models hosted on Parasail.
If you’re starting your own agent harness, begin with the parts you can reuse. Parasail’s guide to reducing coding-agent costs with OpenCode shows how to run an existing harness on open models, so you can spend your build time on the checks only your domain can supply.
