Most Agent 'Skills' Barely Help. NVIDIA Now Has a Way to Tell Which Ones Do
Quick verdict
Everyone is shipping "skills" for their coding agents now, reusable instruction files that supposedly make Claude or a local model better at a task. NVIDIA's new evaluation work says most of the ways people judge those skills are close to worthless. When they scored skills by scanning their structure, the checklist approach almost everyone uses, the scores predicted actual usefulness at a Spearman correlation of just 0.14. That is barely above random. Their proposed replacement, Skill Lift, is simple: run the same task twice, once with the skill and once without, under identical conditions, and measure how much more work got finished. If the delta is zero, the skill is decoration. It is a small idea with a large consequence, because it means a lot of the skill libraries being passed around right now have never been tested against the only thing that matters.
What the research actually says
The finding comes out of NVIDIA evaluation research summarized by Elvis Saravia. The core claim is that structural checks on a skill, does it have the right sections, is it formatted cleanly, does it list steps, barely track with whether the skill helps an agent do the job. The correlation between those scan scores and human-judged quality came in at 0.14.
So they flipped the measurement around. Instead of grading the skill's paperwork, Skill Lift grades its effect:
- Take a real task the agent is supposed to do.
- Run it once with the skill loaded and once without, keeping everything else the same.
- Score the difference in completed work. That number is the skill's lift.
A skill that reads beautifully and adds nothing scores zero. A terse, ugly skill that unblocks the agent scores high. The method costs more than a static scan because you have to actually run the tasks, but it measures the real variable rather than a proxy for it.
This lands next to a second thread from the same week. A position paper on Anthropic-style harnesses, summarized by DAIR.AI, argues enterprises should standardize on one reusable coding-agent harness rather than build a new orchestration graph for every project, and that the harness choice can matter more than the model choice on real work. Put the two together and the message is consistent: the scaffolding around the model, the skills, the harness, the tool loop, is now where a lot of the quality lives, and almost nobody is measuring it properly.
The evidence showed up on Reddit the same week
You did not need a paper to see this. In a widely shared r/LocalLLM thread, someone who had declared Qwen 3.8 27B "not Opus level" re-ran the same task under a different harness and reversed himself. Same model, same prompt. Under one setup it produced a black screen and failed. Under an agentic harness with execution and screenshot feedback, it built a working ocean renderer, and even wrote its own PNG decoder when vision was not wired up. The model did not change between runs. The scaffolding did, and that alone moved the result from failure to success.
That is Skill Lift in the wild, minus the formal scoring. The gap between "this model is bad" and "this model is great" turned out to be a property of the harness, not the weights. If your evaluation only looks at the model, or only reads the skill file, you miss the thing that actually decided the outcome.
Why it matters for anyone using AI to code
Skill marketplaces and shared prompt libraries are filling up fast, and the instinct is to judge them by how polished they look. This research says stop doing that. The only honest test is whether a skill changes what the agent finishes, and you can run a cheap version of it yourself: do the task with the skill, do it without, and see if there is a real difference. If there is not, delete it. A leaner skill set that each earn their place beats a bloated one where half the entries are dead weight the agent still has to read.
It also reframes how to shop for a coding agent. Benchmark charts rank models. They rarely tell you how good the harness around the model is, and that harness is doing more of the work than the leaderboard admits. We made the broader version of this case in why the coding agent harness now matters more than the model, and the Skill Lift work is the measurement side of the same story. When you compare tools, weigh the whole loop, the execution feedback, the tool access, the retry behavior, not just which model sits underneath. Our rundown of the best AI coding agents and the best free coding agents both lean on that harness quality, because it is what separates a demo from something you can actually ship with.
There is a cost angle too. The same issue that surfaced this research also showed persistent, always-thinking agent setups like Headlong running background reasoning at a dollar or two an hour. If you are going to pay for scaffolding that runs continuously, you want to know it produces lift, not just activity. Skill Lift is the discipline that keeps you from paying for motion that never moves the task forward.
Video: how to evaluate and test agent skills
A practical walkthrough of testing whether an agent skill helps before you trust it, which is exactly the habit the Skill Lift research is arguing for.
FAQ
What is Skill Lift?
It is a way to measure whether an agent skill actually helps. You run the same task with the skill and without it under identical conditions, then score the difference in completed work. That difference is the skill's lift. A skill with zero lift is not helping, no matter how well it is written.
Why can't you just read the skill to judge it?
Because reading it barely predicts anything. NVIDIA's work found that structural scan scores correlated with real usefulness at a Spearman value of 0.14, close to random. A skill can look complete and organized and still do nothing for the agent, or look messy and be the thing that unblocks it.
Does this only apply to enterprise setups?
No. The method is cheaper and more useful the smaller you are, because you can run the with-and-without test on your own tasks in a few minutes and prune skills that add nothing. It scales up to teams standardizing on one harness, but a solo developer benefits the same day.
How does this connect to picking a coding agent?
It says judge the whole harness, not just the model. Execution feedback, tool access, and retry behavior often decide the outcome more than which model is underneath, which is why the same model can fail under one agent and succeed under another. See our coding agent comparison for how those differences play out.
Sources
- @omarsar0 - NVIDIA Skill Lift research summary, skill scan scores correlate with usefulness at Spearman 0.14
- @dair_ai - position paper on standardizing a single reusable coding-agent harness
- r/LocalLLM - "Qwen 3.8 isn't Opus level": I re-ran the test under a different harness
- @andykonwinski - Headlong, a persistent agent harness running background reasoning at $1 to $2 an hour
Further reading
Try all the models mentioned in this article
Admix gives you GPT-5, Claude, Gemini, and 350+ AI models in one app. Compare responses side by side. Free to start.
Start free on Admix