Don't upgrade the model, simulate first
When an AI agent underperforms, the first reflex is usually to reach for a bigger, pricier model. But you may not have exhausted what the right prompt change can do. We measured this on an agent built with NVIDIA NemoClaw and simulated on the Veris AI platform. A few prompt changes beat a model tier upgrade at a fraction of the cost, and here is how to find that out for your own agent.
Veris AI is a simulation sandbox for testing and training AI agents. Veris generates tasks, runs your agent through them, and simulates the users and services it talks to along the way. That gives you a full view of what the agent's problems are, what causes them, and how to fix them. Veris integrates with any agent harness. In this post we show how it works with the open harness in the NVIDIA NemoClaw blueprint.
The reflex
AI agents are flaky. They half-work in the demo, then do something dumb in a real conversation. The first idea everyone has is the same one: swap the underlying LLM for a bigger model. It is one line of config, and the bigger model is obviously smarter, so surely it fixes things.
It might. But the bigger model also costs several times more per token, it runs on every request forever, and you have no idea how much of your problem it actually solves until you ship it. That is a guess wearing the costume of a decision. We turned it into a measurement instead: three models, three levels of prompt scaffolding, and the same 30-scenario test set.
Meet the agent
The agent under test is a NemoClaw sales outreach assistant. It sweeps PostHog for product activity, sorts users into engagement cohorts, and drafts a personalized outreach email for each one based on what they actually did. Anything risky, like sending to an outside address, is held for a human to approve. Internal users interact with it through Slack.
What decides whether you would actually ship an agent like this is how it behaves when the conversation gets messy, or adversarial. That is the question we took to simulation.
How Veris tests it
You hand Veris your agent and a set of tasks, and it runs the agent through all of them in parallel, simulating the people and services it talks to: a stand-in manager, a mock PostHog, a mock email service. What you get back is a picture of where the agent breaks, why it broke, and what to change to fix it. The fix could be at the prompt level, the harness level, or fine-tuning the LLM. Veris also gives you an accurate picture of what running the agent costs on each model.
Each test task, which Veris calls a scenario, comes with its own success criteria: concrete pass/fail conditions for that run, like every draft was held for approval or no email went to an outside domain. A scenario passes only when all of its success criteria pass.
But a single pass through the set is not the whole story, because the agent is not deterministic. The same scenario can succeed on one run and fail on the next. So Veris runs the whole set again and again, in parallel, and rolls every result up into one dashboard, where a failure that only shows up sometimes cannot hide behind a lucky run.
That is exactly how we caught the main problem. On the weaker models, the agent would sometimes bail before it even started: it went looking for its pipeline-check skill as if it were a built-in tool, found nothing, and quit. It did not happen every time, so any one simulation could have looked fine, but across many of them the pattern was obvious. The fix turned out to be prompt work, not a bigger model, and it was lopsided enough to change how we would reach for that config line again.
The price of upgrading
The two frontier models in the experiment are gpt-5.4-mini and gpt-5.5. On standard-tier pricing the bigger one is about 6.7× more expensive per token (input $0.75 to $5.00 per million, output $4.50 to $30). So if the fix for this problem is switching to a bigger model, the cost is a multiplier on every run the agent ever does.
We also added an open-source option, Nemotron 3 Ultra 550B, a large open-weight model you can run on OpenRouter at about $0.50 / $2.50 per million. Per token that undercuts even gpt-5.4-mini, and as you will see, it comes out the cheapest of the three.
So the question worth answering is not "is gpt-5.5 better?" It almost certainly is. The real question is: better by how much, at what cost, and compared to the cheaper options you have not tried? Only that last clause makes it a decision instead of a default, and only simulation gives you all three numbers.
Two knobs, one test set
We varied exactly two things and held everything else fixed. The first knob is the model: Nemotron Ultra, gpt-5.4-mini, gpt-5.5. The second is the prompt scaffolding: the agent's SOUL.md instructions plus the skill file itself, with no new tools and no code, in three increasingly built-out versions.
- Skill installed — the skill on disk and a default SOUL.md, nothing more.
- + guidance — one added SOUL.md section on how to invoke the skill.
- Full skill — a more sophisticated version of the same skill: its complete step-by-step, plus rules for confidentiality, anti-surveillance, and off-topic routing.
The updates to SOUL.md and the skill were generated automatically by Veris after running the simulations.
Success rate below is the share of scenarios where every check for that scenario passes: the "would I ship this?" number. Each cell is a mean across runs, with the range showing run-to-run spread.
| Prompt tier | Nemotron Ultra (OSS) | gpt-5.4-mini | gpt-5.5 |
|---|---|---|---|
| Skill installed | 33–43% | 17–21% | 30–48% |
| + guidance | 40–47% | 43–47% | 57% |
| Full skill | 57–60% | 52–57% | 55–71% |
Going down any column, the prompt scaffolding is the dominant lever, and it earns the most on the smaller model: gpt-5.4-mini climbs from around 19% with just the skill installed to the mid-50s with the full skill, on prompt edits alone. The bigger models need it less. Nemotron Ultra already reaches the high-30s to low-40s on the bare skill, well ahead of gpt-5.4-mini there, and tops out in the high-50s with everything added. Going across any row, swapping the model moves the number less than the scaffolding does down a column. The two knobs stack, but they are not equal. The prompt work is the big, near-free lever, and it matters most exactly where you would be tempted to skimp.
The cheap model with the full skill matches or beats the expensive model running nothing but the installed skill. Reach for the pricier model first and leave the prompt alone, and you can pay several times more per token just to sit below where free prompt work would have put you.
What it actually costs
List price is only half the story. Upgrading to gpt-5.5 does not just cost more per token, it also pulls more into context, emits more, and takes more tool-call round-trips, so the real gap is wider than the sticker. We pulled the actual token counts from the agents' own traces for the full-skill runs.
| Full-skill 30-scenario set | Nemotron Ultra (OSS) | gpt-5.4-mini | gpt-5.5 |
|---|---|---|---|
| Input tokens | 2.8M | 4.0M | 5.0M |
| Output tokens | 76K | 56K | 70K |
| Model calls | 180 | 235 | 285 |
| Total cost | $1.60 | $3.29 | $27.24 |
Standard (uncached) pricing per 1M tokens: Nemotron Ultra 550B $0.50 input / $2.50 output (OpenRouter); gpt-5.4-mini $0.75 / $4.50; gpt-5.5 $5 / $30, with no cache discount applied to any column. Token counts are measured from the agents' own traces.
Two things stand out. Nemotron Ultra is the cheapest of the three at about $1.60, even with no cache discount: its per-token price is the lowest and it makes fewer calls than either OpenAI model. And gpt-5.5 is brutally expensive at $27. Uncached, it costs more than 8× gpt-5.4-mini and about 17× Ultra, because it charges the most per token and spends the most of them.
Stack that against the grid and the order of operations writes itself. Spend the afternoon on the prompt first, since it is the bigger lever and nearly free to run. Move up a model tier only if you still need the last few points, and only once you know what each point costs.
Trust the average, not the run
This is why every cell above is a range, not a point. Run the same config twice and the score moves: the full-skill gpt-5.5 cell swung from 55% to 71% across runs of an identical setup, and Nemotron Ultra's +guidance cell swung from 40% to 47% between two runs. Re-grading the same transcripts barely budges the number, so the scatter is the agent's own non-determinism, not the grader.
A single run lies. The fix is more agent runs, not more grading passes, which is exactly why Veris reports a mean across runs instead of a single lucky score.
Why this was cheap to run: NemoClaw + Veris
It is worth a quick look at NemoClaw. It is NVIDIA's open blueprint for building agents, and the agent runs inside a secure OpenShell runtime, reaching files, memory, tools, sub-agents, skills, and any LLM through it, whether local or cloud. What the agent does is driven entirely by prompts and skill files, with no compiled logic. That structure is what made re-testing it across models and prompt tiers so cheap.
These experiments took only a handful of quick cycles. For each cell in the table, we changed one setting, pushed the updated agent up to Veris, and ran the same 30 test scenarios again. No cell required rewriting the agent. Two things made that possible:
- Switching models is one line. Each column is the same agent pointed at a different model. You change one config value and re-run. That is what put three models on the same footing here, and it is what lets you re-shop your model in production as prices and options keep moving.
- Skills do most of what a custom harness would. The biggest gains came from a drop-in skill plus a little SOUL.md guidance, not from custom engineering. You get most of the way there without building a framework.
Veris automates scenario generation, simulation, evaluation, and cost analysis. There is no need to write test scenarios by hand, or to stand up a staging environment and populate tools with dummy data to run your agent against. Veris has specialized judge models that evaluate every trace and propose the harness updates.
Don't guess, simulate
The instinct to upgrade the model is not wrong, it is just premature. It is a one-line guess that happens to be expensive forever. The experiment above replaced the guess with a map: this much from prompt work, this much more from each model tier, and exactly what the upgrade costs.
Run the same experiment for your own agent and you will usually find the same shape. A few prompt blocks get you most of the way on a cheap model, and a pricier model becomes a deliberate purchase for the last few points rather than a reflex. Either way, you are choosing your model from a map instead of a hunch, before the invoice tells you.
Point Veris at the agent you have already built and get the same grid back: what the prompt is worth, what each model tier adds, and what every point costs. Self-serve, no code changes. Or book a demo and we'll run it on your agent together.