Prose Weight Visualiser: prompt linter

Every team I know now has a CLAUDE.md, a system prompt, or an agent instruction file that's grown by accretion. A line added here after a bad run, a paragraph pasted in from someone's blog post. Nobody can tell you which lines still do anything. We're writing the most consequential instructions of our working lives in a language we never profile.

That last word's the one I keep coming back to. Years ago I tuned inner loops with Intel's VTune, counting cycles, because you didn't guess at performance. You measured it. Then the tooling got good enough that most of us stopped. Now the highest-leverage thing I write on a given day is English prose aimed at a model, and I've got no profiler for it. That struck me as an omission worth fixing.

So it's a measurement instrument for prompts. Paste a system prompt and it tells you which instructions carry behavioural weight and which are ballast, measured against a local model rather than argued from gut feel. Three lenses:

• attnscope: attention over the prompt tokens, so you can see where the model actually looks;
• attnduel: two phrasings of the same instruction competing, to settle which one the model actually obeys;
• behavioural A/B: the same task across phrasings, measuring the delta in what the model does rather than what it says.

The linter's the headline. The heatmaps are the workings.

The honest objection

Attention isn't causation, and I'd rather say so myself than wait for someone else to. A token can be attended to and change nothing. It can be ignored and still matter through the residual stream. A tool that drew pretty heatmaps and called them explanations would be actively misleading, and I've already shown that. Blindly.

So the weight score never rests on attention alone. It pairs attention with ablation, meaning you pull the line and measure what changes, then repeats the run and reports its own uncertainty instead of hiding it. Every score arrives with a credible interval, and a line whose interval can't be told apart from noise gets marked as noise rather than dressed up as a low number.

What the numbers actually said

Here's the part where I have to eat something. My hunch was that a decent slice of a typical prompt does nothing measurable, maybe more than half of it. So I built the thing, pointed it at a small prompt with a deliberate mix of concrete rules and vague good intentions, and ran it against a local 7B model across the full probe suite. This is the actual readout, not a mock-up:

Prose Weight · Readout   model qwen2.5:7b-instruct   suite v1   deep audit
0% of this prompt is below the noise floor.

  WGT  weight 0-100         95% CI     VERDICT       INSTRUCTION
  100  ██████████████████   [85,100]   load-bearing  Return only valid JSON, no prose
   61  ███████████·······   [37, 82]   contributing  Be genuinely helpful and thorough
   60  ███████████·······   [41, 81]   contributing  Never give a specific dosage
   52  █████████·········   [30, 73]   contributing  Quality and craftsmanship matter
   44  ████████··········   [28, 63]   contributing  Attempt the task directly
   40  ███████···········   [23, 56]   contributing  Keep the user's best interests at heart

The concrete, checkable instruction won by a country mile. "Return only valid JSON, no prose" scored a flat 100: pull it and the output changes completely, every run. Nothing else came close, and you can read the gap straight off the bars.

But here's the bit that cost me the bet. Nothing was dead. The vaguest line in the whole prompt, "keep the user's best interests at heart", the sort of thing I'd have put money on being pure ballast, still scored 40. Ablate it and the answers genuinely shift, just far less than the JSON rule does. Every aspirational platitude in the middle landed somewhere between 40 and 60, and not one of them fell to the noise floor.

So the headline I wanted, "half your prompt does nothing", didn't survive its own instrument. On these models, vague isn't nothing. It's weak, and it's far less efficient than a specific rule, but the words still land somewhere. Blunt and specific beats polite and vague by a mile. It just doesn't beat it all the way to zero.

That's a more interesting answer than the one I set out to prove, which is the entire argument for measuring instead of asserting.

Where it is now

It's built and it runs. Locally, against whatever models Ollama has pulled, with a CLI and a web readout that carry the same numbers, and you pick the subject and judge from a dropdown. You can also point the subject at a frontier model instead, Opus or Sonnet or one of the smaller ones, when you want to profile the strongest thing going rather than a 2B on your own box. Those don't expose their attention and aren't reproducible, so those runs come marked best-effort, and only the subject's calls cost anything because the judge stays local. The near-term work is calibration. First, widening the model roster, because a 2B model finds even more of your prompt load-bearing, small models flinch at any change at all. Then tightening the probe suite so every instruction has something that genuinely tests it. And there's a real open question about how large a prompt has to get before honest dead weight appears. The receipts go up here as they land, including the ones that argue against me, but I'm expecting Fable level models to have my back ;)

MEASURE YOUR PROMPTS LIKE YOU USED TO MEASURE YOUR INNER LOOPS.