What’s new: Claude Sonnet 5.5 improved verdict accuracy and supplied more accepted proof than Claude Sonnet 5 on our 48-case security investigation benchmark. Both ran at high reasoning. The new pass cost $21.57, down from $37.13.
Why it matters: A suspicious command on a work laptop might be an intrusion or authorized administration. An analyst needs a defensible answer and the records that establish it. A model that reaches more correct conclusions at a lower measured cost is useful to test further; its supporting evidence still needs scrutiny.
The verdict and proof improvements
Verdict accuracy asks whether the final classification is right. Evidence-backed accuracy also requires the accepted supporting records, retrieved and cited by the agent. These measures answer different questions.
| Model | Reasoning | Correct verdict | Verdict with proof |
|---|---|---|---|
| Sonnet 5 | High | 58.3%21 / 36 | 0.0%0 / 36 |
| Sonnet 5.5 | High | 91.7%33 / 36 | 30.6%11 / 36 |
| Model | Reasoning | Correct verdict | Verdict with proof |
|---|---|---|---|
| Sonnet 5 | High | 66.7%8 / 12 | 0.0%0 / 12 |
| Sonnet 5.5 | High | 83.3%10 / 12 | 33.3%4 / 12 |
The clearest gain: Sonnet 5.5, at high reasoning, got 33 of 36 basic verdicts right, compared with 21 for Sonnet 5 at high reasoning. It met the proof requirement in 11 basic cases and four intrusion cases. The earlier pass met it in none.
The remaining gap: Most correct verdicts still did not satisfy the proof requirement. That does not establish that every unsupported answer was a guess. It means the submitted evidence did not meet this benchmark’s accepted sets.
Cost fell and every workflow completed
Both models completed all 48 workflows. Sonnet 5.5’s high-reasoning pass cost $12.52 for 36 basic cases and $9.05 for 12 intrusion cases. Together, that is 41.9% less than the prior Sonnet 5 high-reasoning pass.
These are measured charges for this task and tool workflow, not an estimate of what another team will pay. Completing the workflow means submitting the required checkpoints and final report; it does not mean every answer was correct.
What this comparison supports
The practical takeaway: In this run, the upgrade improved both classification and evidence-backed verdicts while reducing cost. Teams testing the same kind of work have a concrete reason to compare it in their own environment.
The limits: Each model ran once per case. The 36 basic cases belong to 12 related families; the 12 intrusion cases belong to four. Variants within a family are not independent incidents. These preliminary results do not establish repeatability, statistical superiority, or performance across an organization’s alert queue.
Technical settings
Both models used high reasoning through OpenRouter’s pinned Anthropic endpoint, with no provider fallback. Each response allowed 16,384 output tokens. Cases, answer keys, prompts, tools, evidence limits and grading stayed on the same frozen DFIR 0.6 protocol. Independent replay verified all 48 new outcomes, and all 537 request receipts matched the spending ledger with no unresolved charges.
Data sources and release details
Sonnet 5 high-reasoning results · Sonnet 5.5 high-reasoning results · Scoring and benchmark overview · Main benchmark tables
Release hashes
Prior release:
5e38e2573494a3ce7136d2c680c7dfd55d10d55834c6c8d5883e189b91abcb33
Sonnet 5.5 release:
ba4bc2aab466f11fa90a06f073f57401f7f5c67bdc1cbc575d84bad84d6a3042