The Security Benchmarks blog

Read past the score.

Research updates on AI security agents, the evidence behind their answers, and what the scores leave out.

Latest dispatchFollow via RSS ↗

No. 07 / Meta, GPT, Claude and Gemini

Muse Spark 1.3 leads Meta’s basic proof scores

All High effort. Spark 1.3 returned more basic verdicts with proof than Wonder, at a lower normal-price estimate. Compare eight models across both investigation tiers.

Read the Muse comparison →
Verdicts with accepted proof, basic / intrusion: Muse Spark 1.3 11/36 and 4/12; Muse Wonder 9/36 and 4/12; Muse Spark Open 1/36 and 1/12; GPT-6.1 Sol 25/36 and 5/12; GPT-6 Astra 25/36 and 5/12; Gemini 4 Argon 29/36 and 4/12; Claude Opus 5.5 12/36 and 5/12; Claude Sonnet 5.5 11/36 and 4/12. All High effort. DFIR 0.6; one pass; preliminary.
Verdicts with accepted proof · All High effort · DFIR 0.6

No. 06 / Gemini, GPT and Claude comparison

Gemini 4 Argon reaches 29/36 basic verdicts with proof

At High effort, Gemini led basic proof counts in this pass, but trailed Sol, Astra and Opus on intrusions. Compare verdicts, proof, Standard cost, time and token coverage.

Read the Gemini comparison →
Basic verdicts with accepted proof: Gemini 4 Argon 29/36, GPT-6.1 Sol 25/36, GPT-6 Astra 25/36, Claude Opus 5.5 12/36, Claude Sonnet 5.5 11/36. All High effort. Estimated 48-case Standard costs: $21.69–$23.59, $4.83, $31.01–$53.31, $54.57–$55.35, $21.57. DFIR 0.6; one pass; preliminary.
Gemini, GPT and Claude 5.5 · High effort · DFIR 0.6

No. 05 / GPT and Claude comparison

GPT-6.1 Sol matches Astra’s proof scores for $4.83

At High effort, GPT-6.1 Sol matched Astra’s accepted-proof counts in both DFIR 0.6 tiers for $4.83. Compare verdicts, proof and cost with GPT and Claude 5.5 models.

Read the GPT and Claude comparison →
GPT-6.1 Sol and GPT-6 Astra, both High effort: 25/36 basic and 5/12 intrusion verdicts with accepted proof. Estimated Standard costs: $4.83 vs $31.01–$53.31 for 48 DFIR 0.6 cases at October 14, 2026 prices, with the same caching and retries. One pass; preliminary.
GPT-6.1 Sol, GPT peers and Claude 5.5 · High effort · DFIR 0.6

No. 04 / Sonnet comparison

Sonnet 5.5 improves security verdicts and cuts run cost

At high reasoning, Sonnet 5.5 improved verdicts and accepted proof on the same 48 DFIR 0.6 cases as Sonnet 5. The run cost fell 41.9% to $21.57.

Read the Sonnet comparison →
Claude Sonnet 5 and Sonnet 5.5, both at high reasoning: evidence-backed basic verdicts improved from 0/36 to 11/36 (30.6%). DFIR 0.6, one pass. Total 48-case cost fell from $37.13 to $21.57.
Sonnet 5 and 5.5 · High reasoning · DFIR 0.6

No. 03 / Incident scope

We tested 3 AI decision models. Clef found more of the attack trail.

We asked 3 AI decision models to trace a security incident from one confirmed starting point. Clef recovered more in this early eight-case pilot.

Read the incident-scope findings →
Lighthouse print showing average incident scope scores: Clef 70.8, Clef Flash 62.1, Jev 1.13 20.8. Eight cases, three runs each; early pilot.
Incident-scope 1.0.0 · Eight-case pilot

No. 02 / Decision-model analysis

18/18 verdicts. 12/18 next steps.

Jev’s perfect verdict score, the next-step gap, and what six SOC decision models tell us about choosing an action.

Read the decision-model post →
Jev 1.13 · SOC decision suite 1.0.0
18/18

Correct
verdict answers

12/18

Accepted
next-step answers

Six cases. One pass.
Classification and action are separate tests.

No. 01 / Benchmark analysis

AI security benchmark results: decisions and evidence

Our first 19-model DFIR release shows why a correct answer and a supported answer deserve separate columns.

Read the launch post →
Claude Opus 5.5 · Basic investigations
33/36

Correct final
decisions

12/36

Decisions with
accepted proof

Same model. Same cases.
A different question about performance.

Our approach

Useful findings. Visible evidence. Room to be wrong.

Security Benchmarks is an independent research project studying AI agents in digital forensics and incident response. This blog explains new results, investigates specific tasks and records changes to the methods.

We keep posts brief, show the denominators and distinguish a measured result from an interpretation. Read the benchmark overview or go straight to the results tables.