18/18 verdicts. 12/18 next steps.

The evidence is already on the desk

Each model receives the same prepared telemetry, question and answer choices. It must choose an answer. It does not search for logs or call investigation tools. Jev uses a native decision interface; the chat models return a choice through a JSON wrapper.

The suite asks six kinds of question: classify the alert; update a hypothesis; check a conclusion; judge evidence relevance; choose a next step; and follow an escalation policy. Those are separate scores. Fixed answer keys determine credit, with multiple next actions accepted where appropriate.

Two decisions, different results

All six published models. Each shown category has three questions per case. Scores weight cases equally; these 18 questions are not 18 independent incidents. Primary results · GLM results.

The next-step gap matters. Jev’s 66.7% is below the category’s 72.2% best constant-answer baseline. That baseline always chooses the single option that scores best across the category. It exposes the ease of exploiting an imbalanced test; it is not a useful operating policy.

A verdict score alone would miss this distinction. These results suggest testing the action you want a model to take, rather than treating accurate classification as a proxy for every downstream decision.

The output contract is part of the score

Claude Opus 5.5 had 13 invalid verdict answers out of 18; GLM-5.3 Flash had nine. Invalid or missing choices receive zero. Prose or fenced JSON can fail this protocol even when it contains the correct choice.

Those failures matter to a system expecting a usable decision. They also limit the interpretation: low accuracy here does not isolate reasoning quality from answer-format compliance. Opus’s investigation results come from a different task and response contract.

A cheap decision still needs to be the right one

Jev’s recorded cost for its 272-question pass was about $0.02; Astra’s was about $2.16. These are observed run costs, not current price quotes or production cost forecasts. The models used different interfaces and settings.

The useful question is which task a cheaper model can handle reliably. This small evaluation does not establish a safe routing threshold. The escalation category tests compliance with a stated policy, not whether a larger model will solve the next case better.

Read the category you would actually use

On the leaderboard, select Archived model results, then SOC decisions. Compare the category score with its baseline, invalid answers and critical errors. A perfect verdict column is one useful observation. The next action is another test.