Claude Agents + Chat SDK, Gemini cuts tokens 17%

Data Point

DeepsecBench ranks models finding code vulnerabilities

Open benchmark evaluates vulnerability detection across 25 models with scoring weighted 2:1 on recall over precision; cheapest competitive option is Grok 4.5 high ($5.60) vs GPT-5.6 Sol xhigh ($55.98) with top score 35.58.

Developers can now compare vulnerability detection cost-to-performance across models to optimize their automated security scanning pipelines, replacing expensive frontier-model-only approaches with cheaper multi-model strategies that fit their codebase complexity and budget constraints.

Replaces manual model selection guesswork with empirical recall/precision/cost data. Requires integrating deepsec tool and selecting model tier by cost-benefit tradeoff. Ready now—benchmark is live and public. Caveat: best model only finds 30.7% of vulnerabilities; no single pass is comprehensive.

• “the benchmark is run three times and the data published in the report is the median of the three runs”
• “we don't disclose the repository, the commit, the files, or the findings, so there is nothing for models to train against”
• “higher price does not buy proportionally more”
• “Kimi K3, from Moonshot AI, ranks eighth at a score of 17.56 for $12.38 on the high setting, half the top score for about a fifth of the cost”
• “Anthropic's most capable model, Fable 5, is absent because it declines security work, including defensive tasks”
• “weighting recall (R) twice as much as precision (P), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure”