Holdline Benchmarks AI Agent Guardrails Against Injection Attacks
TL;DR. A new benchmark, Holdline, evaluates AI agent guardrails using metrics like catch rate and false-block rate, including a novel injection-attack class for testing robustness. - Holdline measures guard effectiveness in blocking undesirable actions by AI agents in DeepSeek Harness. - The benchmark uses a hand-authored corpus and real agent trajectories for comprehensive guard evaluation. - It provides a standardized method for developers to test and compare the safety performance of their AI agent plugins.
- Holdline is a neutral benchmark for AI-agent guardrails.
- It measures catch rate, false-block rate, and class-balanced Cohen's kappa.
- A key feature is the inclusion of an injection-attack class to test guardrail robustness.
- The benchmark addresses the lack of standardized testing for AI agent policy plugins in ecosystems like DeepSeek Harness.
- It allows developers to add and test their own guard implementations against a standardized corpus.