Researchers release benchmark for AI agents that invent tool calls

An AI agent’s permission rules are little help if its requested tool does not exist. A preprint hosted on arXiv examines that problem and releases the Hallucinated-Tools Benchmark to compare systems designed to catch it.

The authors report 322 genuine hallucinations across ten hosted models and two ways of submitting calls. Fabricated tool names appeared more often when models could emit unconstrained raw JSON than through the alternative interface tested.

This is a measurement study, not peer-reviewed evidence that every agent system has the same weakness. Within the tests, greater model size did not reliably remove it.

Check existence before permission

The proposed resolver checks a call against a fixed registry of tools and verifies its argument signature before any subsequent permission decision. It requires no model training.

The authors also examined Model Context Protocol setups, where several servers’ tools share a namespace. They report another 154 hallucinations, including failures from models that had passed the single-registry tests. Name collisions and tools obscuring one another introduce problems beyond checking one list.

The approach has a defined limit: an argument borrowed from another context can still look valid under the declared schema. A correctly shaped call is not necessarily a correctly intended one.