TWIST tests whether AI memory knows when to object

On a human-checked set of 161 items, retrieval-based systems caught 76–97% of real contradictions. They also incorrectly flagged 16–43% of safe drafts. Finding more problems is not much help if the problems were never there.

A deployed system focused on coherence made the opposite trade-off. It left safe drafts alone 98–100% of the time, but caught only 42% of genuine contradictions.

The cost of a false alarm

TWIST pairs each intervention test with a similar-looking example that should be left alone. A system cannot earn a good score simply by objecting to everything.

Its proposed tests cover changing beliefs, checking drafts, retaining the history of an answer and controlling sensitive recall. With the full conversation supplied, calibrated models nearly solved the draft-checking task, pointing to gaps in retrieving the right evidence.

These are preprint results, not general performance ratings for AI assistants. No tested setup scored strongly across catching contradictions, avoiding false alarms and attribution together.