Before your agent presses send.
Three new research papers ask an awkward question: does the machine still know what it is doing?
The briefing
Prepared on Saturday, 5 September, for your Sunday reading. This is a research briefing on papers submitted on 3 September and checked against their original pages tonight, not a report of events that have yet to happen. The common thread is practical: a fluent answer is one thing; a dependable action is rather more demanding.
Fresh memory. Yesterday’s plan.
PlanFence examines agents that receive an update but carry on with a decision made before it arrived.
An assistant can remember the new instruction and still execute the old plan. That is the problem studied in Fresh Memory, Stale Plans, a paper by Evan Chen, Shiqiang Wang and Christopher G. Brinton.
Their proposed PlanFence checks the particular records behind an action immediately before execution. If a relevant input has changed, it replans; if the check cannot be completed, it blocks the action. In 30 controlled workflows with a revision after planning, the authors report that their freshness-only comparison followed the obsolete plan every time, while PlanFence completed all 30 without an invalid action.
That is a result from a deliberately narrow experiment, not a guarantee for every agent. The paper covers three workflow families and explicitly limits its claims.
Our practical test: change a delivery instruction after an agent has prepared an order. Does it rebuild the order or merely acknowledge your message? A polite acknowledgement is lovely. The parcel still needs the right address.
Market signal
When the instructions disagree.
KC-Bench tests how models handle conflicting facts, inconsistent identities and information that changes over time.
KC-Bench offers 238 manually screened tasks drawn from more than 1,000 generated candidates. Its authors tested nine models in controlled, multi-turn environments with tools. They report that no model handled every category of conflict reliably.
The benchmark examines models, not complete commercial agent products. Its simulated failures should not be presented as evidence that a particular deployed service leaked real customer data.
For a buyer, this suggests a useful addition to the demo: give the assistant two records that disagree. Ask it to identify the authoritative source before taking action. Then change the record. A supplier who can explain that behaviour is telling you more than a leaderboard screenshot ever will.
What to watch
The reviewer needs a reviewer.
HalluPeer studies unsupported statements in scientific peer reviews, including the difficulty of separating an error from fair criticism.
HalluPeer pairs papers with reviews and versions containing deliberately inserted errors. The authors describe experiments covering 12,000 papers and 38,000 reviews, and report that existing detectors struggle to distinguish fabricated claims from legitimate criticism. The arXiv page lists acceptance to EMNLP Findings 2026.
The limits matter: its constructed errors are synthetic, and the source material comes from computer-science conferences on OpenReview. It does not establish the error rate of all human reviewers or all AI assistants.
Our takeaway: when using AI to assess a report, require a passage supporting each factual criticism. Separate what the document actually says from what the reviewer would have preferred it to say. Confident disapproval has never been in short supply.
What to watch next
- Before an action: check whether the inputs behind the plan changed.
- When records disagree: establish which source has authority.
- When reviewing a document: attach evidence to the criticism.
The takeaway
Give your next AI trial one ordinary complication: an updated instruction, a conflicting record or a claim without evidence. Watch what happens before the system acts.
The editor’s view
Our reading: the useful question is moving from whether an agent can finish a task to whether it knows when its reasons for acting no longer hold.