A new academic benchmark finds that personal AI agents show measurable gains when they retain information across sessions, but the improvements don't always come from the memory systems vendors claim are doing the work. PAST-Bench, released as a paper on Hugging Face by a group of researchers, tests whether agents that save preferences, workflows, and corrected facts between sessions actually use that stored information to perform better—or whether the underlying language model would have succeeded anyway. The benchmark spans 26 scenarios and 204 episodes, with each test run twice under identical conditions except for a single switch: persistence turned on or off.
Across seven base models and four agent frameworks, turning on persistence lifted the family-balanced performance gap from +0.13 to +0.24, but where each model concentrated that gain varied sharply. GPT-5.4 spread its improvement across Memory and Update tasks, GLM-5.1 directed nearly half its movement to Update capabilities, and Kimi K2.6 put nearly half on Memory. The benchmark evaluates four capabilities: memory, procedural reuse, information gathering, and update. A separate mechanism-evidence score tracks whether improvements actually traveled through the intended save-retrieve-update pathway—two frameworks can hit the same headline gain of Δ=+0.13, but nanobot lands at a mechanism score of 0.57 against Hermes's 0.64, meaning one framework's improvement didn't consistently pass through a write-then-read trace.
The authors argue that a single overall performance number tells the wrong story. The report extends one framework, Hermes, with five targeted runtime interventions—a plan-time consultation gate, typed memory rendering, skill routing, retrieval gating, and a synchronous closeout—packaged as Hermes+. On MiniMax-M2.7 it raised the mean overall gain from +0.13 to +0.15 and the mechanism score from 0.64 to 0.73, with its strongest lift on tasks requiring replacement of outdated state, where the delta jumped from +0.12 to +0.24. The paper includes an honest caveat: the +0.02 overall improvement sits smaller than the run-to-run standard deviation of ±0.04 for Hermes and ±0.06 for Hermes+, so it shouldn't be read as a stable aggregate gain.
The benchmark isolates a question that marketing pitches for agent memory usually skip: is the improvement real, or is the base model carrying the headline number on its own? The matched-ablation design—where every episode runs twice with persistence as the only variable—lets framework builders and buyers point at whether learning mechanisms actually fire. The report notes that same scores can mask different reasons for those scores, with mechanism-evidence revealing whether an agent truly wrote, then read, the information it claims to remember. The authors acknowledge the test doesn't cover real human-authored traces over long horizons, families of memories that interfere with each other, or the token and latency cost of running all five interventions in production.
The forward-looking take positions this less as a leaderboard and more as a diagnostic. If a vendor sells agents that learn from users, the report says, this is the kind of matched-condition test that would reveal whether the learning is real or whether the improvement would have happened anyway. Hermes+ showed its clearest edge on Update tasks but also widened variance and regressed slightly on DeepSeek-V4-Pro and Claude Opus 4.6 when swapped in unchanged, suggesting interventions tuned for one model don't transfer cleanly to another. The bottom line: every base model gained from persistence, but the pathway between storage and performance remains less automatic than the agent-memory pitch implies. Enterprises weighing build-or-buy decisions now have a diagnostic that separates infrastructure claims from testable mechanism, though the production cost and cross-model portability questions remain open. Buyers who treat memory as a checkbox feature may find they're paying for plumbing that doesn't always connect the way the architecture diagram promises.