How do you test an agent that has memory?

You test it for whether the right fact is still in play, not whether a fact got stored. Mem0’s own testing write-up gives the gap directly: a test can assert memory.search("diet") == "vegan" and pass cleanly while the agent still recommends a cheese plate to a user who went vegan six weeks earlier.

Four checks catch what that single assertion misses: whether an old preference still surfaces after an update (a stale fact), whether the old and new values both come back with nothing marking which is current (a contradiction), whether a fact set early stays retrievable as the conversation grows (drift), and whether one user’s memory ever turns up in another user’s results (a leak). None of those show up in a five-turn test; Mem0 ran its checks across several contradiction rates and several random seeds to tell a consistent failure from one bad roll.

That’s the same discipline a quality shift after a change needs generally: a rate measured against a baseline, across enough runs to mean something, not a verdict from whichever run you happened to look at.