The agents we're trusting to shortcut our research are hallucinating storefronts.

The Summary

The Signal

Podqi ran shopping queries through three major AI assistants in September. Simple stuff: "What are the top websites for buying [brand]?" and "Where can I find [brand] online?" The kind of questions people ask when they're trying to save time, not lose money.

The results show these models doing exactly what we've been warned they would: confidently hallucinating answers when they don't know, which in e-commerce means inventing URLs or linking to imposter sites designed to steal credentials and credit cards.

"These aren't research tools anymore. People are using them as shopping assistants, and the models are recommending fraud."

This matters because we're at an inflection point in how AI agents handle commerce. Shopping assistants are supposed to be the gateway drug to agent-mediated transactions. You ask, the bot researches, eventually the bot buys on your behalf. But if the bot can't reliably identify legitimate retailers, the entire trust model collapses before it starts.

The test covered 151 brands, mixing household names with smaller companies. The diversity matters because:

  • Large brands have more fake sites mimicking them
  • Small brands have less comprehensive web presence for models to learn from
  • Both scenarios produced errors, suggesting this isn't a data problem, it's an architecture problem

Models weren't just linking to sketchy third-party marketplaces. They fabricated URLs entirely. Made up domain names that sound plausible but point nowhere, or worse, to parked domains waiting to be weaponized. When a model invents www[dot]brand-official-store[dot]com and you click through, you're not getting redirected to safety. You're entering your payment information into a site that exists solely to harvest it.

The timing of this test is critical. September 2024 means these are current-generation models, post all the RLHF refinements and safety layers we've been promised. These aren't early beta hallucinations. This is production behavior.

The Implication

If you're building agent infrastructure for commerce, you need a verification layer between model output and user action. Not content moderation. Transaction verification. Every URL an agent suggests needs to hit a reputation database before it reaches the user. Every "buy here" recommendation needs a cryptographic handshake with the actual brand.

For users: treat AI shopping recommendations like you'd treat a random Reddit comment. Verify the URL yourself. Go directly to the brand's known site. Don't click through from the chatbot, no matter how confident it sounds.

The companies training these models need to solve hallucination in high-stakes domains before pushing agents into commerce. A bad movie recommendation wastes two hours. A bad storefront recommendation empties your bank account.

Sources

The Verge AI