AI chatbots are already a trusted source of shopping advice. Roughly 70% of American consumers report using AI for shopping, with nearly two-thirds saying AI had influenced a recent shopping decision, according to a recent survey from LDWW.
But a new study shows that AI may not be the impartial shopping buddy users think it is: Rather, some of the most popular AI models, including ChatGPT and Claude, recommend more expensive products to users they think are wealthy based on personal data, even if those users specifically ask for the cheapest option.
AI’s pocket-watching tendencies
The study, which was published to arXiv, an open-access archive for scientific research papers prior to peer review, put 13 AI models to the test across 325,000 trials. Researchers gave the models access to fake user profiles with information including employment, health, and finances, then made identical requests across three kinds of purchase decisions: flights, health insurance, and graduate programs.
The researchers found that eight of the models routinely recommended more expensive purchases to users they perceived as more wealthy. Major models including Claude Opus 4.8, Gemini 2.5 Flash, and GPT-5 all recommended products more than $100 more expensive to high-income users versus low-income users. Claude Opus 4.8 displayed the largest gap, suggesting that high-income users purchase flights costing an average of $198 more dollars and health insurance plans costing an average of $284 more dollars per month than those it recommended to low-income users.
Specifically requesting cheap options didn’t stop the phenomenon. Wealthy profiles asking for cheap flights, for example, were still offered more expensive options than low-income users making the same requests, though the severity varied by model. When high-income profiles request the cheapest flights available, Gemini 2.5 Flash recommended flights that were $208 more expensive on average, while GPT-5 and Claude Opus 4.8’s recommendations were only $21 and $20 more pricey on average.
AI models don’t need full financial information to make assumptions about users’ wealth, either. Even when the researchers removed models’ access to structured financial data and only let them go through users’ inboxes, the models still inferred users’ wealth from the data in their emails and expressed similar pricing gaps in their recommendations.
What is adversarial delegation?
The researchers named this phenomenon “adversarial delegation,” referring to the double-edged sword of personalized AI agents: While increased access to personal information might make AI assistants more useful, it also enables them to act against their users’ interests.
According to the study’s authors, adversarial delegation reflects real-world predatory sales tactics. “Even when you delegate to your own agent, the LLM leverages your private information about you just like an arm’s-length seller would,” they wrote.
The study builds on anxieties around surveillance pricing, through which customers are offered different prices set by algorithms based on their buying behaviors. Surveillance pricing is already a hot topic for legislation around AI regulation, and the researchers argue that adversarial delegation should join that conversation.
“These results highlight the need for policies and designs that go beyond individual data minimization to restrict usage and reframe the debate from the accessibility of personal information to the objective function over that information,” reads the study’s conclusion.
