Can AI Replace Market Research Surveys? What the Colgate and PyMC Labs Synthetic Consumer Study Shows
The short answer
Not yet. A study by PyMC Labs with two Colgate-Palmolive co-authors found that LLMs playing customers, scored with a method called semantic similarity rating (SSR), ranked 57 personal care product concepts about 90% as consistently as a second human panel would. The authors say the method was tuned on those same surveys, missed some demographic patterns, and measures stated purchase intent, not purchases.
A post on X early this morning said “a toothpaste company has quietly killed the entire market research industry.” By early afternoon it had about 13,000 likes and 950,000 views. The claim behind it is real research with real limits, and it is worth reading closely before anyone cancels a consumer panel.
The study is a preprint titled “LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings.” It was first posted to arXiv in October 2025 and revised later that month. Seven of its nine authors work at PyMC Labs, a statistics consultancy; two work at Colgate-Palmolive. Its data: 57 personal care concept surveys run by “a leading corporation in that market,” with 9,300 human responses from US participants.
What did the study find?
LLMs asked to play customers can rank product concepts close to how real panels rank them, when their answers are scored the right way.
The method is called semantic similarity rating, or SSR. The model gets a demographic profile and a product concept, and writes a short answer about whether it would buy. That text is compared with reference statements for each point on a five-point scale, using OpenAI’s text-embedding-3-small model. The output is a spread of likely scores.
With GPT-4o, SSR ranked the 57 concepts at about 90% of human test-retest reliability. In plain terms: a second human panel would not agree perfectly with the first, and the synthetic panel came about 90% of the way to that level of agreement. Score distributions also looked realistic, with a similarity of 0.88 for GPT-4o and 0.80 for Gemini 2.0 Flash.
How does that compare with just asking the AI?
This is the detail the viral post skipped. Asking the models directly for a number from 1 to 5 already reached about 80% of the reliability ceiling on ranking. The problem was the shape of the answers. The models picked 3 again and again, and distribution similarity fell to 0.26 for GPT-4o and 0.39 for Gemini.
So the jump from SSR is mostly about realistic answers, from 0.26 to 0.88 on distribution, and about ten points on ranking. That still matters, because a panel that says 3 to everything tells you nothing about how strongly people feel.
The authors also trained a conventional machine learning model, LightGBM, on half of the real survey data. It reached 65% of the ceiling. SSR, which saw none of the survey answers, reached 88% in that comparison.
What are the limits?
The authors list them plainly, and they change what a marketer should do with the result:
- The anchors were tuned on the same surveys. The paper says the reference statements were “manually optimized for the 57 surveys subject to this study,” and that it “remains elusive” how well they would work on other surveys.
- Some groups did not come through. Age and income patterns were captured “quite well.” Gender, region and ethnicity “were not consistently replicated.”
- It measures what people say, not what they buy. The surveys asked about purchase intent. The paper notes synthetic consumers “cannot fully capture” budget constraints, cultural context or marketing exposure.
- One category, mostly one question. All 57 surveys were personal care concepts. The main results cover purchase intent; a single exploratory experiment on how relevant the concept felt reached 82% of the reliability ceiling with Gemini 2.0 Flash.
A newer preprint points at a deeper problem. In October 2026, researchers at Hakuhodo DY Holdings and MBZUAI tested six LLMs on two US social surveys. The models matched the spread of answers question by question, but the individual synthetic respondents held combinations of views found in no real person. On that measure, every model did worse than a statistical baseline fitted to the human data. That study covers opinion surveys, not product tests, so it does not overturn the PyMC Labs result. It does show that matching the averages is not the same as simulating people.
Presscart analysis: what this means for teams that test ideas
Here is our read. Synthetic consumers are a screening tool. They are not a panel.
The useful finding is narrow and real. If you have 50 concept descriptions and money to test 5 with humans, SSR gives you a cheap first sort, and the paper’s own numbers say that sort tracks human rankings well in at least one category. The code is public on GitHub, so a team can try it this week.
The leap in the viral post is from “ranks concepts like a panel” to “you don’t need to wait a month to see if a product will sell.” The study never measured sales. It never tested pricing across demographics; the price result in the paper is that synthetic consumers rated products across price tiers about the way humans did. And the groups the models got wrong are the ones a launch team usually cares about when it picks a market.
It’s a useful instrument, and someone still has to decide what to trust. The honest version of this story is smaller than “market research is dead” and more useful: run the cheap synthetic sort first, then spend the panel budget on the concepts that survive, and keep a human sample for any subgroup you plan to target.
There is also a media lesson. A 13-month-old preprint became the day’s news through one post, and the post stretched the findings. When we asked ChatGPT and Gemini today whether synthetic consumers can replace market research surveys, both answered more carefully than the post did. ChatGPT said they can replace “some” surveys but not traditional surveys altogether, citing survey-methodology sources such as AAPOR. Gemini began its answer with “No” and said they cannot replace panels completely. Neither cited the viral post.
Sources
- Maier et al., “LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings,” arXiv:2510.08338 (v1 Oct. 9, 2025; v3 Oct. 27, 2025): https://arxiv.org/abs/2510.08338
- PyMC Labs, “AI-based Customer Research: Faster & Cheaper Surveys with Synthetic Consumers”: https://www.pymc-labs.com/blog-posts/AI-based-Customer-Research
- SSR code: https://github.com/pymc-labs/semantic-similarity-rating
- Kumagae and Kaneko, “Silicon Samples Match the Answers but Not the Respondents,” arXiv:2610.30900 (Oct. 22, 2026): https://arxiv.org/abs/2610.30900
- The viral post, @HowToPrompt__ on X, Nov. 21, 2026 (engagement as of 1:50 p.m. ET): https://x.com/howtoprompt__/status/2124040349068964215
- ChatGPT and Gemini answers to “Can AI synthetic consumers replace market research surveys?”, captured Nov. 21, 2026 via DataForSEO.
Frequently Asked Questions
The what, why, and how of Presscart.
What is semantic similarity rating (SSR)?
SSR is a way to get survey-style scores from an AI model. Instead of asking the model for a number from 1 to 5, researchers ask it to answer in its own words as a given customer, then compare that answer to a set of reference statements for each point on the scale using text embeddings. The result is a spread of likely scores, not a single number.
Did Colgate prove AI can replace market research?
No. The paper is a preprint led by PyMC Labs, with two Colgate-Palmolive researchers among its nine authors. Its main test was one question, purchase intent, on 57 personal care concepts, plus a single exploratory run on how relevant people found each concept. The authors call synthetic consumers a way to run scalable simulations and list limits, including demographics the models did not reproduce.
What does 90% of test-retest reliability mean?
If you split a real panel in half, the two halves will not rank products in exactly the same order. That agreement is the ceiling. The SSR results with GPT-4o reached about 90% of that ceiling when ranking concepts by average purchase intent. It is a measure of ranking, not of how many units a product will sell.
How well did simply asking the AI for a rating work?
Better than the viral post suggests on ranking, worse on realism. Asking GPT-4o or Gemini 2.0 Flash for a number directly reached about 80% of the test-retest ceiling, but the answers clustered on 3, so the score distributions looked little like real panels (similarity of 0.26 and 0.39, against 0.88 and 0.80 with SSR).
Which customer groups did the synthetic consumers get wrong?
Age and income patterns were captured quite well, while gender, region and ethnicity were not consistently replicated. The authors warn that persona prompts cannot yet be treated as a reliable stand-in for every subgroup.
Is there research that disagrees?
A separate October 2026 preprint by researchers at Hakuhodo DY Holdings and MBZUAI tested six LLMs on two US social surveys, a different task. It found the models matched answer distributions question by question but produced respondents unlike any real person, and did worse than a statistical baseline built from the human data.
Can I try the SSR method myself?
Yes. The authors published a Python implementation on GitHub under pymc-labs/semantic-similarity-rating. Their reference statements were tuned for the 57 surveys in the study, so results on other products or questions may differ.
Presscart Team
Editorial Team
The Presscart editorial team covers PR strategy, media placements, and the tools founders and marketers use to get press coverage.