Synthetic focus groups, and how to hold one to account
We run one inside our own product, so read this accordingly. It is written as the checklist we would use on somebody else's panel, including the check that our own current number is not ready to publish.
Short answer
What is a synthetic focus group, and when is it worth using?
A panel of AI personas standing in for a target audience, questioned like a real focus group. It is worth using to rehearse a hypothesis in minutes instead of weeks. It is not worth using as evidence of what customers will actually do.
The four checks, before you trust any panel
- Where did the personas come from?
- Was anything held back?
- Does the panel ever disagree with you?
- Does the vendor publish where it is weak?
The four checks
1.Where did the personas come from?
Good answer
Built from real posts by real people in that market, with the sources nameable.
Bad answer
Demographics plus a prompt: "you are a 34-year-old marketing manager".
A persona invented from demographics returns the model's idea of that demographic. It will be fluent, coherent and worth nothing, because it was never anchored to anything a person actually wrote.
2.Was anything held back?
Good answer
Personas built from part of the data, predictions checked against the part they never saw.
Bad answer
A quality claim with no test described, or a test on the same data the personas were built from.
Without a held-back set there is no way to tell reproduction from prediction. This is the single check most vendors skip, and it is the one that decides whether the number means anything.
3.Does the panel ever disagree with you?
Good answer
Sessions that come back with objections you did not want to hear.
Bad answer
Enthusiasm. Every idea lands somewhere between "interesting" and "I would use this".
A panel that likes everything is a compliment generator. Compliments are the most expensive thing in early-stage product work, because they cost months.
4.Does the vendor publish where it is weak?
Good answer
A named weak spot, with the measurement that found it.
Bad answer
One accuracy number, prominently, and nothing else.
Any honest calibration produces uneven results: some dimensions match well, others badly. A vendor showing only the good dimension is showing you a marketing number, not a measurement.
How ours is built, and what we are not claiming
Personas are built from real posts across eleven platforms, not from demographic sketches. Part of the collected material is held back before the personas are made. The personas then describe the problem space, and that description is compared with what the real people in the held-back part actually wrote.
That comparison produces an agreement figure. We are not publishing ours right now, for a reason worth stating plainly: the runs behind it were made in February on the model we used then, and the engine has changed since. A number measured on a retired model is not a property of the product we ship today. It is being re-measured, and it goes back on this page when it is fresh.
We also know where the method is weak, and it is not the headline dimension. Identifying which pains exist works well; ranking which pain matters most works markedly worse. That is the number we would want to see from any vendor, and it is the one nobody shows.
What it can never tell you
Whether anyone will pay. A synthetic panel has no wallet. It can tell you which objection comes up first and which words land badly; it cannot tell you that someone would move money.
Whether you will still care in six months. The most common cause of death for a validated idea is the founder losing interest. No panel scores that.
Treat the output as a list of things to go and check, ordered by how badly they would hurt if true. That is the whole use.
Common questions
What is a synthetic focus group?
A simulated panel: AI personas standing in for a target audience, asked the questions you would ask a real focus group. Sessions run in minutes rather than weeks, cost a fraction of a recruited panel, and can be re-run with variations. What they produce is a rehearsal of the conversation, not evidence of what customers will do.
Are synthetic focus groups accurate?
The question is unanswerable as asked, because accuracy depends on what was held back. If a panel is built from a dataset and then tested against a part of that dataset it never saw, the resulting agreement figure means something. If it is built and then praised, the figure means nothing. Ask which of the two happened before you compare vendor numbers.
Can a synthetic focus group replace talking to customers?
No, and any vendor saying otherwise is selling. Synthetic panels rehearse a hypothesis: they surface the objections you have not thought of, cheaply, before you spend a real person's time. The moment the question becomes "what will this specific customer actually pay for", the answer has to come from that customer.
How do synthetic focus groups compare with surveys?
A survey answers "how many", a focus group answers "why", and a synthetic panel answers "what might they say" before you have spent anything finding out. They stack rather than replace each other: rehearse with the panel, ask the humans the questions the rehearsal exposed, then use a survey to size whatever the humans confirmed.
Rehearse, then go outside
A synthetic panel runs inside our validation. It is one of six research stages, and it feeds the interview questions rather than replacing them.