Step by step · what a chat cannot do · 9 min read

Is ChatGPT enough to validate an idea?

It is enough for exactly one part of validation, and that part decides nothing. The conversation can clarify your thinking, name the obvious competitors and sketch the arithmetic. All of it happens inside the conversation. Validation is the part that happens outside.

We sell a tool in this category, so this page tries to earn the claim: eight steps, what the chat does at each, what happens here, and — at the end — the things ChatGPT does better than us.

Short answer

Is ChatGPT enough to validate a business idea?

It is enough for one part, and that part decides nothing. A chat can clarify the idea, list the questions worth asking and name competitors — all inside the conversation. Validation is the part that happens outside it: someone other than you being asked, a page real people land on, a number that comes back and disagrees.

What the chat window cannot do

It only knows what you typed. The part of your thinking you never wrote down stays out of the analysis, and no prompt reaches it
It checks its claims against itself. Self-consistency looks like accuracy and is a different thing
Its verdict is text, so a more confident description moves it — you can raise your score without changing your business
It cannot publish a page real people land on, and cannot send questions to anyone but you
It never finds out whether it was right, so it has no error rate and no way to acquire one
Its known agreeableness is the weakest of these objections, because a prompt really can push back against it. The other five are missing steps, not missing instructions

Grade your own questions free

The substitution, named

Here is the sequence, and it is extremely common. You describe the idea to a chat. It tells you the market is large, the timing is good and the main risk is execution. You feel the thing you came for — a weight coming off — and you start building.

What just happened is not that you skipped rigour. It is that every step that touches somebody other than you was skipped, and the step that stayed was the one you were already good at: describing your idea to someone sympathetic.

The usual complaint about this is that the model flatters you, and that is true. It is also the weakest version of the argument, because flattery is fixable: ask it to be harsh, tell it to find the strongest case against, and it will. So take that objection as granted and conceded. The interesting question is what stays broken after the prompt is perfect.

Eight steps, side by side

The input

In a chat

You type a paragraph. The model knows exactly what you chose to tell it, which is the part you already understand.

Here

You are interviewed for fifteen minutes in your own language, and a thin answer draws a follow-up instead of passing. The conversation fills thirteen of fourteen context fields that the research then runs on.

Why a prompt does not close it: A prompt cannot make you say what you did not think of. Only a question can.

The method

In a chat

Frameworks are recalled from training data and applied loosely. Ask for a Mom Test review and you get the spirit of it, differently each time.

Here

The methods are written down as rules and run as code — the Mom Test grader, Kano, Van Westendorp, Gabor-Granger, RICE. The same input gives the same reading, and each one shows how much to trust its own output.

Why a prompt does not close it: Consistency is a property of code, not of a well-worded request.

The objections

In a chat

You can ask a model to role-play a sceptical customer. It will play the role you described, which is your own picture of a sceptic.

Here

A separate persona engine builds a focus group from public writing by people in that market, and you can question each persona by voice. They are still synthetic, and we say so — but they are not your imagination reflected back.

Why a prompt does not close it: A role-play draws from your description. An engine draws from someone else.

The checking

In a chat

One model verifies its own claims. Self-consistency looks exactly like accuracy and is not the same thing.

Here

Key claims are cross-checked by three independent models, and disagreement between them is reported rather than smoothed away.

Why a prompt does not close it: Nothing you type makes one model into three.

The verdict

In a chat

Produced as text. Rewrite the description more confidently and it moves, which means it is partly measuring your prose.

Here

Computed against fixed thresholds. It cannot be talked into a kinder answer, and a NO-GO is a normal outcome rather than an edge case.

Why a prompt does not close it: A generated verdict is steerable by definition. A computed one is not.

The build question

In a chat

It will estimate cost and stack in general terms, which is the one question where general terms are worthless.

Here

A separate stage works through architecture and stack choice against your actual constraints, and produces a technical specification rather than an opinion about one.

Why a prompt does not close it: Not a knowledge problem. A stage that either exists in the process or does not.

The demand test

In a chat

It can write you a landing page as text. Publishing it, putting it somewhere real and finding out whether anyone signs up is not something a chat window does.

Here

A landing page is generated and published at its own address, and the signups that arrive are recorded against the project.

Why a prompt does not close it: This is where opinion ends. A page either exists on the internet or it does not.

Closing the loop

In a chat

Whatever the model told you is never afterwards compared with anything. It has no error rate and no way to acquire one.

Here

Questions generated from your own hypotheses go out as a public survey; nothing is scored until at least five real people have answered, and the process then records how far its earlier prediction was from what they said.

Why a prompt does not close it: The last one is the real difference: being willing to find out you were wrong, and keeping the number.

What ChatGPT does better than us

A page that only listed the other side’s problems would be advertising. These are real, and if the list below covers what you need, you do not need us.

1It is free, instant and unlimited. Nothing we do competes with asking a question at two in the morning and getting an answer immediately.

2It is the best tool in existence for sharpening a problem statement. Paste your idea, ask it to restate the problem in one sentence without naming your solution, and argue until the sentence is true.

3It will generate twenty candidate interview questions in a minute. Grading them is a separate job, but producing them is a real saving.

4It explains a framework you half-remember better than most textbooks, and it never gets bored of the fourth follow-up question.

5For breadth — what exists, what words this market uses, which adjacent problems keep coming up — it is genuinely excellent, and that is exactly the stage where breadth is what you need.

The mistake is not using ChatGPT. It is mistaking a conversation with a model for contact with a market. Those feel identical from the inside, which is the whole problem.

What to do with the answer you already got

Do not throw it away — convert it. Take the three statements the model was most confident about. For each one, write the sentence that would have to be true for it to be false: a competitor that already does this well, a customer who solves it another way for nothing, a price nobody will pay.

Now you have three falsifiable claims instead of one verdict, and they are ordered by how cheap they are to check. That list is worth more than the verdict was, and the model helped you build it.

If nothing on your list can be falsified, you did not receive an analysis. You received an opinion — and an opinion from a model carries the same evidential weight as one from a friend, with the same bias towards making you feel good about yourself.

Start with the free part

Paste the questions you were planning to ask your customers. The grader marks the ones that pitch, ask for an opinion, or ask about the future instead of the past — the three failures that make an interview useless. No account needed.

Grade my questions →

Runs in your browser · no signup

Frequently asked questions

Is ChatGPT enough to validate a business idea?+

It is enough for one part and that part decides nothing. A chat can restate your idea more clearly, list the questions worth asking, name obvious competitors and sketch an arithmetic. Every one of those happens inside the conversation. Validation is the part that happens outside it: someone other than you being asked something, a page real people land on, a number that comes back and disagrees with you. A chat has no mechanism for any of that, and it has no record of ever having been wrong.

Can I just write a better prompt?+

A better prompt fixes tone, not structure. You can make a model argue with you, refuse to flatter, demand evidence — and people sell prompt packs that do exactly this. What no wording changes: the model still only knows what you typed, still checks its claims against itself, still produces the verdict as text that a more confident description can move, and still never finds out whether it was right. Those are missing steps, not missing instructions.

What is ChatGPT actually better at here?+

Speed, breadth and cost. It is free, instant and unlimited; it will rewrite your problem statement until it is sharp, produce twenty candidate interview questions in a minute, explain a framework you half-remember, and summarise a market you know nothing about. Use it for all of that. The mistake is not using ChatGPT — it is mistaking a conversation with a model for contact with a market.

So what should I do with the answer ChatGPT already gave me?+

Treat it as a list of hypotheses rather than a verdict. Take its three most confident statements and write, for each, what you would have to see to know it was false. Then go find out about the cheapest one. If nothing on the list can be falsified, you have received an opinion, and opinions from a model have the same evidential weight as opinions from a friend — with the same bias towards making you feel good.