Stop Asking AI to Roleplay

An hour ago, an AI told me my own website was invisible to AI search. Then the live web proved it right.

Most "AI council / persona panel" demos are theater. You assemble a few AI characters, they disagree colorfully, you nod, you ship. You never find out if any of them were right. A disagreement you can't grade is just entertainment.

So I built mine differently. Six grounded "seats" review a draft: a target reader, the Google-ranking lens, the AI-answer-engine lens, a hook/distribution lens, an editor, and a fact-checker. None run on vibes. The answer-engine seat reads actual GEO research, the hook seat is built on the public playbooks of top short-form creators, the fact-checker knows FTC rules. They produce a scorecard.

But a scorecard is still theater until the last step: every run has to commit to a falsifiable prediction, and I log it.

Not "this looks good." A checkable claim: this page will NOT get cited by AI search for this query. No prices, no specs, its only proof trapped in an Instagram embed. Confidence: high. Logged. Then you go check reality and grade it. You score by confidence band, so I can see whether "high confidence" means accurate or just confidently wrong. That move turns a plausibility engine into a measurable one.

The test: I ran it on the metal-fab business I used to run (custom smokers and grills). With no web access, the answer-engine seat predicted my homepage would not appear when someone asks an AI "best custom offset smoker builders," and named exactly why. Then I pulled the live results. It was right: my site was absent. Every site that did show up had what mine lacked: prices, spec tables, reviews, lead times.

Here's why brochure sites go invisible. The web shifted from a clicks economy to a citations economy, and some answer engines surface as few as three to seven cited sources per answer. A 2024 study (Aggarwal et al., KDD) tested what moves the needle: adding quotations, statistics and source citations raised a page's visibility in AI answers by up to about 40%, while keyword stuffing actually hurt. The machines reward specific, sourced, structured content. Most homepages are none of those.

It even caught me inflating a number in my own first draft. I fixed it. That's the whole point.

The fix for AI hype is falsifiability. If your AI system can't make a claim you could check, and be embarrassed by, all you have is a vibe with good PR. Make it predict. Log it. Grade it. Let the score be unflattering.

Honest disclosure, because the thesis demands it: my log is one-for-one so far. That's not a track record. It's the first entry in one. The point was never the score. The point is that there is a score.

And yes, I ran this post through the same panel before publishing it.

So if you're building with agents: what's the last falsifiable prediction yours made, and did you actually check it? (And if you've never checked one, that's the post.)