Why is my perfectly built website still invisible to AI?
Updated: 1 day ago
I've run AI visibility scans on 453 businesses so far, 100 SaaS companies, 241 financial services firms and 112 law firms. Sometimes the websites AI struggles with are the modern-looking ones: fast, well-designed sites built in the last year or two with real investment behind them, with slick interactions, embedded tools and content that loads smoothly as you scroll or click. Those same qualities can make a site hard to read for the systems answering questions on ChatGPT, Claude, and Perplexity. Even when AI can read a site, it can still describe the business wrongly, a problem I cover in why ChatGPT gets your services wrong.
HubSpot says it lost 80% of its website traffic in a year and still grew leads by 20%, and puts that down to the higher value of the visits it kept. Buyers who arrive after asking AI have usually already narrowed their options. A site AI can't read misses those buyers.
This is written for exactly that person: someone who's already put real time and budget into their site and can't work out why it's still invisible.
Getting the crawler basics right, robots.txt, sitemaps, TLS, is table stakes at this point. What catches most people out is the interactivity built for humans.
Here are the four patterns that show up a lot, starting with the most common:
Content built by JavaScript (client-side rendering)
The biggest one, and the most common on modern SaaS marketing sites. If a site relies on JavaScript frameworks to build the page in the browser, an AI crawler that only reads raw HTML can get back what amounts to an empty page, even though a visitor sees a fully built site seconds later. A late 2024 study of crawler traffic by Vercel and MERJ found that none of the major AI crawlers, including those run by OpenAI, Anthropic and Perplexity, ran JavaScript. Google's Gemini was the exception, because it uses Googlebot's rendering. It's a default outcome of a lot of modern web builds, so nobody chose it on purpose.
Third-party embedded widgets
FAQ tools are one version of this, but the same underlying issue applies to review widgets, team bio plugins, testimonial carousels, and booking or contact forms served from a third-party domain inside an iframe. In one scan, a firm had FAQ content marked up correctly with schema, the code that tells AI systems "this is a Q&A", but the actual answers loaded inside an embedded widget. The markup was right, yet no AI crawler could read the answers.
Lazy-loaded or click-triggered content
This one hides in plain sight while attention goes to robots.txt and TLS certificates. An accordion, tab, or "load more" section that only fetches its content via JavaScript when someone interacts with it is invisible to a crawler that never clicks anything. A plain CSS-hidden accordion is different. The content already sits in the page's HTML and just isn't shown by default, so it's usually fine.
The real detail living in a PDF
Several firms I've scanned had their sharpest positioning, who they serve and what makes them different, written up clearly, just not on the website itself. It lived in a PDF instead, a services brochure or fee schedule, which AI systems often can't or won't fully read, especially if it's a scanned document rather than real text.
Two basics that still trip sites up
Robots.txt blocking AI search crawlers
A site can be fully open to Googlebot while explicitly or accidentally disallowing AI crawlers in robots.txt. The ones that matter most for visibility are the search crawlers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Training crawlers like GPTBot and ClaudeBot matter less here. Blocking a search crawler is a small setting that works as an opt-out from that engine's answers, while everything else keeps working normally.
TLS or certificate errors blocking access outright
This one stops crawlers reaching the site at all, AI or otherwise, so content quality doesn't come into it. A handful of firms in one scan batch were unreachable purely because of a certificate error on their domain.
How to check your own site
Open a key page, right-click and choose "View page source" rather than "Inspect". Search for a sentence you can see on screen. If it isn't in the source, AI crawlers probably can't see it either.
Repeat this for your FAQs, reviews and any accordion or tab content.
Go to yourdomain.com/robots.txt and check whether OAI-SearchBot, Claude-SearchBot or PerplexityBot appear under a Disallow rule, or whether a rule for all crawlers ("User-agent: *") disallows everything.
A design review won't catch any of this. Most SEO audits won't either. They're built to check how a page looks and ranks for Google, not how it reads to an AI crawler that never renders it visually at all.
If you want to know whether any of these apply to your site, I run sector-specific AI visibility audits for SaaS, finance, and law firms.
Frequently asked questions
Can ChatGPT read websites built with JavaScript?
Often not. ChatGPT's crawlers read the raw HTML a server sends, and a late 2024 study by Vercel and MERJ found they didn't run JavaScript. If your content only appears once JavaScript builds the page in the browser, ChatGPT, Claude and Perplexity may see an empty page. Google's Gemini is the exception, because it uses Googlebot's rendering.
Why does my website rank on Google but not show up in ChatGPT?
Googlebot renders JavaScript, and most AI crawlers don't, so a page can be fully indexed by Google and still look empty to ChatGPT. ChatGPT's live search also relies on Bing's index, so if Bing hasn't indexed your pages, ChatGPT is unlikely to cite them. Check both the raw HTML and Bing Webmaster Tools.
Does blocking GPTBot stop my site appearing in ChatGPT?
Not necessarily. GPTBot collects content to train OpenAI's models. ChatGPT's search results use a separate crawler, OAI-SearchBot. Anthropic separates its crawlers in the same way, with ClaudeBot for training and Claude-SearchBot for search. Check each one in your robots.txt individually rather than relying on a single rule.
Can AI crawlers read content inside accordions and tabs?
It depends on how they're built. If the text is already in the page's HTML and just hidden with CSS, crawlers can usually read it. If the accordion or tab only fetches its content with JavaScript when someone clicks, AI crawlers never see it, because they don't click anything.
Does FAQ schema help if my FAQs are in a widget?
No. Schema tells AI systems that a page contains questions and answers, but the answers themselves still need to be readable in the page's HTML. If they load inside a third-party widget or iframe, the markup describes content the crawler can't reach.
How can I check whether AI can see my website content?
Open a key page, right-click and choose "View page source". Search for a sentence you can see on screen. If it isn't in the source, AI crawlers probably can't see it either. Then check yourdomain.com/robots.txt for rules that block OAI-SearchBot, Claude-SearchBot or PerplexityBot.
Comments