Are business websites ready for Ai search?

Key findings

• 27% of reachable sites turned away our automated checker with an error or a bot-challenge page. Security filters, not robots.txt, are the most common technical barrier.
• Only 1.1% of robots.txt files block an Ai search crawler (OAI-SearchBot, PerplexityBot or Claude-SearchBot). 4.5% block at least one training crawler, usually while still allowing search.
• 40% of homepages that loaded have no Organization or business structured data, so engines have to guess who the business is.
• 21% have no H1 heading and 19% have no meta description, two of the first things engines read.
• 23% already publish an llms.txt file, many of them created automatically by SEO plugins, even though major engines haven’t said they use it.
• Only 34% of reachable sites pass all five basics we tested. Law firms scored best (55%); breweries and wineries scored lowest (19%).

Why we ran this study

When someone asks ChatGPT, Perplexity, Claude, Gemini or Google’s Ai Overviews for a recommendation, the engine usually searches the web first and reads the pages it finds. A business can only be cited if those engines can reach its site and understand who it is. We wanted to know how often that basic technical layer is in place, across industries, using the same checks our free Ai visibility checker runs.

This study measures technical readiness only. Whether an engine actually recommends a business depends far more on reputation across the web: reviews, brand mentions and backlinks. We cover that side in our guide to how ChatGPT recommends businesses.

Finding 1: security filters turn away automated visitors

Of the 1,144 sites we could reach, 27% answered our checker with an error or a bot-challenge page instead of the homepage. 17% served a challenge page (“Just a moment…”, a CAPTCHA or similar), and 16% returned an error status such as 403 Forbidden or 429 Too Many Requests. 42% of the sites we reached sit behind Cloudflare.

Our checker identifies itself honestly, requests one page at a time and runs from a cloud server, much like many Ai fetchers. This is not proof that a given Ai engine is blocked: security services often allowlist verified crawlers from Google, Bing and the larger Ai companies. But it shows how many sites sit behind filters that decide, without the owner looking, which automated visitors get through. In July 2025 Cloudflare also began blocking Ai bots by default for sites it serves, with settings owners can change.

What to do: open your CDN or security plugin and check how it treats Ai crawlers, specifically the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot). Our checker tests this by requesting your homepage as each of them.

Finding 2: robots.txt is rarely the problem

Much of the advice about Ai visibility starts with robots.txt. In our sample it is rarely the issue. Of the 958 sites with a readable robots.txt file, just 1.1% block an Ai search crawler from the homepage. Blocks on training crawlers are more common but still unusual: 4.5% block at least one, and 3.4% block training while leaving search crawlers allowed, which is the setup we recommend for owners who don’t want their content used for training.

Two smaller signals stand out. 16% of robots.txt files name at least one Ai crawler, so owners are starting to manage them deliberately. And 7.7% of reachable sites have no robots.txt at all, while another 8.6% served it behind a challenge page, which means crawlers that respect robots.txt can’t read the rules, or the sitemap location, before they visit.

Finding 3: many businesses don’t say who they are in code

Structured data is how a site tells machines, in a standard format, which business it belongs to: its name, logo, address, phone, opening hours and linked profiles. Of the 834 homepages that loaded, 40% have no Organization or business-type structured data, and 29% have no JSON-LD structured data of any kind. Among sites that do mark up their business, 30% leave out sameAs links to their other profiles, the field that connects a website to its review, social and directory listings.

Engines can still work out who a business is from the page text, but clear, consistent structured data removes the guesswork. It is also one of the cheapest fixes on this list. See structured data in our glossary.

Finding 4: basic page signals are often missing

The title, meta description and main heading are among the first things any search or Ai engine reads to understand a page. They are often missing:

9.4% of homepages had fewer than 150 words of text in the HTML itself. On those sites, most of the content appears only after JavaScript runs, and many Ai crawlers don’t run JavaScript. Only 0.7% of homepages were set to noindex, so accidental de-indexing is rare in this sample.

Finding 5: llms.txt is spreading faster than its use

llms.txt is a proposed file that summarizes a website for language models. 23% of the reachable sites in our sample already publish one. Many carry a note showing they were created automatically by an SEO plugin such as Yoast or All in One SEO. Major engines haven’t confirmed they use it, and Google says it isn’t needed for its Ai features in Search, so our checker treats it as optional and doesn’t score it. It is harmless to have, but it is no substitute for the basics above.

How industries compare

We scored each reachable site on five basics: the homepage loads for an automated visitor, robots.txt allows the Ai search crawlers and the main search engines, the homepage has business structured data, it has a title, meta description and H1, and an XML sitemap can be found. Only 34% of reachable sites pass all five (46% of the sites whose homepage loaded).

Law firms and real estate companies come out ahead, mostly because more of them mark up their business in structured data (81% and 73% of homepages that loaded). Retailers, hotels and restaurants were the most likely to turn away our checker (37%, 35% and 32% of reachable sites). Breweries and wineries trailed on the basics: only 47% of their homepages have business structured data, and 33% have no meta description.

Methodology

Sample. We drew the sample from Wikidata, the open knowledge base: organizations located in the United States that have an official website on record and no recorded dissolution date, in eight industry groups. We randomly selected up to 160 unique domains per group (fixed seed 20261022; all 88 unique domains for real estate companies, which had fewer entries). That gave 1,208 websites. Because Wikidata favors organizations notable enough to be recorded, the sample leans toward established businesses; small local businesses may look different.

Checks. On October 22, 2026 we checked every homepage with the same rules as our free Ai visibility checker: robots.txt parsed under RFC 9309 for 16 crawlers, the homepage’s robots meta tags and X-Robots-Tag header, title, meta description, H1, readable text, JSON-LD and microdata structured data, Open Graph tags, contact links, XML sitemap and llms.txt. Requests came from a cloud server, identified as our checker with a browser user agent, with a 10-second timeout.

What we didn’t do. We did not run the checker’s firewall test, which requests the homepage while identifying as each Ai search crawler; from a server that isn’t the real crawler, that would not be a fair test at scale. We checked homepages only, not every page. 64 sites (5.3%) could not be reached at all and are excluded from every percentage. Robots.txt figures use the 958 sites with a readable file; homepage figures use the 834 homepages that loaded.

Limits. This is a snapshot of one day. Sites change, and security tools may treat a real crawler differently from our checker. The figures describe technical readiness only; they say nothing about whether engines recommend these businesses.

What to do about it

1. Check your security layer. Make sure your CDN, firewall or security plugin lets verified search and Ai search crawlers through.
2. Keep Ai search crawlers allowed in robots.txt. Block training crawlers separately if you prefer; it doesn’t affect search answers.
3. Add business structured data. Organization or the right LocalBusiness type, with your name, logo, address, phone and sameAs links to your profiles.
4. Fix the basics. One clear H1, a descriptive title and meta description, and key information in plain HTML text.
5. Publish a sitemap and list it in robots.txt.
6. Then work on reputation. Reviews, brand mentions and backlinks decide who gets recommended once engines can read you.

Our free Ai visibility checker tests your own site against the same rules in about 20 seconds.

How to cite this study

You’re welcome to quote these findings. Please credit Ai Boutique and link to this page.

Journalists and researchers can contact us for the aggregate figures by industry or for questions about the method.