AI reputation management begins with a website AI can read.

When a buyer asks an AI assistant about your company, the answer comes from the pages the assistant can find and read. AI reputation management is the work of making that answer accurate, up to date and in your own words. It starts with AI visibility: whether AI assistants can read your website at all.

A lot of AI visibility advice is guesswork. The companies that run AI assistants publish what their crawlers do and how to let them in. This article turns that into nine checks your web team can work through, then explains what to do about the answers once AI can read your site.

Why AI reputation management matters

Assistants answer in sentences. They say what a company makes, how it compares with its rivals and whether a buyer should trust it, and they build that account from whatever they can read (we described how in The brand story buyers hear first is the one AI tells). Being named in the answer is visibility. What the answer says about you is reputation, and it is the part a buyer acts on.

The measurement tools draw the same line. Bing’s AI Performance report counts how often a site’s pages are cited in Copilot and Bing’s AI summaries, and Bing says the figures show how often content is cited, not its ranking or its placement within an answer (Microsoft Bing, February 2026). A citation count records that a page was used. What the answer said about the company is a separate question.

Google says its AI Overviews and AI Mode need no special optimization: to appear as a supporting link, a page must be indexed and eligible to show with a snippet, and even then nothing is guaranteed (Google Search Central). Its generative AI optimization guide points site owners toward content that is unique, expert or first-hand (Google Search Central, May 2026). Both statements put the weight where reputation work starts: on what your pages say, once a crawler can read them.

How AI reputation management works, in four steps

1. Read the answers. Put the questions your buyers, investors or journalists ask to several AI models, worded the way they ask them, and keep the full text of every answer. This is AI narrative analysis: reading each answer as an account, claim by claim.
2. Compare each claim with the primary record, meaning your own pages, filings and documents. A figure only matches when its unit, its condition and its date match too.
3. Trace each claim to the sources the answer drew on, including pages you do not own. As we argued in A citation is where the investigation begins, the link is where that work starts.
4. Correct at the source. Write the sentence, name the page it belongs on, publish it, and put the same questions to the same models later to see what moved, as set out in After the work, did AI’s description change?

None of these steps helps if the assistants cannot read what you publish. That is what the toolbox checks.

The AI visibility toolbox

Work through the checks in order, since there is no point rewriting a page a crawler cannot fetch. The table matches common symptoms to the check that usually explains them.

1. Let the search crawlers through robots.txt

Each AI company runs several bots with different jobs. Some fetch pages to train models, some index pages for the assistant’s search, and some fetch a page because a user asked about it. Blocking a training bot does not remove you from the assistant’s search. Blocking its search bot does.

OpenAI says each of its settings is independent, so a site can block GPTBot and still allow OAI-SearchBot, and that sites which opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI). Anthropic says blocking Claude-SearchBot prevents indexing for search and may reduce a site’s visibility in Claude’s results (Anthropic). Perplexity describes PerplexityBot as the crawler that surfaces sites in its search and says it is not used to train foundation models (Perplexity). Google-Extended governs Gemini training and grounding, and Google says it has no effect on a site’s inclusion in Search, which covers AI Overviews and AI Mode (Google). Microsoft documents that Copilot Chat grounds its answers in the Bing search service (Microsoft Learn). Apple says Applebot-Extended crawls nothing and only decides whether Applebot’s data may train Apple’s models (Apple). Meta lists meta-webindexer as the crawler behind Meta AI search results, and meta-externalagent as one that both trains models and indexes content (Meta). The bots that fetch a page because a user asked, such as ChatGPT-User and Perplexity-User, may not follow robots.txt at all.

A robots.txt file that keeps the main training bots out and lets search crawlers in can look like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Whether to allow training is a business decision. The search crawlers are the ones that decide whether you can appear in the answers. A few details catch teams out. robots.txt manages crawling and is not access control, and a page blocked there can still be indexed without a description if other pages link to it; to keep a page out of results, use noindex on a page the crawler is allowed to fetch (IETF, RFC 9309; Google Search Central). Google generally caches robots.txt for up to a day (Google), and OpenAI and Perplexity say a change can take about a day to apply. Anthropic’s rules apply per subdomain, so a blog on its own subdomain needs its own file.

2. Check what your CDN or firewall decides

A robots.txt file can invite a crawler in while the network in front of the site turns it away. Cloudflare has changed its defaults twice. Since July 2025, new domains are asked at sign-up whether to allow AI crawlers (Cloudflare, July 2025). In July 2026 it split AI traffic into Search, Agent and Training, and since September 15, 2026, new domains block Training and Agent by default on pages that show ads (Cloudflare, July 2026).

The settings now sit under AI Crawler and Agent access policies, and Cloudflare warns that blocking Training also blocks crawlers that do several jobs at once, such as Googlebot, Applebot and Bingbot (Cloudflare). Its managed robots.txt, when switched on, adds Disallow rules for GPTBot, ClaudeBot, Google-Extended and other training bots to your own file (Cloudflare). Bot Fight Mode on the Free plan challenges traffic that looks automated across the whole domain and cannot be skipped with custom rules (Cloudflare). AI Crawl Control lists AI crawlers one by one, with an allow or block setting for each, and tracks whether they follow your robots.txt (Cloudflare).

Other CDNs and firewalls have their own bot rules, and the check is the same: read your logs for the search crawlers by name and confirm they receive pages, not challenges. OpenAI, Anthropic and Perplexity each publish the IP ranges their bots use, and many firewalls identify bots by those ranges, so a test with a borrowed user agent shows only part of the picture.

3. Serve the content in the first HTML response

In Vercel and MERJ’s measurement of crawler traffic, the crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity did not execute JavaScript, while Gemini, through Googlebot, and Applebot rendered pages fully (Vercel, December 2024). We found no newer measurement from a crawler operator. A page that builds its text in the browser can therefore reach Google intact and arrive empty everywhere else. A page an AI crawler cannot fetch, or fetches as an empty shell, plays no part in the answer.

Google’s own guidance points the same way. It renders JavaScript in a separate phase after crawling, and still calls server-side rendering or pre-rendering a great idea because not all bots can run JavaScript (Google Search Central). It describes dynamic rendering, which serves bots a different version of the page, as a workaround and not a long-term solution, and recommends server-side rendering, static rendering or hydration instead (Google Search Central).

The check takes a minute. Request the page without a browser, for example with curl -s https://www.example.com/page, or open the page source rather than the rendered page, and search it for a sentence buyers need. If the sentence is not in that response, assume most AI crawlers do not see it.

4. Keep the sitemap and its dates honest

A sitemap tells crawlers which pages exist and when each one last changed. Google uses the lastmod date only if it is consistently and verifiably accurate, and ignores the priority and changefreq fields (Google Search Central). It has also said it may stop trusting a site’s lastmod values when they do not match real changes (Google, June 2023). Bing calls lastmod a key signal for deciding what to recrawl, asks for dates in ISO 8601 format with a time, and names Copilot in the same guidance (Microsoft Bing, July 2025).

The common failure is a sitemap that stamps every page with the date of the last deploy. Each release then claims that every page changed, and crawlers learn to ignore the dates. Give each page its own date and move it only when that page’s content changes. Bing also accepts IndexNow, a notice a site sends when a URL is added, updated or deleted, which is shared among the engines that support it, including Bing, Naver, Seznam.cz, Yandex and Yep; Google is not on the list (IndexNow).

5. Give each page one clear H1 and headings that say something

Google says heading order does not matter for Search and that there is no ideal number of headings (Google Search Central). Its generative AI guide recommends headings that give content a clear structure and calls semantic HTML generally a good idea. The case for a tidy hierarchy is about meaning more than rules: one H1 that names what the page is about, H2s for the questions a buyer asks, and H3s for the parts of each answer.

A heading such as “Lab reports for every tested lot” gives a model a label it can repeat along with the section beneath it. “Our commitment to quality” gives it nothing to repeat. Write headings a buyer could search for and a model could quote.

6. Keep titles, descriptions and dates accurate, then leave them alone

Google builds the title shown in its results from the page’s title element, its main heading and other prominent text, and asks for titles that are descriptive, concise and unique to each page (Google Search Central). It uses the meta description when that describes the page better than the text does (Google Search Central). For dates, it asks for a visible date that matches the publication and modification dates in the page’s structured data (Google Search Central), and its guidance on helpful content asks whether a site changes dates to seem fresh when the content has not substantially changed (Google Search Central).

In practice, keep one record of each page’s title, description and modified date, and have the page, its structured data and the sitemap all read from it, so the three cannot disagree. Change the date when the words change, and only then. That is how this site is run.

7. Check that you have not opted out of AI features

Some controls remove a site from AI answers on purpose, and they are easy to set by accident. Google’s nosnippet rule also stops a page’s content from being used directly in AI Overviews and AI Mode, and max-snippet limits how much of it can be used (Google Search Central). Since August 2026, every site can also switch off its appearance in AI Overviews and AI Mode from Search Console (Google, June 2026). If other assistants cite you and Google’s AI features never do, check these settings before rewriting anything. Microsoft has announced page-level grounding preferences for Bing, which it will honor from February 1, 2027 (Microsoft Bing, October 2026).

8. Leave the folklore aside

Two ideas circulate widely without support from the companies that run the assistants. The first is llms.txt, a proposed Markdown file that lists a site’s key pages for language models (llmstxt.org). Google’s generative AI guide says Google Search does not use it and that creating one neither helps nor harms visibility. As of November 2026 we found no statement from OpenAI, Anthropic, Perplexity or Microsoft that their assistants read it either. Keeping one does no harm.

The second is rewriting pages for machines. The same Google guide lists special files, splitting content into chunks and rewriting for AI among its myths.

9. Write pages worth repeating

The checks above make a site readable. What the assistants repeat depends on what the pages say. Google asks for content that is unique, expert or first-hand, and its helpful content questions ask who wrote a page, how and why. Bing’s suggestions for its AI answers are similar: clear headings, tables and FAQ sections, claims supported by evidence, and names used consistently (Microsoft Bing, February 2026).

For a company page, that means stating each claim the way a buyer would check it: the figure with its unit, the condition it holds under and the date it was true. “28g of protein in a 34.9g serving” can be repeated and compared. “Industry-leading protein” cannot.

What the toolbox cannot fix

Once the assistants can read your site, the question becomes what they say. In our whey protein study, all three brands appeared in every model’s answers, yet the claims the models repeated set them apart, and a buyer reading the answers could not compare our client’s protein per serving with a rival’s. No crawler setting changes that. It takes reading the answers claim by claim, tracing each claim to its sources and writing the sentence that should replace it. We set out the difference in AI visibility isn’t the fix for your reputation, and the steps in how our AI reputation management works.

We read these answers for companies, products and the people who lead them, across the latest AI models. The first consultation is free and includes a sample reading of your own website. Tell us what AI should say about you.

Sources

1. Microsoft Bing, Introducing AI Performance in Bing Webmaster Tools Public Preview, February 10, 2026.
2. Google Search Central, AI features and your website.
3. Google Search Central, Optimizing your website for generative AI features on Google Search, May 15, 2026.
4. OpenAI, Overview of OpenAI Crawlers.
5. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?.
6. Perplexity, Perplexity Crawlers.
7. Google, List of Google’s common crawlers.
8. Microsoft Learn, Data, privacy, and security for web search in Microsoft Copilot and Microsoft Copilot Chat, October 29, 2026.
9. Apple, About Applebot, September 4, 2026.
10. Meta, Meta Web Crawlers.
11. Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, RFC 9309: Robots Exclusion Protocol, IETF, September 2022.
12. Google Search Central, Introduction to robots.txt.
13. Google, How Google interprets the robots.txt specification.
14. Matthew Prince, Cloudflare, Content Independence Day: no AI crawl without compensation, July 1, 2025.
15. Cloudflare, Your site, your rules: new AI traffic options for all customers, July 1, 2026.
16. Cloudflare, AI Crawler and Agent access policies.
17. Cloudflare, robots.txt setting.
18. Cloudflare, Bot Fight Mode.
19. Cloudflare, AI Crawl Control.
20. Vercel and MERJ, The rise of the AI crawler, December 17, 2024.
21. Google Search Central, Understand the JavaScript SEO basics.
22. Google Search Central, Dynamic rendering as a workaround.
23. Google Search Central, Build and submit a sitemap.
24. Gary Illyes, Google, Sitemaps ping endpoint is going away, June 26, 2023.
25. Fabrice Canel and Krishna Madhavan, Microsoft Bing, Keeping Content Discoverable with Sitemaps in AI Powered Search, July 31, 2025.
26. IndexNow, FAQ.
27. Google Search Central, Search Engine Optimization (SEO) Starter Guide.
28. Google Search Central, Influencing your title links in search results.
29. Google Search Central, Control your snippets in search results.
30. Google Search Central, Influence your byline dates in Google Search.
31. Google Search Central, Creating helpful, reliable, people-first content.
32. Google Search Central, Robots meta tag, data-nosnippet and X-Robots-Tag specifications.
33. Mrinalini Loew, Google, New opportunities, control and insights for website owners, June 3, 2026, updated August 31, 2026.
34. Krishna Madhavan, Microsoft Bing, Your content, your choice: clearer preferences for training and grounding, October 30, 2026.
35. Jeremy Howard, The /llms.txt file, v2, first proposed September 3, 2024.