AI website scan

See how a site treats AI crawlers like GPTBot, ClaudeBot and PerplexityBot, from its robots.txt and llms.txt.

All tools
Some AI access is restricted

13 of 13 AI crawlers allowed.

Scanned example.com

Findings

  • No robots.txt was found at example.com. Every AI crawler is allowed by default — add one if you want to control that.

    Warning
  • All 13 major AI crawlers can access this site, so it can appear in AI answers and citations.

    Pass
  • No llms.txt found. This optional file points AI models at the pages you most want them to read.

    Info
  • No markdown response detected. Serving markdown to AI crawlers gives them your content without the surrounding HTML clutter.

    Info

AI crawlers

13 of 13 allowed
  • OpenAI GPTBot default

    Trains ChatGPT models on your content.

    Allowed
  • OpenAI SearchBot default

    Indexes pages for ChatGPT Search results.

    Allowed
  • ChatGPT-User default

    Fetches pages live when a user asks ChatGPT about them.

    Allowed
  • Anthropic ClaudeBot default

    Crawls content for Claude.

    Allowed
  • Claude-User default

    Fetches pages live for a Claude user request.

    Allowed
  • anthropic-ai default

    Legacy Anthropic crawler token.

    Allowed
  • Google-Extended default

    Gates use of your content in Gemini and AI Overviews.

    Allowed
  • PerplexityBot default

    Indexes pages for Perplexity answers and citations.

    Allowed
  • Common Crawl (CCBot) default

    Open dataset many AI models are trained on.

    Allowed
  • Applebot-Extended default

    Gates use of your content in Apple Intelligence.

    Allowed
  • Amazonbot default

    Crawls content for Amazon AI products such as Alexa.

    Allowed
  • Bytespider default

    ByteDance/TikTok crawler used for AI training.

    Allowed
  • Meta AI default

    Crawls content for Meta AI models.

    Allowed

default means robots.txt does not name the crawler, so it inherits the site wide rule.

Why AI crawler access matters

Generative AI engines (ChatGPT, Claude, Perplexity, Google's AI Overviews) each crawl the web with their own bot. When a user asks one of them a question, your pages can only be quoted or cited if that bot was allowed to read them. Blocking them keeps your content out of training sets, but it also keeps you out of the answers more and more people start their research with.

How the scan works

We fetch your site's robots.txt and resolve the effective rule for every well known AI crawler. A bot named explicitly wins over the site wide * rule, exactly as RFC 9309 specifies. We also look for an llms.txt, the emerging convention for handing AI models a curated map of the content you most want them to read. Finally we ask the homepage for text/markdown to see whether the site serves AI agents a clean markdown rendering instead of full HTML.