AI website scan
See how a site treats AI crawlers like GPTBot, ClaudeBot and PerplexityBot, from its robots.txt and llms.txt.
DNS & Email
MX lookup See which mail servers accept mail for a domain, in priority order. DNS lookup Look up any record across several public resolvers and see if a change has propagated. DMARC checker Check the DMARC policy that tells receivers what to do with spoofed mail. DKIM checker Find and validate the DKIM key that signs mail for a domain. SPF checker Validate the SPF record and count its DNS lookups against the limit of 10. CAA checker See which certificate authorities may issue certificates for a domain. MTA-STS checker Check that mail to a domain is always delivered over an encrypted connection. Blacklist checker Find out if a server IP or domain is listed on spam blocklists. Deliverability scan Check SPF, DKIM and DMARC in one pass, with a graded breakdown of what to fix. WHOIS lookup See the registrar, name servers and expiry date of any domain.Performance
Status code checker Follow every redirect and see the status code of each hop. HTTP version checker See which HTTP versions a site speaks: HTTP/1.1, HTTP/2 and HTTP/3. HTTP header checker Inspect the response headers a server sends back. TTFB checker Measure time to first byte, split into DNS, TCP, TLS and server time. Port checker Check whether a port on a server is open and reachable. Ping Measure latency and packet loss to any host, from our servers. Traceroute Follow every router between our servers and a host to spot where latency creeps in. What is my IP? See the public IP address your connection is using right now. What is my user agent? See the user agent your browser sends and what it says about you.Security
HSTS checker Check that a site forces HTTPS and is ready for the preload list. SSL checker Check a certificate, its chain and when it expires. Security headers scanner Grade the security headers a site sends, with a fix for each one. Internet.nl scan Score a site on modern internet standards like IPv6, DNSSEC and HTTPS.SEO
AI website scan See how AI crawlers read a site and what they are allowed to use. Indexability checker Find out if a page can be indexed, and what blocks it if not. robots.txt Tester Test which URLs a robots.txt allows or blocks for each crawler. Sitemap validator Validate an XML sitemap against the sitemaps.org protocol and its limits. OG image preview Preview how a link looks when it is shared on social media.Nothing matches “”
1 of 13 AI crawlers allowed.
Scanned nytimes.com
Findings
-
Pass
A robots.txt was found and parsed.
-
Warning
12 of 13 AI crawlers are blocked: OpenAI GPTBot, OpenAI SearchBot, ChatGPT-User, Anthropic ClaudeBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Common Crawl (CCBot), Applebot-Extended, Bytespider, Meta AI.
-
Info
No llms.txt found. This optional file points AI models at the pages you most want them to read.
-
Info
No markdown response detected. Serving markdown to AI crawlers gives them your content without the surrounding HTML clutter.
AI crawlers
1 of 13 allowed-
Blocked
OpenAI GPTBot
Trains ChatGPT models on your content.
-
Blocked
OpenAI SearchBot
Indexes pages for ChatGPT Search results.
-
Blocked
ChatGPT-User
Fetches pages live when a user asks ChatGPT about them.
-
Blocked
Anthropic ClaudeBot
Crawls content for Claude.
-
Blocked
Claude-User
Fetches pages live for a Claude user request.
-
Blocked
anthropic-ai
Legacy Anthropic crawler token.
-
Blocked
Google-Extended
Gates use of your content in Gemini and AI Overviews.
-
Blocked
PerplexityBot
Indexes pages for Perplexity answers and citations.
-
Blocked
Common Crawl (CCBot)
Open dataset many AI models are trained on.
-
Blocked
Applebot-Extended
Gates use of your content in Apple Intelligence.
-
Allowed
Amazonbot
Crawls content for Amazon AI products such as Alexa.
-
Blocked
Bytespider
ByteDance/TikTok crawler used for AI training.
-
Blocked
Meta AI
Crawls content for Meta AI models.
default means robots.txt does not name the crawler, so it inherits the site wide rule.
robots.txt
# New York Times content is made available for your personal, non-commercial
# use subject to our Terms of Service here:
# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.
# Use of any device, tool, or process designed to data mine or scrape the content
# using automated means is prohibited without prior written permission from
# The New York Times Company. Prohibited uses include but are not limited to:
# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in
# the Digital Single Market;
# (2) the development of any software, machine learning, artificial intelligence (AI),
# and/or large language models (LLMs);
# (3) creating or providing archived or cached data sets containing our content to others; and/or
# (4) any commercial purposes.
# Contact https://nytlicensing.com/contact/ for assistance.
User-agent: *
User-agent: Googlebot
Disallow: /ads/
Disallow: /adx/bin/
Disallow: /athletic/wp/wp-admin/
Allow: /athletic/wp/wp-admin/admin-ajax.php
Disallow: /athletic/async-*
Disallow: /athletic/search/*
Allow: /athletic/search/$
Disallow: /athletic/checkout/
Disallow: /athletic/checkout?plan_id*
Allow: /athletic/checkout/$
Disallow: /athletic/checkout2*
Disallow: /athletic/login/
Disallow: /athletic/login?login_source*
Disallow: /athletic/login?ref_page*
Allow: /athletic/login/$
Disallow: /athletic/login2/
Disallow: /athletic/login2?login_source*
Disallow: /athletic/login2?ref_page*
Allow: /athletic/login2/$
Disallow: /athletic/report/
Disallow: /athletic/*/discuss/*
Allow: /athletic/live-blogs/discuss/
Allow: /athletic/nfl/game/discuss/
Disallow: /athletic/register/
Disallow: /athletic/register?welcome_redirect*
Disallow: /athletic/register2/
Disallow: /athletic/register2?welcome_redirect*
Disallow: /athletic/betmgm-redirect*
Disallow: /athletic/cdn-cgi/
Disallow: /athletic/verizon/*
Disallow: /athletic/forgot-password/*
Disallow: /athletic/forgot-password2/*
Disallow: /athletic/amp-social-login*
Disallow: /athletic/track-analytics/
Disallow: /athletic/amp-auth/
Disallow: /athletic/rss-feed/
Disallow: /athletic/*?*rss=1
Disallow: /athletic/global-color-test.php
Disallow: /athletic/global-font-test.php
Disallow: /athletic/graphql*
Disallow: /athletic/api*
Disallow: /athletic/ip*
Disallow: /athletic/call-set-cookie-with-context/*
Disallow: /athletic/get-current-user/
Disallow: /athletic/pv.json
Disallow: /athletic/following-feed-test/*
Disallow: /athletic*/boxscore/*
Disallow: /athletic/feed-test/
Disallow: /athletic*/signed-mp3-redirect-url/*
Disallow: /athletic/embedded-interactive/*
Disallow: /card/panel/
Disallow: /panel/
Disallow: /puzzles/leaderboards/invite/*
Disallow: /svc
Allow: /svc/crosswords
Allow: /svc/games
Allow: /svc/letter-boxed
Allow: /svc/spelling-bee
Allow: /svc/wordle
Allow: /svc/connections
Allow: /svc/sudoku
Allow: /svc/strands
Allow: /svc/pips
Disallow: /video/embedded/*
Disallow: /search
Disallow: /multiproduct/
Disallow: /hd/
Disallow: /inyt/
Disallow: /*?*query=
Disallow: /*.pdf$
Disallow: /*?*login=
Disallow: /*?*campaignId=
Disallow: /*?*mcubz=
Disallow: /*?*smprod=
Disallow: /*?*ProfileID=
Disallow: /*?*ListingID=
Disallow: /*?*campaign_id=
Disallow: /*?*hybrid=
Disallow: /*?*entry=
Disallow: /*?*embed=
Disallow: /*?ls=
Disallow: /*?*&ls=
Disallow: /v1/initialize*
Disallow: /v1/rgstr*
Disallow: /wirecutter/wp-admin/
Disallow: /wirecutter/*.zip$
Disallow: /wirecutter/*.csv$
Disallow: /wirecutter/deals/beta
Disallow: /wirecutter/data-requests
Disallow: /wirecutter/*?s=
Disallow: /wirecutter/*&xid=
Disallow: /wirecutter/*?q=
Disallow: /wirecutter/*?l=
Disallow: /wirecutter/*?merchant=
Disallow: /wirecutter/out/
Disallow: /search
Disallow: /subscription/*?*source=
Disallow: /subscription/*?*onboarded=
Disallow: /*?*smid=
Disallow: /*?*partner=
Disallow: /*?*utm_source=
Allow: /wirecutter/search
Allow: /wirecutter/gifts/search
Allow: /wirecutter/*?*utm_source=
Allow: /ads/public/
Allow: /svc/news/v3/all/pshb.rss
Allow: /wirecutter/reviews/*?*utm_source=
Allow: /wirecutter/blog/*?*utm_source=
Allow: /wirecutter/lists/*?*utm_source=
Allow: /wirecutter/gifts/*?*utm_source=
# Googlebot Specific Rules
User-agent: Googlebot
Disallow: /athletic*adgroupid*
Disallow: /athletic*campaignid*
Disallow: /athletic*ad_id*
Disallow: /athletic*access_token*
Disallow: /athletic*amp_reader_id*
Disallow: /athletic*/?source=*
Disallow: /athletic/*?*embed=1
# Disallow Rules
User-agent: AliyunSecBot
Disallow: /
User-agent: Amazonbot
Disallow: /wirecutter/
User-agent: anthropic-ai
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: AudigentAdBot
Disallow: /
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Brightbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Diffbot-User
Disallow: /
User-agent: DuckAssistBot
Disallow: /
User-agent: EchoboxBot
Disallow: /
User-agent: ExaBot
Disallow: /
User-agent: ExaSearchBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: FirecrawlAgent
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: Google-CloudVertexBot
Disallow: /
Allow: /wirecutter/
Allow: /athletic/
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Jetslide
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Disallow: /
User-agent: Meta-WebIndexer
User-agent: meta-webindexer/1.1
Disallow: /
User-agent: MyCentralAIScraperBot
Disallow: /
User-agent: NewsNow
Disallow: /
User-agent: news-please
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: peer39_crawler
User-agent: peer39_crawler/1.0
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Poseidon Research Crawler
Disallow: /
User-agent: quillbot.com
Disallow: /
User-agent: Quora-Bot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: SeekrBot
Disallow: /
User-agent: SeznamHomepageCrawler
Disallow: /
User-agent: ShapBot
Disallow: /
User-agent: TaraGroup Intelligent Bot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ViennaTinyBot
Disallow: /
User-agent: YouBot
Disallow: /
# Ad Bot Rules
User-agent: AmazonAdBot
Allow: /
# Social Bot Rules
User-agent: facebookexternalhit
Allow: /*?*smid=
User-agent: Facebot
Allow: /*?*smid=
User-agent: Twitterbot
Allow: /*?*smid=
User-agent: RedditBot
Allow: /*?*smid=
User-agent: LinkedInBot
Allow: /*?*smid=
User-agent: Bluesky
Allow: /*?*smid=
User-agent: TelegramBot
Allow: /*?*smid=
User-agent: WhatsApp
Allow: /*?*smid=
User-agent: Discordbot
Allow: /*?*smid=
User-agent: Mastodon
Allow: /*?*smid=
# Sitemaps
Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/sitemap.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/collections.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/video.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/cooking.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/regions.xml
Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml
Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml
Sitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/espanol.xml.gz
Sitemap: https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz
Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-authors.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-verticals.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-teams.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-cities.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-players.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-tags.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-stats.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-schedule.xml
Sitemap: https://www.nytimes.com/athletic/sitemap-roster.xml
Sitemap: https://www.nytimes.com/athletic/sitemap.xml
Sitemap: https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml
Why AI crawler access matters
Generative AI engines (ChatGPT, Claude, Perplexity, Google's AI Overviews) each crawl the web with their own bot. When a user asks one of them a question, your pages can only be quoted or cited if that bot was allowed to read them. Blocking them keeps your content out of training sets, but it also keeps you out of the answers more and more people start their research with.
How the scan works
We fetch your site's robots.txt and resolve the effective rule for every well known
AI crawler. A bot named explicitly wins over the site wide * rule, exactly as RFC 9309
specifies. We also look for an llms.txt, the emerging convention for handing AI models
a curated map of the content you most want them to read. Finally we ask the homepage for
text/markdown to see whether the site serves AI agents a clean markdown rendering
instead of full HTML.