AI Crawler Access Checker
Find out if your website is visible to AI search engines. Enter your domain to see which AI crawlers your robots.txt allows or blocks.
Enter your domain without https:// — e.g. acmeplumbing.com.au
Enter your domain above to check which AI crawlers can access your site.
Want AI crawlers to find your content?
We help businesses show up in AI search results — not just Google. Let’s look at your site’s AI visibility together.
How to allow AI crawlers in your robots.txt
If the checker flagged a blocked crawler, here is exactly what to add to your robots.txt file. You can edit it directly in your CMS or ask your developer to update the file at the root of your domain.
Allow a specific crawler
Copy and paste the relevant block for whichever bot you want to allow:
User-agent: GPTBot
Disallow:
User-agent: ClaudeBot
Disallow:
User-agent: PerplexityBot
Disallow:
User-agent: Google-Extended
Disallow:An empty Disallow: line means the crawler is allowed. No value after it is not a typo.
Allow all AI crawlers at once
If you want to open your site to everything, add this to the top of your robots.txt (before any other User-agent rules):
User-agent: *
Disallow:This allows every bot, including AI crawlers, to access your entire site. If you already have specific rules for Googlebot or other crawlers below it, those will still apply.
Block a specific crawler only
If you want to block one crawler and allow everything else:
User-agent: GPTBot
Disallow: /Disallow: / blocks the crawler from your entire site. You can also block specific folders – for example Disallow: /members/ to protect gated content while keeping the rest of your site open.
AI crawler FAQs
It is a text file that sits at the root of your website (e.g. yourdomain.com/robots.txt). It tells bots which pages they can and cannot access. It was originally built for search engine crawlers like Googlebot, but AI platforms including ChatGPT, Perplexity and Claude now use it too.
The major ones do. OpenAI, Anthropic, Google and Perplexity have all confirmed their crawlers respect robots.txt rules. Smaller or less reputable bots may not, but they are not the ones powering the platforms your audience is using.
No robots.txt means all bots are allowed by default, including AI crawlers. It is not a problem for AI visibility, but it is worth adding one to give you more control over what gets indexed and what does not.
No. Google’s main crawler (Googlebot) is separate from Google-Extended, which is the one that feeds Gemini and AI Overviews. Blocking Google-Extended does not affect your organic rankings. But it does remove you from AI-generated answers, which is increasingly where users find information.
Sometimes. Publishers protecting paywalled content, law firms with confidential documents, or businesses that have not yet decided on their AI content strategy may choose to block crawlers while they figure it out. The important thing is that it is an active decision, not something that happens by accident from an old robots.txt setting.
Any time you rebuild your site, switch CMS platforms, or migrate to a new domain. A lot of robots.txt blocks come from a developer setting Disallow: / during a staging build and forgetting to change it before launch. It is a two-minute check worth doing.
Ready to hit the mark?
Let’s get started!
Ready to take your business to the next level? Let DNM Digital’s expert team create a tailored strategy that drives results. Contact us today and start seeing real, measurable growth!
