Robots.txt and AI crawlers: what to allow, what to check

Keep three decisions separate

Crawling concerns fetching a URL. Indexing concerns whether a search engine stores a URL for its results. Access control concerns who is actually allowed to retrieve information. These decisions can require different controls.

Google explains that a URL blocked by robots.txt can still appear in results if discovered elsewhere. A noindex directive must be accessible to the crawler that needs to read it; blocking the crawl can prevent that. Private information belongs behind authentication or another real access restriction. Read Google’s robots.txt guide.

Know which OpenAI agent you are addressing

OpenAI documents independent controls: OAI-SearchBot supports discovery in ChatGPT search, while GPTBot crawls material that may be used for model training. ChatGPT-User is associated with user-requested visits rather than automatic search crawling; robots rules may not apply to those actions. See the current crawler documentation before changing a policy.

Allowing a search crawler is a discovery choice, not a promise of a citation. A rule for one named agent does not define policy for every AI company or crawler.

Review a specific example, not a universal template

This illustrative fragment expresses one possible choice: allow OpenAI search crawling while declining GPTBot training crawling.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Do not replace an existing file with this fragment. First inspect its other groups, path restrictions, sitemap declarations, and the rules for each relevant host. An explicit crawler group may need the same public-path restrictions you intended elsewhere. A permissive example is inappropriate if it contradicts your existing policy.

Check the complete path to your content

1. Save the current robots.txt response and record why you are changing it.
2. Identify the exact scheme and host: the main domain, a subdomain, and a separate protocol can have different configurations.
3. Review representative public and restricted paths against the intended crawler group.
4. Check redirects, response status, caching, CDN rules, and firewall behavior. A permissive file cannot fix a blocked request elsewhere.
5. After release, fetch the public file again and inspect your important pages with the relevant search-engine tools.

Do not treat a bot-like user-agent string alone as proof of a crawler’s identity. Where verification matters, use the provider’s published verification guidance.

Make the change easy to reverse

Keep the previous version, the responsible owner, the release time, and the expected effect in a change log. Recheck a known public page and a genuinely private route after the release. The private route should still require authorization regardless of the crawler name.

If visibility changes later, investigate the whole chain instead of immediately blaming the robots file. Content changes, canonical selection, downtime, and demand can matter too. Finish this task with a documented policy and tested configuration; do not sell it internally as an instant ranking improvement.