Direct AI crawlers with managed robots.txt

Protect your website or application from AI crawlers by implementing a robots.txt file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.

AI bots are expected to follow the robots.txt directives.

Note

Respecting robots.txt is voluntary. If you want to prevent crawling, use AI Crawl Control's manage AI crawlers feature.

Compatibility with existing robots.txt files

Cloudflare will independently check whether your website has an existing robots.txt file and update the behavior of this feature based on your website.

Existing robots.txt file

If your website already has a robots.txt file — verified by a HTTP 200 response — Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response.

For example, without this feature enabled, the robots.txt content of crawlstop.com would be:

Feature not enabled

User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest

Sitemap: https://www.crawlstop.com/sitemap.xml

With the managed robots.txt enabled, Cloudflare will prepend our managed content before your original content, resulting in what you can view at https://www.crawlstop.com/robots.txt ↗.

Feature enabled

# NOTICE: The collection of content and other data on this
# site through automated means, including any device, tool,
# or process designed to data mine or scrape content, is
# prohibited except (1) for the purpose of search engine indexing or
# artificial intelligence retrieval augmented generation or (2) with express
# written permission from this site’s operator.

# To request permission to license our intellectual
# property and/or other materials, please contact this
# site’s operator directly.

# BEGIN Cloudflare Managed content

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content
User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest


Sitemap: https://www.crawlstop.com/sitemap.xml

No robots.txt file

If your website does not have a robots.txt file, Cloudflare creates a new file with our managed block directives and serves it for you.

Implementation

To implement a robots.txt file on your domain:

Old dashboard

1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
2. Go to Security > Bots.
3. Select Configure Bot Fight Mode.
4. Turn Manage bot traffic with robots.txt on.

New dashboard

1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
2. Go to Security > Settings.
3. Filter by Bot traffic.
4. Go to Manage AI bot traffic with robots.txt.
5. Turn Manage AI bot traffic with robots.txt on.

Availability

Managed robots.txt for AI crawlers is available on all plans.