Who is using Cloudflare and blocking AI bots? Should you block them or not?
Currently I have:
- Manage your robots.txt on, with "Instruct AI bots to not scrape content creates or updates your robots.txt file to signal that your content should not be used for AI training." On
- Control AI crawlers > Do not block (allow crawlers)
Though when I look at my robots.txt file, it says "Disallow: /" to the ai bots. Confused