Cloudflare is now blocking all major LLMs by default (ChatGPT, Claude, Perplexity, etc). Since over 20% of the internet is behind Cloudflare, a lot of sites are already excluded from AI summaries unless they've explicitly allowed those bots. - Default Blocking: Cloudflare has recently set its AI crawler protections to be turned on by default for new domains. - Permission-Based Model: This means that AI companies are now required to obtain explicit permission from website owners before scraping content. When a new domain signs up with Cloudflare, they are given the choice upfront to allow or deny AI crawlers access. - Opt-Out Option: Website owners who wish to allow AI crawlers can choose to opt out of the default blocking setting. Some discussions suggest this could involve a "pay per crawl" model where website owners charge AI companies for scraping their content. - Sophisticated Protections: Cloudflare's anti-crawler protections are designed to be quite sophisticated, even employing generative AI to create scientifically accurate but unrelated content to mislead crawlers. - Impact on LLM Developers: This shift is likely to have a significant impact on LLM developers, who will need to adapt their strategies for data collection and potentially engage in direct agreements with website owners.  https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/#:~:text=Now%2C%20Cloudflare%20is%20taking%20the,their%20settings%20to%20opt%20out

We’re excited to have you, to get the most out of the community: introduce yourself and get to know each other, there's immense opportunity hiding in plain sight here. - Who are you?, Tell us about your business! - What’s your superpower (What are you really good at or what sets you apart?) - What are you most excited to learn about the future of search and how AEO can work for you? 🚀This is your space to share, ask questions, and begin your AEO journey Now, it's time to get to know each other! 👇 Please introduce yourself 👇 ----------------------------------------------------------- Here's my Introduction: Hey, everyone! 👋 My name is Kasim Aslam, I built and sold the #1 Google Ads Agency in the world, and today, as co-founder of AEO.co I’m here to help you navigate the exciting world of Answer Engine Optimization (AEO). My superpower? Sourcing and training exceptional talent from emerging nations, and the proof is in this community (You gotta meet Julian!) What am I most curious about in the future of search? I’m super interested in how to leverage AEO for ultra-niched businesses—especially when you're in a space with a very specific target audience and trying to get AI Answer Engines to recommend your brand. I’m thrilled to be on this journey with all of you, and I can’t wait to see how AEO will revolutionize the way we all approach search and visibility in this new AI-driven landscape.