I recently got moved to a new Cloudflare account and was curious what insights I could gain on AI crawl behaviour. I was shocked at the huge number of requests coming from ClaudeBot followed by Baidu which is interesting as China isn’t a key audience for that website. It’s also interesting that ClaudeBot is 10x the number of requests vs Googlebot but Google sends about 95% of all the search traffic to that website. Applebot was a new entrant that I haven’t seen in almost a decade when they were trying to improve Spotlight search functionality and were aggressively increasing their crawling.
I was also curious what were the bots landing on my 4xx status pages as was I missing out on some organic traffic I should redirect to a suitable landing page. Sadly one of the biggest offenders was MistralAI-User which was attempting to crawl all sorts of pages that don’t exist and seemed to be more so focusing on finding exploits to hack.
/.env.bak
/config/.env.php Database logins, SMTP credentials, payment keys, WordPress salts. The
.bak copy is the one everyone forgets to delete.
Critical
/config.js Front-end bundle configs. Developers routinely bake “public” API keys in here that turn out to be rather less public than assumed. High
⚠️ Every request in this sample returned a 4xx. Severity reflects what would have been exposed if the file existed, not what actually leaked.
They were scanning non-existant subdomains: preprod, staging, assets, www. Staging and preprod are where this kind of sweep usually pays off — they’re built fast, hardened last, and often carry real production credentials in their .env. This AI scanner knows that, which is why it aimed /config/.env.php at staging specifically…. looking for stuff that it shouldn’t be trying to access. If you haven’t blocked MistralAI-User bot yet I’d start blocking that bot once you can get access to CloudFlare AI Crawl control.
One other great free feature of running Cloudflare is the free Agent Readiness scan and register interest in the free AEO feature coming soon that helps you see how often AI assistants cite your site. The quick wins and technical groundwork are a reasonable effort to ensure more sites are ready for the AIbot flood that is pending.
So I’d say if you haven’t starting testing Cloudflare for the AI insights you should start today it seems to have addressed a number of pain points for website owners and content creators without significant financial investment.