1. Blocks at shared hosts
Many low-cost hosting plans protect themselves against too many requests. That is understandable, but it also hits search engines and AI crawlers that read several pages in quick succession.
Our own observation: on two websites on a widely used shared hosting plan, we measured that after 17 rapid requests the server answered only with error 503 (“unavailable”) for about 29 minutes – for all visitors, including crawlers.
- Ask your host whether and when such blocks apply, and whether known crawlers can be exempted.
- Or deliver the website through a service with a worldwide cache (CDN). The domain can stay with your current provider.
- Slow servers are a related problem: if crawlers give up because of long load times, pages are missing.
2. Bot protection that blocks AI crawlers
Services such as Cloudflare offer settings against automated access. They are useful, but should be set deliberately:
- AI bot rules: in its newer settings, Cloudflare distinguishes between crawlers for search, for agents and for training. The older “Block AI bots” setting targets training crawlers. Since 15 September 2026, new domains get changed defaults. Check that search crawlers are allowed.
- Bot Fight Mode: applies to the whole domain and, according to Cloudflare, cannot be bypassed with your own firewall rules. If you need exceptions, you have to switch to other products.
- Rule of thumb: always allow search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). Whether training crawlers are allowed is your business decision.
3. Email obfuscation
Cloudflare switches on “Email Address Obfuscation” automatically when you sign up. It replaces email addresses in the source code with an encoded link; only a small script in the browser reveals the real address. That keeps simple address harvesters away – but also crawlers that do not run scripts.
Our own observation: on pickedfirst.ai the feature was initially active. Instead of our address, the source code only showed “[email protected]”. For AI visibility we therefore switch it off.
Why these pitfalls often go unnoticed
For visitors in a browser the website looks perfectly normal: the email address appears, the page loads, the bot protection does not kick in. The pitfalls only hit crawlers – exactly the programs through which AI assistants read your content.
Website builders such as Wix, Jimdo or Squarespace often do not let you edit robots.txt or llms.txt freely. What is possible there is covered for your system by our Implementation Kit.
Which of this applies to your business is what we check concretely in the AI Visibility Check: on your website, with your clients’ questions and compared with your competitors.