robots.txt: the door for crawlers
robots.txt is a small text file in your website’s root directory. It tells crawlers which areas they may fetch. For AI visibility, what matters most is that search crawlers are not locked out.
- For search answers
- OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot and Bingbot. If you block them, you will not appear there.
- Fetches on behalf of users
- ChatGPT-User, Claude-User, Perplexity-User: they fetch a page when a user asks for it. OpenAI and Perplexity point out that robots.txt rules may not always apply here.
- For training
- GPTBot, ClaudeBot, Google-Extended: they concern whether content is used to train future models. According to OpenAI the settings are independent – blocking GPTBot does not stop you from appearing in ChatGPT search. According to Google, Google-Extended has no effect on Google Search.
Whether you allow training is a business decision. We deliberately allow it on pickedfirst.ai. Important: robots.txt does not keep a page out of Google’s index. That requires a “noindex” directive.
Structured data: facts in machine language
Structured data is an invisible block in the source code that states in a fixed format (schema.org, usually as JSON-LD): this is a law firm, this is its name, this is where it is, these are its services. Google recommends JSON-LD and writes that such data helps it understand a page. Google does not promise better placement.
- Matching the visible text: Google explicitly points out that structured data must match the visible content.
- Make nothing up: do not enter opening hours, postal codes or reviews that do not exist. If you have no shop, you do not need a local-shop entry.
- Be realistic about FAQ data: since August 2023, Google only shows expandable FAQ results for well-known government and health websites. For most businesses, FAQ data therefore brings no special display on Google. The visible question-and-answer text remains useful nonetheless.
For AI Overviews and AI Mode, Google also writes that no special structured data is needed. Whether other AI assistants specifically evaluate structured data is not documented by the providers. We still use it because it makes facts unambiguous and costs little.
llms.txt: a proposal, not a standard
llms.txt was proposed by Jeremy Howard in 2024: a Markdown file in the root directory that briefly explains to AI systems what the website is about, with links to the most important pages. The initiative calls itself a proposal; it is not an official standard.
For its AI features, Google explicitly says that no new machine-readable files or AI text files are needed. That ChatGPT, Gemini, Claude or Perplexity reliably use an llms.txt in their answers is not proven as far as we know. Our check therefore does not deduct points if it is missing. We still create one (our llms.txt) because it takes ten minutes.
What matters most
In our experience, reachability comes first: a website that locks out crawlers – in robots.txt or unnoticed through its host and bot protection (see hosting pitfalls) – cannot be named. Clear, unambiguous facts come next. llms.txt is a small extra, not a shortcut.
Which of this applies to your business is what we check concretely in the AI Visibility Check: on your website, with your clients’ questions and compared with your competitors.