HomeFree toolsllms.txt & AI crawler checker
llms.txt & AI crawler checker
Paste a public URL to check for /llms.txt and see whether robots.txt blocks common AI crawlers like GPTBot, ClaudeBot, and PerplexityBot.
FAQ
- What is /llms.txt?
- /llms.txt is a plain-text file some AI tools look for at your site origin. It can summarize what the product is and point to authoritative pages. This checker reports whether the file returns HTTP 200.
- Which AI crawlers does this tool check?
- It reads robots.txt and flags named bots such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and others when they are blocked with Disallow: /.
- Does a missing robots.txt mean AI cannot crawl my site?
- Not necessarily. Without robots.txt there is no explicit AI-bot block. Many sites still benefit from a clear robots policy and an optional /llms.txt summary for AI systems.