AI crawlers are automated bots from AI companies, such as GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, that read your website so AI tools can learn about your business and use that information when answering user questions.
Your robots.txt file controls which crawlers are allowed in. Many businesses block AI crawlers by default without realizing it, often because their web platform or a security plugin did it for them. A blocked crawler means that AI tool is building its picture of your business entirely from third party sources, or not at all.
The visibility tradeoff
Blocking AI crawlers is a legitimate choice for publishers protecting content. For a local business that wants customers, it is usually self sabotage: you are telling the exact systems your customers now ask for recommendations that they may not read your site. If AI visibility matters to your business, your robots.txt should explicitly allow the major AI crawlers, and your server logs will show you which ones actually visit.
How to check yours
Look at yourdomain.com/robots.txt. If you see AI bot names under a Disallow rule, or a blanket disallow for all bots, AI tools are being turned away at the door. Fixing it is a few lines of text, and it is one of the first things checked in any honest AI visibility review.