AI crawlers are reading your WordPress site. See them, then decide.
GPTBot, ClaudeBot and PerplexityBot read websites to answer questions. How to see them, count the visitors they send, and choose who may read what.
Search engines are no longer the only machines reading your site with purpose. AI assistants send their own crawlers to read pages so they can answer questions, and they increasingly send visitors back when they cite a source. For many sites this is now a measurable share of traffic, and it is invisible in most analytics tools.
Who is reading
The crawlers you will meet most often include:
- GPTBot and OAI-SearchBot (OpenAI)
- ClaudeBot (Anthropic)
- PerplexityBot (Perplexity)
- Google-Extended, a token Google uses for AI training choices, separate from Googlebot
- Applebot (Apple), CCBot (Common Crawl) and a growing list of others
Some read to train models, some to answer a live question, some for both. Most of them say who they are in their user agent, and the well-behaved ones respect robots.txt.
Seeing them
Ironhelm's AI visibility feature records which AI crawlers and search engines read your pages, which pages they read and how often, even when the page was served from cache. Its visitor statistics count people who arrive from AI assistants such as ChatGPT, Perplexity, Claude and Gemini, so you can see whether being read turns into being visited. All of it is counted on your own server, without cookies.
Deciding
There is no single right answer. A business that wants to be recommended by assistants usually lets them read; a publisher whose articles are the product may refuse training crawlers while allowing assistants that cite and link.
Under SEO & AI, Crawlers and AI, Ironhelm lets you allow or refuse each crawler, and writes the matching rules into your robots.txt. It can also publish an llms.txt file: a short, readable index that points AI assistants to the pages that best explain what you do.
Being understood
Being read is not the same as being understood. Structured data tells machines, in their own vocabulary, who is behind the site, what each page is, who wrote it, and, on a shop, what each product costs and whether it is in stock. Clear page titles, honest descriptions and a sensible internal link structure help assistants as much as they help search engines.
None of this replaces writing useful pages. It makes sure the useful pages you write are found, read correctly and credited. The SEO & AI guide covers each setting.
Ironhelm Suite is one WordPress plugin for security, speed, SEO and analytics. See plans or tour the features.