---
title: AI crawlers are reading your WordPress site. See them, then decide.
description: GPTBot, ClaudeBot and PerplexityBot read websites to answer questions. How to see them, count the visitors they send, and choose who may read what.
url: https://ironhelmsuite.com/blog/ai-crawlers-are-reading-your-site/
---

# AI crawlers are reading your WordPress site. See them, then decide.

GPTBot, ClaudeBot and PerplexityBot read websites to answer questions. How to see them, count the visitors they send, and choose who may read what.

Published 2026-11-17.

Search engines are no longer the only machines reading your site with purpose. AI assistants send their own crawlers to read pages so they can answer questions, and they increasingly send visitors back when they cite a source. For many sites this is now a measurable share of traffic, and it is invisible in most analytics tools.

## Who is reading

The crawlers you will meet most often include:

- **GPTBot** and **OAI-SearchBot** (OpenAI)
- **ClaudeBot** (Anthropic)
- **PerplexityBot** (Perplexity)
- **Google-Extended**, a token Google uses for AI training choices, separate from Googlebot
- **Applebot** (Apple), **CCBot** (Common Crawl) and a growing list of others

Some read to train models, some to answer a live question, some for both. Most of them say who they are in their user agent, and the well-behaved ones respect `robots.txt`.

## Seeing them

Ironhelm's AI visibility feature records which AI crawlers and search engines read your pages, which pages they read and how often, even when the page was served from cache. Its visitor statistics count people who arrive from AI assistants such as ChatGPT, Perplexity, Claude and Gemini, so you can see whether being read turns into being visited. All of it is counted on your own server, without cookies.

## Deciding

There is no single right answer. A business that wants to be recommended by assistants usually lets them read; a publisher whose articles are the product may refuse training crawlers while allowing assistants that cite and link.

Under **SEO & AI, Crawlers and AI**, Ironhelm lets you allow or refuse each crawler, and writes the matching rules into your `robots.txt`. It can also publish an `llms.txt` file: a short, readable index that points AI assistants to the pages that best explain what you do.

## Being understood

Being read is not the same as being understood. Structured data tells machines, in their own vocabulary, who is behind the site, what each page is, who wrote it, and, on a shop, what each product costs and whether it is in stock. Clear page titles, honest descriptions and a sensible internal link structure help assistants as much as they help search engines.

None of this replaces writing useful pages. It makes sure the useful pages you write are found, read correctly and credited. The [SEO & AI guide](/docs/seo-ai/) covers each setting.
