SEO & AI
Search results, structured data, sitemaps and the AI crawlers now reading the web.
AI crawlers are reading your WordPress site. See them, then decide.
GPTBot, ClaudeBot and PerplexityBot read websites to answer questions. How to see them, count the visitors they send, and choose who may read what.
Robots.txt in the age of AI crawlers: a field guide
Which user agents belong to which AI companies, what robots.txt can and cannot do, and example rules for allowing AI search while refusing AI training.
llms.txt: what it is, who reads it, and whether you need one
A plain-text map of your site for AI assistants, proposed in 2024. What the format looks like, what the evidence says about adoption, and our honest recommendation.
XML sitemaps: what Google actually uses, and what it ignores
Priority and change frequency do nothing in Google, the ping endpoint is gone, and lastmod only helps if it is honest. A short, accurate guide to sitemaps.
Redirects and 404s: keeping links alive through a redesign
A redesign that changes page addresses can cost years of search traffic. How Google treats 301s and 404s, and a practical plan for moving a site without losing ground.
Structured data that still earns rich results
Google cut back FAQ and How-to rich results in 2023. Which structured data still changes how your pages appear, and which markup is quietly wasted effort.