How I Track Which AI Crawlers Actually Read My Site (With Code)

Milestone: 2.1k AI crawler hits in 7 days on Wysera. Here is the logging setup behind that number, and why it beats keyword rankings as a GEO signal.

Every SEO tool I pay for reports rankings and impressions. Not one of them told me which AI engines were fetching my pages, or why. In the AI answer era that input signal matters more than the keyword position, because generative engine optimization starts at the crawl, not the SERP. If GPTBot never fetches your page, no amount of on-page work gets you cited in ChatGPT. So I stopped waiting for a tool and instrumented it myself.

Here is the whole setup.

Not all bots want the same thing

The mistake most people make is treating "AI bot traffic" as one number. It is at least three distinct intents, each with its own user-agents.

Intent What it means User-agents Training Page pulled into a model's knowledge GPTBot, ClaudeBot, Amazonbot, Google-Extended, Applebot-Extended Search / index Page indexed for an AI or classic search surface OAI-SearchBot, Bingbot, Googlebot, PerplexityBot, Applebot Live answers Page fetched in real time to answer a user's question ChatGPT-User, Perplexity-User

That last bucket is the one nobody watches, and it is the most valuable. A live-answer fetch means a real person asked a question right now and an engine reached for your page to respond. That is the earliest possible proof GEO is working, and it will never show up in a Google-shaped dashboard.

Step 1: let them in

You cannot measure a crawler you accidentally blocked. Check robots.txt first. One stray Disallow silently removes an entire engine from your funnel.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

If you want to be in AI answers, opt in explicitly. Do not rely on a permissive default that a framework or CDN might override.

Step 2: log and classify

Your access logs already have everything. This is the entire classifier I started with. Node, no dependencies, reads a combined-format log and buckets hits by intent and by page.

import { readFileSync } from "node:fs";

const BOTS = {
 training: [/GPTBot/i, /ClaudeBot/i, /Amazonbot/i, /Google-Extended/i, /Applebot-Extended/i],
 search: [/OAI-SearchBot/i, /Bingbot/i, /Googlebot/i, /PerplexityBot/i, /\bApplebot\b/i],
 answers: [/ChatGPT-User/i, /Perplexity-User/i],
};

const tally = { training: 0, search: 0, answers: 0 };
const byPage = {};

for (const line of readFileSync("access.log", "utf8").split("\n")) {
 const parts = line.split('"');
 const ua = parts[5] || ""; // user-agent field
 const path = (parts[1] || "").split(" ")[1]; // "GET /path HTTP/1.1" -> /path

 for (const [intent, patterns] of Object.entries(BOTS)) {
 if (patterns.some((re) => re.test(ua))) {
 tally[intent]++;
 byPage[path] = (byPage[path] || 0) + 1;
 break; // check training first so Applebot-Extended is not double-counted
 }
 }
}

console.table(tally);
console.log(Object.entries(byPage).sort((a, b) => b[1] - a[1]).slice(0, 15));

Order matters: training patterns run first, so Applebot-Extended counts as training and never leaks into the search bucket via the plain Applebot match. Ship this as a cron job, pipe the tallies into a table, and you have crawler observability in an afternoon.

Step 3: read the ratio, not the total

The total is vanity. The ratio between the three buckets is the actual GEO diagnostic.

• All three growing: healthy. Content is being read, stored, and surfaced.

• Training and search up, answers flat: your pages are being stored but not chosen. That is a relevance problem, not a crawl problem. Rewrite for answer-first extraction and add structured data.

• Everything flat or dropping: check robots.txt, check for JS-only rendering, check if your host is rate-limiting bots.

The byPage map is the other half. It tells you exactly which URLs the engines value, which is where you double down.

The receipt

Seven days on my own site: 2,074 hits. OpenAI led with 1,243 (906 search, 337 training), then Amazon 237, Bing 149, Google 125, Perplexity 96, Anthropic 64. The split by intent came out ~1.3k indexing, 645 training, 129 live answers. Domain rating climbed 0 to 34 over the same build.

For a young domain that is a modest number. The point is not size. The point is that it is finally legible. You cannot optimize a funnel you cannot see, and this is the cheapest way I have found to see it.

I eventually folded this logging into PostWyse, the marketing module inside Wysera, so I stop grepping logs by hand. But the script above is genuinely all you need to start today.

Next experiment: correlating crawl volume per page against actual citations in ChatGPT and Perplexity, to find which content structures convert crawls into answers fastest. I will publish those numbers here too.

Full framework, robots.txt config, and the complete bot list are at gkotte.com. Following the build-in-public GEO series if you want the correlation data when it lands.

Build. Ship. Impact.