Cloudflare — who fetches your pages
The stages as a bar, and two lines by week
The card draws the stages as one bar: every published page sits in exactly one segment, from not fetched to cited, with its count printed; the sentence beneath is the same record in words. Two lines then run week by week: pages a crawler first fetched (indigo) and pages an AI answer first cited (green). A page joins each line once, in the week of its own first date; a page with no such date adds nothing - it is not a zero. Hover over, or tab to, a week for the running counts and the pages behind that week's step, each with its next action; where the plugin found a page cache every count is "at least". Hatched segments on the indexed and looked-at rungs mean not checked, not read or unknown - never no. Nothing is drawn where neither a Cloudflare zone nor the plugin reads the site.
What this shows you
Publishing is something we do. Being cited is something we measure. In between sit two steps nobody could see: whether an AI crawler ever came for the page, and whether an assistant fetched it while answering somebody's question. Your own site's edge is the only place those are recorded. Cloudflare keeps a log of every request, including the ones from GPTBot, ClaudeBot, PerplexityBot and the rest — so if your site sits behind Cloudflare, you can see exactly which of the pages we published anything has come for. Cloudflare's side of this is free. The token is yours, you create it, and you can revoke it whenever you like.
Or sign in: Connect Cloudflare
Under Connections, Cloudflare, press Connect Cloudflare and sign in. Cloudflare asks which account to share and shows what we ask for: read your zones, read analytics, read logs, and edit logs (so we can set up a crawler log feed later; nothing is set up without you). We match each project to the zone that serves its site; where more than one fits, you choose. You need no token for a project that reads through the sign-in, and a token you pasted before keeps working. To stop, press Disconnect there: we ask Cloudflare to withdraw our access, and you can choose to remove the crawler counts of the projects that read through it. You can also revoke us in Cloudflare under your profile, Manage OAuth authorizations. Growth and above, like the token.
Creating the token
1. In Cloudflare, open My Profile → API Tokens → Create Token → Create Custom Token. 2. Give it a name you will recognise later, such as "The Ranking Factory — crawler reads". 3. Add TWO permissions: • Zone → Analytics → Read • Zone → Zone → Read 4. Under Zone Resources, choose Include → Specific zone → the one site you are connecting. 5. Create the token and copy it. Cloudflare shows it once. 6. Copy the Zone ID too: it is on the Overview page of that site in Cloudflare, in the right-hand column. 7. Paste both into "AI crawlers: your Cloudflare connection" on the project's Settings tab. 8. The report itself - which crawler fetched which page, and when - is "AI crawlers on your site" on the project's overview. Settings hold only the zone and the token. 9. To see every project's state in one place, open Connections → Cloudflare. Each project is listed as connected, not connected or not on this plan, and "Open settings" takes you to that project's card. Scope it to the ONE zone you are connecting. A token that covers every zone on your account is a bigger thing to lose than one that covers a single site, and we never need more than the site in front of us.
Why two permissions and not one
Analytics → Read is what lets us count the requests. On its own that is enough to report who fetched what. Zone → Read tells us one extra thing: the date your site started being served by Cloudflare. That matters more than it sounds. Cloudflare only has records for traffic that actually passed through it, so a zone you connected this morning holds nothing at all about last week — and without knowing when it started, an empty week is indistinguishable from a quiet one. We will not report "nothing came for your pages" when the honest answer is "we have only been watching since Tuesday". If the token cannot read the zone's start date we fall back to the first date we successfully read your analytics, and say so on the card. Adding the second permission simply makes that line accurate from the beginning rather than cautious.
What the numbers mean, and what they do not
A hit is a request whose user agent gave a name. Anyone can put any name in a user agent, so read it as "something calling itself GPTBot fetched this page" — for the question this answers, which is whether anything came for the page at all, that is both honest and sufficient. A fetch is not a citation. A crawler taking your page means it was reachable and worth taking; it does not mean an answer used it. Those are different steps and we keep them apart. Crawler names ending in "-User" — ChatGPT-User, Claude-User, Perplexity-User — are different again. They fetch because a person asked something right now, so they are the closest thing to evidence that your page was pulled into a live answer. Search engine crawlers such as Googlebot and Bingbot are counted as search crawlers, apart from AI training crawlers and assistants: AI Overviews, AI Mode and Copilot draw on their indexes, so a page they fetched is on its way to being found. GPTBot and ClaudeBot gather pages a future model may learn from and feed no search index, so a page only they fetched is said apart. The report's sentence says how many pages each kind fetched.
Where each page stands, and what to do about it
Every page we published on this site is placed at one of five stages, counted over every night we hold rather than the last week - "first crawled" means the first time ever: • Not fetched - nothing has come for it. Link to it from a page crawlers already fetch, and check it is in the sitemap. • Refused - crawlers came and every fetch we have a status for was turned away (a firewall rule, a bot filter, an error). Check those rules for that address. • Crawled - a crawler took it, but no assistant has fetched it for somebody's question yet. Where a search crawler took it, make it answer one question plainly in its opening lines, and link to it from pages assistants already fetch. Where only AI training crawlers took it, link to it from pages search crawlers already fetch, and check it is in the sitemap. • Retrieved - an assistant fetched it while answering somebody, but no answer we measured cited it. Check that its opening paragraph answers that question with a fact an answer can quote. • Cited - an answer we measured cited it. A citation counts even without a crawl we saw: an engine can cite from an index built before your edge was being watched. "Served" is read from the status Cloudflare returned for each fetch, since 28 September 2026. Nights read before then, and zones whose plan does not report a status, say nothing either way - never "all refused". The same stage and next step reach your AI assistant through get_published_evidence. This report and the card at the top of each project split the Crawled step the same way, from one crawler list: pages fetched only by AI training crawlers (GPTBot, ClaudeBot), which feed no search index, pages fetched by search crawlers, and pages fetched by neither. Both read the same sources by the same rule. See Your site, at the top of each project.
Indexed, and looked at but not cited
Beside each page's stage, two more columns say what your edge cannot. Each names where it comes from, and each says "not checked" or "unknown" when nothing could tell us - never "no". • Indexed - whether Google has the page in its index, read only through your Search Console's URL Inspection, for addresses on a property you have connected. A page with no reading there, or one whose only reading is its own "page not found", is "not checked". • Looked at, not cited - an engine's search returned the page for an answer, and the answer did not cite it. It shows which engines, in how many answers, and when. Only OpenAI, and Claude asked directly rather than through OpenRouter, say what their search returned. Claude asked through OpenRouter, Gemini, Perplexity and the other engines do not, so for most pages this column reads "unknown". That means no engine that reports has listed the page - not that nothing looked at it. These are visibility runs' answers; the same two readings reach your AI assistant through get_published_evidence.
Pages we cannot see
Only pages on this site's own domain are covered. Anything we publish to Google Docs, Blogger or another third-party property is served by somebody else's edge, so no token of yours can report on it. Those pages are counted separately and named, rather than listed as though nothing came for them. The count is this project's own pieces published elsewhere, not the whole account's (before 15 Oct 2026 every project showed the account's figure).
When it runs
Once a night, shortly after each day closes, plus a backfill of any day within Cloudflare's retention that has not been read yet — up to 30 days back, which is what Cloudflare keeps on every plan. Read now on the crawler card runs that same read for one project straight away (once in ten minutes), and while a token you saved again waits for its first read the card keeps showing what earlier reads found and says so. Cloudflare limits how much can be read at once. If a backfill runs out of budget the job stops that site cleanly and picks up where it left off the following night, so a first read across several sites may take a couple of nights to fill in. Nothing is lost by waiting. Available on Growth and above. On WordPress, our plugin shows the same report on every plan - see "WordPress plugin - who fetches your pages". Where a site has both, both are kept and shown side by side, never added: the counts use the Cloudflare read for every day it covers, because the edge also counts the visits a cache answered, and the plugin adds whether each visit came from the crawler's own addresses.