What three days of crawler logs said after I shipped an llms.txt

This site has published its prices for a year. Every package figure is the hourly rate times a stated number of hours, the service area is in the markup, and the FAQs are machine-readable. All of that was built for one consumer: Google.

Then a prospect told me they had asked ChatGPT for a developer in Vejle before they found me, and I realised I could not answer the obvious follow-up. Had any assistant ever fetched a page on this site? micci-web emitted no access log at all. The analytics I had were a browser script, so they saw humans and nothing else. I had spent months making the site quotable and had no way to tell whether anything had come to quote it.

So I spent a day on the plumbing, and then three days reading what came back.

GEO is technical SEO with a different consumer

The acronym going around is GEO, generative engine optimisation: being visible in an answer rather than in a list of results. Most of what it asks for is work I already sell as technical SEO, aimed at a client that behaves differently.

The difference that matters is JavaScript. Google runs it, on a later pass and not always. Most assistant crawlers do not run it at all. They fetch the HTML, parse what is in it, and answer from that. A site that assembles its content in the browser hands them an empty shell, and the assistant does not tell anybody it got nothing — it just answers from somewhere else.

The other difference is that there is no ranking to buy, read off, or report on. Nobody can sell you a place in an AI answer. What you can do is remove the reasons a page gets skipped, which is a shorter and more boring list than it sounds.

The trap in robots.txt

The first thing I changed was naming the assistant crawlers explicitly. Functionally this changed nothing — my wildcard group already allowed them — but "block the AI scrapers" is a one-line edit somebody makes in a hurry, and on a site that publishes its prices in order to be quoted, that edit would be self-harm.

While writing it I re-read the specification and found the thing that quietly breaks this for other people. A named user-agent group makes that bot ignore the * group entirely. It does not merge them. So this:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

User-agent: GPTBot
Allow: /

hands GPTBot /admin and /api/, because GPTBot now reads only its own group and the disallows live in a group it is no longer looking at. Every named group has to repeat the disallow lines:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /ws

That is now pinned by a test, because it is exactly the kind of correctness that survives until the next hurried edit.

llms.txt, and the section that earns its place

/llms.txt is one fetch that states what the studio is, where it is, what it costs, which stacks it works in, and where the important URLs are. Whether the convention lasts, I do not know. It cost an hour.

The section I would keep even if the rest went away is the one listing what I do not sell. "SEO" pattern-matches onto "SEO agency" hard enough that an assistant will cheerfully invent link-building packages for a studio that does not sell them, and a lead arriving on that premise wastes an afternoon for both sides. So the file says outright: no marketing SEO, no advertising, no ranking guarantees, one developer with a real limit on concurrency.

Every figure in it is generated from the same table that prices the site. A hand-typed price in that file would be worse than a stale price on a page — it is a number a model will quote back long after it changed.

One log line per crawler

Then the part that made the rest checkable:

crawler ai/ClaudeBot GET /robots.txt

One line per recognised crawler fetch, and nothing for anybody else. Matching a known crawler token first means human traffic never reaches the log line, so a site with a privacy page does not start keeping a request log about its visitors.

The ordering of those tokens is behaviour rather than tidiness. The strings nest: Claude-SearchBot contains Claude, and Google-Extended gets swallowed by a plain Googlebot test. First match wins, so a cosmetic reorder silently relabels traffic.

What the log actually said

Three days later, the last thousand log lines held 697 crawler fetches. By client:

• ClaudeBot 91, GPTBot 68, OAI-SearchBot 59, ChatGPT-User 59, Amazonbot 50

• Claude-User 38, Perplexity-User 30, meta-externalagent 28, Claude-SearchBot 26, PerplexityBot 22, Google-Extended 20

• Googlebot 51, Applebot 19, bingbot 11

• The social unfurlers — Twitter, WhatsApp, Discord, Slack, LinkedIn, Facebook — around 15 each

Two things stood out. The first is that assistant crawlers outnumbered search crawlers by a wide margin on a small Danish business site, which I did not expect three days in.

The second is what they asked for. Of the assistant fetches, the two most requested paths were /robots.txt (48) and /sitemap.xml (37). Actual content pages were in the single digits each. Most of what an assistant crawler does is reconnaissance: check what it is allowed to take, take the map, come back later. If your robots.txt is wrong, that is the whole visit.

/llms.txt was fetched once, by GPTBot. One fetch is not a result. It is the difference between a file nobody has looked at and a file something has looked at, which is the only claim I will make for it this week.

A user-agent is a claim, not an identity

The log also caught something I was not looking for. Sorted by path, the assistant traffic included /.env, /.env.local, /.env.production, /wp-config.php.bak, /@vite/env and a dozen variations.

Assistants do not do that. Somebody is running a credential scanner and rotating well-known crawler user-agents while doing it. In my log there is a WhatsApp bot asking for /.env, a Slackbot asking for /.env.openai, and a Perplexity-User asking for /.env.docker.

That is a useful correction to have in writing. A user-agent string is self-declared. If you want to know that a fetch really came from OpenAI or Anthropic, you verify it against their published IP ranges or by reverse DNS, the same way you would verify Googlebot. For my purposes — is anything reading this site — the unverified log is enough. For anything you would act on, it is not, and the numbers above should be read with that discount applied.

What I can measure, and what I still cannot

There is no ranking to read, so GEO gets measured three ways here. A fixed set of questions, the ones a customer would actually ask, put to ChatGPT, Perplexity and Google's AI answers before the work and again afterwards, with the answers saved. Traffic referred from the assistants, split out in the analytics. And the crawler log, which answers the retrieval half directly.

What I cannot fix from the code is corroboration. Language models ground entity claims in sources other than you, and almost everything this site says about MICCI is published by MICCI. There is a Proff listing, generated from the CVR register, which is the one third-party record that already existed without anybody creating it — it is now in the sameAs list. There is still no Google Business Profile. That gap is not a technical problem and I am not going to pretend otherwise on a page where I am selling the technical half.

If your site is client-rendered

The one thing worth checking today, on any site: turn JavaScript off and load your most commercially important page. If what is left is a spinner, that is what a good share of the assistants are working from, and no amount of writing fixes it.

The fix is rarely a rewrite. It is usually server-rendering or pre-rendering the handful of pages that have to be found — the front page, the services, the prices, the catalogue — while the rest of the application stays exactly as it is behind the login. That work, plus the markup and the crawler access, is what I sell as technical SEO and GEO. The prices are on the page, as usual.