How to block AI training crawlers with robots.txt

On Fotobank, the stock-photo marketplace I built, the photos are the product. Contributors upload work to sell it under Standard, Premium and Exclusive tiers, so a crawler copying the catalogue into a training set works against the whole point of the site. One of the changes I made there was a robots.txt that blocks AI training crawlers while leaving search engines alone. This post covers how that file works, which user-agent tokens to list, and what robots.txt can and cannot do for you.

robots.txt is a voluntary request

Start with the limit, because it shapes everything else. robots.txt is a plain text file at the root of your domain, standardised as the Robots Exclusion Protocol in RFC 9309. Well-behaved crawlers fetch it before they crawl and follow its rules. Nothing in the protocol forces a crawler to comply. A bot that ignores the file can still fetch every public page, and the file itself is public, so anyone can read which paths you care about.

So treat robots.txt as the polite first layer. The operators of the large AI crawlers publish their user-agent tokens and state that they honour robots.txt, which makes the file worth writing. It is a request with no enforcement behind it, and you should plan the rest of your protection with that in mind.

How user-agent groups work

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler looks for the group that names its token and follows that group. Only when no group names it does it fall back to the * group.

That fallback rule is the gotcha people hit most often. A crawler that matches a named group ignores the * group completely. Rules do not stack. If * has Disallow: /account/ and you add a separate group for some crawler, that crawler no longer sees the account rule unless you repeat it there.

A few other details worth knowing:

• Several User-agent lines can share one group, which keeps the file short.
• Token matching is case-insensitive. Paths are case-sensitive and match by prefix.
• Disallow: / blocks the whole site. When an Allow and a Disallow both match a URL, the longer (more specific) rule wins.
• Crawlers cache robots.txt, so a change can take a while to be picked up.

The crawler tokens I'd list

I only list tokens that their operators document themselves. Guessing at names from forum posts gives you a file full of entries that match nothing.

• GPTBot: a major model provider's training crawler. The same operator documents separate agents for its search feature (OAI-SearchBot) and for fetches a user triggers (ChatGPT-User), and blocking GPTBot leaves those two untouched. Decide on each one separately.
• Google-Extended: a control token with no crawler of its own. Googlebot still does the fetching; Google-Extended tells Google whether the content may be used to train its generative models. Google states that it has no effect on Search inclusion or ranking.
• Applebot-Extended: the same pattern from Apple. Applebot crawls for search features, and Applebot-Extended controls whether that content is used for training.
• CCBot: Common Crawl's crawler. Its open dataset is widely used as training data, so blocking it cuts off a lot of downstream use you would never see directly.
• meta-externalagent: Meta's crawler for AI training and related product uses.

Operators add and rename tokens over time. I'd check each operator's crawler documentation a couple of times a year and update the list.

A file that blocks training and keeps search

Here is the shape I'd use. The disallowed paths in the first group are placeholders for whatever your site keeps private.

cat > public/robots.txt <<'EOF'
# Search engines and everyone else
User-agent: *
Disallow: /account/
Disallow: /checkout/
Allow: /

# AI training crawlers and training-use tokens
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml
EOF

Never put Googlebot or Bingbot in the blocked group to stop training. Those are the crawlers that put you in search results, and blocking them removes you from search. Use the training-specific tokens instead.

If your site runs on the Next.js App Router, an app/robots.ts file can generate the same output. Its userAgent field accepts an array, so the blocked group stays one entry. After deploying, fetch the live file with curl https://yourdomain.com/robots.txt and read it. A build step or a CDN rule that serves a stale or default file is a common way for this to silently fail.

Meta tags, ai.txt and other signals

You will see other opt-out mechanisms recommended. Here is how I weigh them.

• noai and noimageai meta tags: some platforms adopted these, but they are not part of any standard. I would not rely on them on their own.
• Standard robots meta and X-Robots-Tag: directives like noindex work, but they are about search indexing. Using noindex to stop training also takes the page out of search.
• ai.txt and similar files: there are several competing proposals, none of them standardised or widely honoured yet. Adding one costs little, so add it if you like, but keep robots.txt as the file that matters.

What actually protects your content

Because robots.txt is voluntary, the real protection lives in your server and your product.

• Watch your access logs for the tokens you blocked. A crawler that keeps fetching pages after reading a Disallow: / is ignoring you, and now you know.
• Block at the edge for crawlers that ignore the file. A user-agent match in Nginx or your CDN's firewall can return a 403. User-agent strings are easy to fake, so this filters honest bots and lazy scrapers only.
• Rate-limit everyone. Aggressive crawling looks the same whatever it calls itself.
• Keep originals out of public reach. For a photo marketplace, the strongest protection is in the product design: public pages should only ever serve previews, with full-resolution files behind a purchase. I covered how I'd structure that in an image processing pipeline with Sharp and S3.

One more limit: an opt-out only applies going forward. Content already collected stays collected.

If you are on the other side of this and building a scraper yourself, structured extraction from scraped pages covers how I'd approach it, and reading robots.txt first belongs on that checklist too.

Where I'd start

Add a named group with the documented training tokens, keep the * group open for search, and verify the live file with curl. Then assume some crawlers will ignore it, and put the real protection in your logs, your edge rules and what your public pages expose. You can see how this fits into Fotobank and my other builds on the projects page.