That mismatch, billions of pages harvested by automated crawlers versus a publishing model built for human readers, is now getting sorted out in courtrooms, in robots.txt files, and in policy language that varies by jurisdiction and rarely gets simpler. If you run a website, or just want to know what happens to the things you write online, it's worth understanding what the current opt-out system actually does.
The crawlers doing the harvesting
Training a large model requires enormous volumes of text, and the fastest way to collect it is the same way search engines have always collected it: automated bots that follow links and download pages. OpenAI runs GPTBot. Anthropic runs ClaudeBot. Google runs Google-Extended, a separate token from its search crawler so site owners can allow indexing while blocking training use. ByteDance runs Bytespider. Common Crawl runs CCBot, and its archives get reused by multiple labs downstream, which means blocking one crawler doesn't guarantee your content stays out of every model trained on aggregated web data.
Each crawler identifies itself with a distinct user-agent string in its HTTP requests. That string is the only lever a site owner has to say no, and it only works because the crawler chooses to honor it.
robots.txt was never built for this
The robots.txt standard dates to 1994. It was designed to stop search engine crawlers from indexing pages a site owner didn't want indexed, or from wasting server resources crawling directories with no public value. It's a voluntary protocol. There is no enforcement mechanism built into the web; a crawler either respects the Disallow directive or it doesn't, and the only recourse for a site that gets ignored is blocking the crawler's IP ranges after the fact, which is a losing game against anyone willing to rotate infrastructure.
That voluntary structure is now carrying weight it was never designed for: the difference between your writing being trained into a model someone else profits from, or not. Several major labs, including OpenAI, document that their crawlers check robots.txt before fetching. Others have been caught operating under undisclosed user-agent strings after being blocked under their known one, which is the kind of finding that made a lot of publishers stop treating voluntary compliance as a real answer.
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: /
This tells each named crawler not to fetch anything from your site. It does nothing about content already used to train an existing model, and nothing against a crawler that doesn't identify itself honestly.
What opt-out actually stops, and what it doesn't
Blocking a crawler in robots.txt is forward-looking only. It has no effect on snapshots already taken, and no effect on models already trained. If your content was crawled before you added the Disallow line, it's already baked into whatever model consumed it, and there's no mechanism to have it removed, the way there might be for a search index.
It also does nothing about republication. If your article gets syndicated, quoted at length by another outlet, or mirrored on a site you don't control, a crawler that respects your robots.txt can still ingest the same words from somewhere else. And it says nothing about training data acquired through licensing deals, where a company pays a publisher directly for bulk access, bypassing the crawl entirely. Several major outlets have signed exactly these deals in the past two years, which effectively splits the media landscape into publishers who got paid and publishers who got scraped anyway.
| Mechanism | What it actually does |
|---|---|
| robots.txt Disallow | Voluntary signal to future crawls. No effect on past crawls or non-compliant bots. |
| Cloudflare AI crawler blocking | Blocks known crawler IP ranges and signatures at the network edge. Stronger than robots.txt but limited to sites behind Cloudflare, and reversible in settings. |
| Copyright takedown / lawsuit | Addresses reproduction of specific protected works after the fact. Doesn't prevent future scraping, and courts haven't settled whether training itself infringes. |
| Licensing deal | Grants explicit, compensated access. Only available to publishers with enough leverage to negotiate one. |
The legal fight catching up
The New York Times sued OpenAI and Microsoft in December 2023, alleging that ChatGPT had been trained on millions of Times articles without a license and could reproduce them nearly verbatim on request. The case is still working through discovery, and its outcome will likely shape how courts apply fair use to model training generally, a question copyright law never anticipated because nothing like this existed when it was written. Music publishers, stock photo libraries, and authors' groups have filed comparable suits against other labs. None have reached a final judgment that resolves the underlying question of whether training on copyrighted text is itself an infringing use.
Policy is starting to catch up in a narrower way. The EU's AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used to train them, a transparency obligation that took effect in August 2025. It doesn't require consent and it doesn't ban scraping. It requires disclosure, which for years plenty of labs simply declined to give at all.
What you can actually do
If you run a site, block known crawler user-agents in robots.txt as a baseline, understanding it's a polite request, not a wall. Cloudflare made blocking AI crawlers the default for new domains behind its network in 2025, which is a meaningfully stronger control since it operates at the network layer rather than trusting the bot to comply, though it's one settings toggle away from being turned back off.
For individuals, there's less you can do retroactively about content already public. The more durable move is upstream: publish behind a login where it makes sense, keep private drafts private, and understand that anything posted to the open web is, practically speaking, available to be scraped by someone eventually, regardless of what any robots.txt file says. The opt-out system that exists today is a courtesy some crawlers honor, not a right you can enforce.
None of this is a reason to stop writing publicly. It's a reason to know, going in, which parts of the current system are load-bearing and which are decorative, in roughly the same spirit as our piece on data minimization: know what you're putting out before you decide it's fine to lose control of it, and treat AI-driven surveillance and AI training scraping as related but distinct problems with different, mostly weak, defenses.