Content Signals and RSL — Definition
Content Signals and RSL are two ways of stating, in machine-readable form, what AI systems may do with your content after they have fetched it — something robots.txt was never able to express. robots.txt answers one question: may this crawler request this URL. It cannot say “you may quote me in an answer but not train on me”. Content Signals adds a preference line for exactly that. RSL (Really Simple Licensing) goes further and attaches licensing and compensation terms.
robots.txt controls access; Content Signals and RSL describe permitted use — and both are declarations, not enforcement.
Content Signals
A line in robots.txt expressing three independent preferences:
Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: *
Allow: /| Signal | Means |
|---|---|
search | Include the content in a conventional search index and link to it |
ai-input | Use the content to ground an AI answer, normally with a citation |
ai-train | Use the content to train or fine-tune a model |
Cloudflare introduced the format in 2025 and began adding default signals for sites on its network, which put the syntax on a large share of the web quickly. For a store, the combination above is usually the honest one: stay quotable, decline training.
RSL — Really Simple Licensing
RSL is an open licensing standard published on 10 September 2025 by the RSL Collective, co-founded by RSS co-creator Eckart Walther. Version 1.0 followed in December 2025. Where Content Signals states a preference, RSL states terms.
- An XML vocabulary describing usage, licensing and legal terms for a set of URLs.
- Discovery through robots.txt, HTTP
Linkheaders, RSS feeds or embedded HTML. - Licence models including free use, attribution, subscription, pay-per-crawl and pay-per-inference — payment when the content is used in an answer, not when it is fetched.
- Optional protocols for acquiring and validating licences, built on OAuth 2.0.
In robots.txt it appears as a global License directive pointing at the licence file, not tied to any user-agent:
License: https://example.com/license.xmlEarly adopters are publishers rather than retailers — Yahoo, Medium, Vox Media, USA Today, BuzzFeed and others.
Neither one enforces anything
Compliance is voluntary, exactly as with robots.txt. What a signal or a licence gives you is a clear, dated, machine-readable statement of terms — useful as a legal position and as a signal of intent, not as a technical control. Enforcement, if you want it, lives at the edge: bot verification, rate limits, or a pay-per-crawl arrangement with your CDN.
The trap for ecommerce
The instinct is to block AI crawlers outright to keep content out of training. On a store that usually backfires, because access and use are the same lever in robots.txt: a Disallow that prevents training also removes you from retrieval, and retrieval is what puts your products in an AI answer.
Content Signals exist precisely to separate those. One line lets you stay citable while declining training — which is what most merchants want and what an all-or-nothing block cannot express.
Second point, specific to shops: your product pages are not the asset publishers are protecting. Prices, availability and specifications are facts you want quoted accurately. The content worth licensing terms is usually elsewhere — original photography, buying guides, review corpora.
On Magento 2
Magento’s robots.txt handling has no concept of any of this, and hand-editing is risky because RFC 9309 groups do not merge — rules in a wildcard group do not apply to a bot that has its own group.
angeo/module-robots-txt-aeo parses and rewrites robots.txt losslessly, so directives you add by hand survive, and can emit usage and licence directives alongside the AI-crawler rules.
Questions
- What is the Content-Signal directive?
- A robots.txt line stating what may be done with content after it is fetched, using three independent values:
search,ai-inputandai-train. Cloudflare introduced the format in 2025. - What is RSL?
- Really Simple Licensing, an open standard published in September 2025 by the RSL Collective. It lets a publisher attach machine-readable licensing and compensation terms — attribution, subscription, pay-per-crawl, pay-per-inference — to their content, discoverable through robots.txt and other mechanisms.
- Do AI companies have to obey Content Signals or RSL?
- No. Both are voluntary declarations, like robots.txt itself. They establish a clear, machine-readable statement of terms rather than a technical control.
- Should an online store block AI crawlers to prevent training?
- Usually not. In robots.txt, blocking access also blocks retrieval, so the store disappears from AI answers as well as from training data. A Content Signal that declines training while allowing search and AI input expresses the intended position without that cost.
- Is Content Signals the same as RSL?
- No. Content Signals is a short preference statement with three values. RSL is a full licensing vocabulary with commercial terms and optional protocols for acquiring a licence. They can be used together.