| Internet-Draft | Signature-Agent for Robot Exclusion Prot | May 2025 |
| Meunier | Expires 30 November 2025 | [Page] |
This document describes a new directive to allow Signature-Agent (Section 4 of [SIGNATURE-DIRECTORY]) directive in the Robot Exclusion Protocol ([RFC9309]).¶
This note is to be removed before publishing as an RFC.¶
The latest revision of this draft can be found at https://thibmeu.github.io/http-message-signatures-directory/draft-meunier-signature-agent-rep.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-meunier-signature-agent-rep/.¶
Discussion of this document takes place on the HTTP mailing list (mailto:ietf-http-wg@w3.org), which is archived at https://lists.w3.org/Archives/Public/ietf-http-wg/.¶
Source for this draft and an issue tracker can be found at https://github.com/thibmeu/http-message-signatures-directory.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 30 November 2025.¶
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Bots are increasigly using Signature-Agent as a way to convey identity. As such, there is interest from Origins to define robot policy based on this header.¶
This documents extends Robot Exclusion Protocol to support these, by defining a new group starting with signature-agent.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
One or more signature-agent lines that are followed by one or more rules. The group is terminated by a signature-agent line or end of file. See Section 3.2.1. The last group may have no rules, which means it implicitly allows everything.¶
Based on the formal syntax defined in Section 2.2 of [RFC9309]¶
startgroupline = user-agent-line / signature-agent-line ; a group can either be a user-agent, a signature-agent, or both. user-agent-line = *WS "user-agent" *WS ":" *WS product-token EOL signature-agent-line = *WS "signature-agent" *WS ":" *WS directory-token EOL directory-token = fqdn fqdn = ... ; domain as defined by signature-agent¶
Crawlers set their own identity, which is called a directory-agent, to find relevant groups. The directory token MUST contain only lowercase letters ("a-z"), underscores ("_"), hyphens ("-"), and dots ("."). The directory token SHOULD be a valid FQDN suffix of the identification string that the crawler sends to the service. For example, in the case of HTTP [RFC9110], the product token SHOULD be a substring in the Signature-Agent header. The identification string SHOULD describe the public cryptographic key material of the crawler. Here's an example of a Signature-Agent HTTP request header with a link pointing to a page describing the purpose of the crawler.example.com crawler, which appears as a suffix in the Signature-Agent HTTP header and as a directory token in the robots.txt directory-agent line¶
+==========================================+==============================+ | Signature-Agent HTTP header | robots.txt signature-agent | | | line | +==========================================+==============================+ | Signature-Agent: crawler.example.com | signature-agent: example.com | +------------------------------------------+------------------------------+¶
Signature-Agent group shares the security consideration of [RFC9309]. In addition, given Signature-Agent MAY present a domain name identifying crawlers public cryptographic key material, implementors should treat the content of signature-agent line as possibly sensitive.¶
This document has no IANA actions.¶
Signature-Agent: example.com Allow: *¶
Not in the draft yet. We don't want to incentivise not rotating public keys.¶
Signature-Agent: poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U Disallow: /path/to/resource¶
TODO acknowledge.¶