DESCRIPTION now names Google LLC as copyright holder
of the bundled Google matcher, and inst/COPYRIGHTS gives
the copyright and license of every bundled component.
The SSRF guard now blocks more spellings of the GCP metadata host
name. The guard compared names exactly, so under the default
ssrf_guard = TRUE it allowed
metadata.google.internal. (trailing dot, in any case, with
or without a port), the bare metadata that GCE resolves
through its search domain, and spellings that UTS #46 maps to the name:
ideographic or fullwidth full stops
(metadata\u3002google\u3002internal) and fullwidth letters.
libcurl built with libidn2 applies that mapping, while the host rurl
reports by default ACE-encodes such a host whole, so the guard now also
classifies the host the WHATWG host parser reports and blocks the URL
when either view is blocked. Trailing dots are dropped before every
check, so 169.254.169.254. and 0x7f000001. are
read as the literals they spell. Host and scheme case folds are
ASCII-only: tolower() follows the locale, and under
tr_TR an upper-case METADATA.GOOGLE.INTERNAL
stopped matching (#ROBO-dxrljrde).
httr2 is now required at >= 1.2.0.
The fetch path reads bodies through
req_perform_connection() and relies on the
StreamingBody that httr2 1.2.0 introduced. Under httr2
1.1.x the body is a bare curl connection, so a fetch that ends without a
stored body leaves its connection open, a body is read whole before
max_bytes is checked instead of being cut off mid-stream,
and the mocked transport the examples and tests use is ignored, so the
examples make live requests. The R 4.1 floor check found it
(#SEOR-thlzqzac).
The engine contract now publishes which product tokens each
bounded_profiles matcher backend accepts, and what each one
does. Schema revision advances to 2026-08-25.1 on both
robots_engine_contract_v1() and
robots_engine_contract_v2(); both contract ids are
unchanged and the addition is additive, so no existing field changes
type or meaning.
The accepted sets are small and not guessable from vendor documentation:
| backend | accepted selectors |
|---|---|
yandex |
Yandex, YandexAdditionalBot |
bing |
bingbot, adidxbot |
Notably YandexBot — Yandex’s main indexing crawler, and
the value a caller is most likely to reach for — is not
accepted, while Yandex is. That is not a typo in either
direction: robots_product_token is a robots.txt
User-agent: group label, not an HTTP
User-Agent crawler identity, and Yandex is an
umbrella group that no crawler ever sends. RFC 9309 treats the two as
one namespace; vendors do not.
Each backend’s matcher_capability entry gains
product_token_role, product_token_comparison
(ascii_case_insensitive_exact for both backends) and
supported_profiles, a data frame carrying
profile_id, accepted_token,
group_label, group_selection and
profile_revision. group_selection matters:
Yandex and bingbot fall back to a
User-agent: * group when no group names them
(exact_else_wildcard), while
YandexAdditionalBot and adidxbot never do
(exact_only), so a robots.txt containing only
User-agent: * does not constrain them at all. The two
accepted selectors on a backend are therefore not interchangeable, which
is why the contract publishes rows rather than a bare token vector.
Backends whose token_policy is arbitrary_valid
or rfc9309 publish none of the three fields: their accepted
sets are not closed (#ROBO-qgxekgph, #ROBO-gcnlckao).
New exported robots_resolve_matcher_profile_v1()
checks a product token against a backend before anything is fetched or
evaluated, returning the resolved profile row — including
group_selection — or raising
robotstxtr_unresolvable_matcher_profile with the accepted
selectors named in the message. Passing an unaccepted selector to an
evaluation call is still not an error: the affected rows come back
undecided, with url_decision NA, which reads
like a transport failure rather than a configuration mistake. Inspecting
results is not an equivalent guard, because a row need never reach the
matcher at all — an allow_all policy returns a decision for
a nonsensical selector without consulting the profile. Unsupported-token
error_message values now also name the accepted selectors
and point at the resolver; the per-row status vocabulary is unchanged
(#ROBO-nvuwpktu).
Added robots_engine_contract_v2() and, through it,
the Bing matcher_backend as an available end-to-end backend
as of schema revision 2026-07-24.1, bounded to the profiles
bingbot and adidxbot; every other Bing-backend
product token resolves to a checked unsupported_profile
non-decision. The v2 schema extends the v1 status vocabulary with four
Bing outcomes — invalid_request_target,
unsupported_profile,
matcher_input_limit_exceeded, and
matcher_work_limit_exceeded — none of which can ever be an
allow/disallow decision. Structured backend identity is published at
robots_engine_contract_v2()$matcher_identity$bing. The
robots_engine_contract_v1() accessor’s own contract id is
unchanged, and its schema revision was unchanged by this addition (it
later advanced for the bounded-profile publication below). This is an
independent, unofficial compatibility profile and does not claim
production-crawler parity; it is not affiliated with or endorsed by
Microsoft, Bing, Bingbot, or AdIdxBot (#ROBO-zcgprxtq, #ROBO-lpedsigv,
#ROBO-xbwlsjzu, #ROBO-yhfjsbzo, #ROBO-qetsyvwv, #ROBO-onwulhga,
#ROBO-ahilajyn).
The SSRF guard now classifies the IPv6 unspecified and loopback
addresses on the expanded address rather than the literal string, so
every spelling of those 128 bits is treated alike —
::0.0.0.1 is recognized as loopback and
::0.0.0.0 as unspecified, matching ::1 and
::. Previously the dotted forms were classified as neither
special nor embedded IPv4 and reached the default allow. Fetches were
not affected, because rurl canonicalizes such literals
before the guard sees them; the guard is now correct on its own rather
than relying on that (#ROBO-pzgzxkoj).
The SSRF guard’s IPv6 link-local and AWS cloud-metadata rules now
match on the expanded address too, completing the change above. Because
a hextet may be written with leading zeros, matching the literal string
mis-decided both rules: fd00:0ec2::254 was allowed through
while the identical fd00:ec2::254 was blocked, and
addresses such as fe8:: were reported as link-local despite
lying far outside fe80::/10. Both are now decided by value
— fe80::/10 and fd00:ec2::/32 — so every
spelling agrees (#ROBO-pjvypcjk).
The SSRF guard now decodes the three IPv6 transition mechanisms —
6to4 (2002::/16), Teredo (2001::/32, whose
embedded IPv4 is XOR-obfuscated), and ISATAP (a *:5efe
marker under any prefix) — and classifies the IPv4 address each one
wraps. All three pack that address outside the low 32 bits, so the
previous decoders missed them and, for example,
2002:a9fe:a9fe:: (6to4-wrapped
169.254.169.254) was allowed. A wrapped public
address still passes: these prefixes are globally reachable, so only
what they carry decides the outcome (#ROBO-laydgesq).
An IPv6 literal that cannot be expanded to exactly 8 hextets
(fe80:::1, ::12345) is now refused with the
reason malformed-address instead of reaching the default
allow. Such literals never survive URL parsing, so no reachable fetch
changes; the guard simply no longer depends on rurl
rejecting them first (#ROBO-udnyuuwn).
An IPv6 literal ending in a single : is now refused
with malformed-address rather than read as the address
without it. R’s strsplit() keeps a leading empty
field but drops a trailing one, so the guard’s arity check
counted 1:2:3:4:5:6:7:8: as eight groups and
::1: as ::1: the first was allowed, the second
classified as loopback, and ::ffff: decoded on to the
IPv4-compatible 0.0.255.255. Each landed on the same
verdict as its well-formed spelling, so no blocked range became
reachable — what failed was the malformed-address invariant
above. A trailing . is deliberately still accepted,
matching the WHATWG host parser (#ROBO-zavqklmi).
Two SSRF reason codes are corrected. 100.64.0.0/10 —
RFC 6598 Shared Address Space, i.e. carrier-grade NAT — was reported as
cloud-metadata; it is not a metadata range, it merely
contains one provider’s endpoint, so a consumer keying off that code
misattributed every CGNAT address it saw. It is now shared.
And only 0.0.0.0/32 is the unspecified address (RFC 1122
§3.2.1.3), so the rest of 0.0.0.0/8 is now
this-network (RFC 791 §3.2) instead of reporting all 16 777
216 addresses as unspecified. Both blocks remain
blocked and no address changes verdict — only the reported
code. 169.254.169.254 keeps its deliberate
cloud-metadata label inside the link-local /16
(#ROBO-cjnsrmgd).
A fetched robots.txt carrying an embedded NUL byte
no longer aborts the call. robots_body(),
allowed_by_robots_url() and
robots_evaluate_url_v1() decoded acquired bytes with a bare
rawToChar(), which raises
embedded nul in string, so one stray NUL — a UTF-16-encoded
file, a WAF challenge page, a truncated or NUL-padded response — turned
a best-effort fetch into an error. NUL bytes are now removed before the
bytes become an R string, the same rule
robots_validate_text() already applied: the document is not
truncated at the NUL and no other byte is rewritten, so every directive
the file spells out still applies. The stored body stays byte-exact
(robots_body(raw = TRUE)), and the malformation stays
visible — robots_validate_text() still reports it as a
nul_byte error diagnostic (#ROBO-hcqrzikz).
The pkgdown documentation site is built and published by GitLab
CI at https://bart-turczynski.gitlab.io/robotstxtr/, replacing
the GitHub Pages site that went away with the suspended
bart-turczynski GitHub account (#ROBO-qqwkldzv).
The documentation site’s address is now the GitLab Pages
namespace path https://bart-turczynski.gitlab.io/robotstxtr/, the
fleet-wide standard, instead of the project’s unique Pages domain, which
no longer resolves. DESCRIPTION, CITATION.cff,
.zenodo.json, codemeta.json and
SECURITY-INSIGHTS.yml all carry the new address
(SEOR-hcmtspmv).
Package metadata now points at GitLab. URL: is https://gitlab.com/bart-turczynski/robotstxtr plus the
documentation site, BugReports: is https://gitlab.com/bart-turczynski/robotstxtr/-/work_items,
and codemeta.json, CITATION.cff,
inst/CITATION, .zenodo.json,
SECURITY-INSIGHTS.yml, THIRD_PARTY_NOTICES.md,
inst/NOTICE, inst/PROVENANCE and the README
badges follow. The vulnerability-reporting instructions in
SECURITY-INSIGHTS.yml now describe GitLab’s confidential
issue, which is what replaces GitHub private vulnerability reporting.
Upstream citations of github.com/google/robotstxt are
unchanged — that is the C++ project this package vendors
(#ROBO-bgcloodt).
The BugReports: field in DESCRIPTION
now reads
https://gitlab.com/bart-turczynski/robotstxtr/-/issues,
superseding the /-/work_items form the metadata entries
elsewhere in this unreleased section record.
tools:::.check_package_CRAN_incoming() accepts a gitlab.com
bug tracker only when the path ends in /-/issues, and the
/-/work_items form is what got a sibling package archived
at the CRAN incoming pretest. GitLab answers /-/issues with
404 to a signed-out, non-browser client and redirects a browser to the
work-items view, so the address a reader clicks still resolves;
cran-comments.md explains the resulting URL note. The
incoming check reads no file but DESCRIPTION, so
codemeta.json — which is .Rbuildignored and
read only by humans — deliberately keeps the /-/work_items
address, which returns 200 (ROBO-npiueuey).
rurl is now required at >= 3.0.1, up
from >= 2.2.1. The old floor named a version that was
only ever a source-forge tag and never reached CRAN, whose
rurl releases are 1.2.0 (archived) and 3.0.1; 3.0.1 is the
lowest version a CRAN install can satisfy, and the full test suite
passes against it installed from CRAN. The installation instructions no
longer set up the R-universe staging repository, since every dependency
now comes from CRAN (#ROBO-depmsgpz).
The agent instructions no longer import FP_AGENTS.md
or FP_CLAUDE.md, the files the fp tracker
generates, which are deleted. AGENTS.md points at the house
agent-workflow and fp skills for the git
workflow (SEOR-ipwcbcov).
Two dependency vulnerability audit jobs, osv-audit
and security-audit, run
tests/testthat/test-osv.R and
tests/testthat/test-security.R on a pipeline schedule that
sets SCHEDULE_KIND=dependency-audit, or by hand from a
web-started pipeline. security-audit sets
OSSINDEX_AUDIT_REQUIRED=true. Both install the package
first, because the OSV test resolves the dependency closure from
installed packages and skips without it, and both fail if any audit test
skipped (SEOR-fftbjnpl).
scripts/check-bugreports.py, run on pre-push and in
the citation-version CI job, keeps
DESCRIPTION’s BugReports: on the
CRAN-incoming-safe /-/issues form and fails if a
human-facing file links that form instead of /-/work_items
(SEOR-ocbtrrnl).
The OSS Index allow-list in
tests/testthat/helper-security.R is now checked by a
validator, oss_index_allowlist_violations(), that returns
findings and is itself tested against broken fixture rows, instead of a
loop of expectations that asserts nothing once the list is empty. The
two curl rows are unchanged and pass it. The audit test
gains a loud-failure mode: under
OSSINDEX_AUDIT_REQUIRED=true, missing oysteR
or missing OSS Index credentials fail the test instead of skipping it,
so a credentialed audit job cannot report success having audited
nothing. No job sets the flag yet. An audit that resolves no packages
now fails rather than passing vacuously (SEOR-fftbjnpl).
DESCRIPTION’s URL: now lists the
package’s r-universe page. r-universe records this repository’s upstream
owner as gitlab-bart-turczynski because it is hosted on
GitLab, which does not match the bart-turczynski universe,
so the package was built and served but hidden from r-universe search.
The URL claims it (SEOR-zfamoutf).
CI now creates exactly one pipeline per merge, on
main, instead of three (branch, merge-request, and main).
Merge-request and feature-branch pipelines are suppressed via a
top-level workflow: block; the dead
pkgdown-preview MR-only job was removed along with them.
Feature-branch pushes get no CI of their own — the local pre-push verify
gate is what blocks bad code before it leaves the machine — and GitLab’s
per-line coverage annotations in merge-request diffs no longer appear,
since no pipeline attaches to the MR. A pipeline started by hand from
Build > Pipelines > Run pipeline is the exception: it works
against any ref and runs the full gate there, so a branch can still be
verified on the server before it merges. pages is pinned to
main and is not reachable that way
(SEOR-bmgkzhvy).
codemeta.json now declares the
issueTracker that DESCRIPTION’s
BugReports: already did, https://gitlab.com/bart-turczynski/robotstxtr/-/work_items.
It was the last tracked copy of the retired issues path, which GitLab
has returned 404 for since issues moved to work items platform-wide; the
file is .Rbuildignored, so no shipped file changes
(ROBO-zghpvlxu).
CI installs rurl from its v3.0.1 tag
instead of tracking rurl’s default branch, so the pipeline no longer
tests robotstxtr against rurl’s development head
(ROBO-hzcnolhe).
The OSS Index dependency audit in
tests/testthat/test-security.R scopes to hard dependencies
(Depends + Imports) instead of the
Suggests tree, and allow-lists by ID the two
curl advisories the narrower scope still reports.
curl is a genuine hard dependency here, reached through
httr2, and both CVE-2026-18924 and CVE-2026-3783 name
libcurl ranges covering CRAN’s current curl 8.0.0 — there
is no version to upgrade to, so the gate had been blocking every push
since 2026-09-06. Each allow-list row carries a written reason and a
review date, and helper-security.R enforces three rules: an
advisory that is reported and not allow-listed fails, an allow-listed
advisory that is no longer reported fails, and drift past a review date
or a package version warns (ROBO-oykjiyjj).
CI folds the lint, readme,
docs, vendor-fidelity:yandex and
vendor-fidelity:bing verify-stage jobs into one
gates job (dev/gates.R), cutting four runner
pickups per pipeline while keeping the combined ~2m9s of gate compute
unchanged. citation-version stays its own job, on its own
python:3.13-alpine image, rather than folding into the R
toolchain’s image just to save one more pickup; check,
coverage and pages also stay separate. The new
job runs every folded gate regardless of an earlier one failing, then
reports one summary naming every gate’s verdict before exiting non-zero,
so a red pipeline still names every broken gate rather than just the
first one (SEOR-pgammbgo).
Added robots_validate_text() and
robots_validate_url() for stable, machine-readable document
validation under an explicit google-parser-compatible
profile. Reports include document byte/line and directive evidence,
per-line syntax and group diagnostics, BOM/NUL/UTF-8 hazards, and
acquisition-limit outcomes without parsing truncated bodies
(#ROBO-htvtajbr).
Added the parallel robotstxtr.engine-aware/v1
evaluation contract through robots_evaluate_text_v1() and
robots_evaluate_url_v1(). Policy rulesets, matcher
backends, robots product tokens, HTTP User-Agents, and acquisition
limits are explicit, independent, and persisted in every result
(#ROBO-unowhvjx).
Added neutral acquisition evidence and explicit non-decision states for safety refusal, incomplete or partial evidence, crawler lifecycle context, documentation gaps, and unavailable matchers. The pinned Google matcher is the first available end-to-end backend; it applies its 500 KiB prefix limit without relabeling itself as another engine (#ROBO-unowhvjx).
Added robots_engine_contract_v1() for
revision/capability negotiation and
as_legacy_robots_decisions_v1() as the explicit bridge to
the unchanged Google-oriented legacy schema (#ROBO-unowhvjx).
Added the Yandex matcher_backend as an available
end-to-end backend as of schema revision 2026-07-18.2,
bounded to profile yandex-0.1.0 and the crawlers
Yandex and YandexAdditionalBot; every other
Yandex-backend product token resolves to a checked
unsupported_crawler non-decision. Yandex-evaluated results
expose the new public matched_rule_value_raw list column
(exact owning-rule bytes, raw(0) for an effective-empty
Disallow, NULL for an absent rule), and
structured backend identity is published at
robots_engine_contract_v1()$matcher_identity$yandex. This
is an independent, unofficial compatibility profile and does not claim
production-crawler parity (#ROBO-sbyndsws, #ROBO-mhzmembp).
Development installs and CI now resolve the pre-CRAN
rurl dependency chain through the author’s CRAN-like
R-universe instead of nested GitHub remotes (#ROBO-yasmzyhl).
The fetch policy now enforces a structural SSRF guard:
robots_fetch() and allowed_by_robots_url()
refuse — before opening any socket — both the initial origin and every
redirect target that resolves to a private, loopback, link-local, or
cloud-metadata address (including IPv6 loopback, IPv4-in-IPv6
embeddings, and numeric/hex/octal literal obfuscation). Such a URL is
reported with the new ssrf_blocked fetch outcome and yields
a fetch_unknown (NA) decision, never a silent
allow (#ROBO-quovenef).
The guard can be disabled per call with the new
ssrf_guard argument (TRUE by default) on
robots_fetch() and allowed_by_robots_url(),
for deliberate use against trusted intranet hosts
(#ROBO-quovenef).
allowed_by_robots_text() — evaluate crawl permission
against a robots.txt document supplied as text.allowed_by_robots_url() — evaluate crawl permission for
a URL, fetching the relevant robots.txt as needed.robots_fetch() — fetch a robots.txt
document under a deterministic, conservative policy.robots_body() — preview or extract a stored
robots.txt body.