← All posts

Helping sites tell Blogtrottr apart from bad bots

Feed requests from Blogtrottr are now cryptographically signed using Web Bot Auth, so sites can tell they really came from us.


Bot protection is everywhere these days, and it's easy to see why. AI scrapers can hammer a site, and there are more automated attacks than ever. The trouble is that a feed reader is a bot too, and even the most well-behaved bots can get caught in the same defensive net as the bad ones.

To help with this, Blogtrottr now signs its feed requests using Web Bot Auth. Sites that check these signatures can recognise Blogtrottr for what it is, which should mean fewer feeds blocked by mistake.

How it works

Until recently, there was no standard way for a site to check that a bot was who it said it was. Web Bot Auth is an emerging standard, built on HTTP Message Signatures (RFC 9421), that lets a bot attach a cryptographic signature to each request. The site can then check that signature against the bot operator's public key, and know the request really came from that bot.

If you're a subscriber

There's nothing you need to do - signing happens automatically for every feed, and on sites that check these signatures, it makes it less likely that bot protection will get in the way of your updates.

If you run a site

For now, Web Bot Auth is mostly supported by commercial bot protection services such as Cloudflare, AWS WAF, Akamai and Vercel, rather than self-hosted tools. Blogtrottr is a verified bot with Cloudflare and AWS WAF, so if you use either of those, just make sure verified bots are allowed through. Verification with Akamai and Vercel is in progress. For anything else, check whether it supports Web Bot Auth.

The robot page has the technical details.