Blogtrottr uses a robot to periodically fetch and monitor specific RSS and Atom feeds, ensuring users receive timely updates whilst attempting to minimize resource consumption on publisher infrastructure.
The robot employs intelligent fetching strategies to reduce unnecessary data transfer:
Conditional Requests: By maintaining a record of the last seen last-modified and ETag HTTP headers, the robot leverages HTTP conditional requests to minimize redundant feed downloads.
If a feed hasn't changed since the last fetch, the server responds with a 304 Not Modified status, saving bandwidth and processing time.
The robot supports WebSub (formerly PubSubHubbub), a web syndication protocol that enables real-time content updates:
When a feed publisher supports WebSub, the robot can receive instant push notifications.
This capability allows for dynamic reduction of polling intervals, ensuring near-instantaneous update detection.
Push notifications significantly reduce the load on both the publisher's servers and the Blogtrottr infrastructure.
User-agent: Blogtrottr/2.1 (+https://blogtrottr.com/robot)