CRAYONBot
CRAYONBot is the program that gathers headlines for CRAYON, a news aggregator that ranks stories by how many independent outlets are covering them. We publish no articles of our own: every headline is the outlet's own, credited and linked to the original. This page is for publishers and site operators who see us in their logs.
How to recognize us
Requests carry this user agent:
CRAYONBot/1.0 (+https://crayon.net/about/bot.html)They come from our server, zulu.maher.org. Its outbound addresses are
currently 107.22.178.194 (IPv4) and 2600:1f18:1b76:f200:b0f7:c263:7b67:5eba
(IPv6). Addresses can change; the user agent is the stable identifier.
What we fetch, and how often
- Your public RSS or news-sitemap feed, one request about every ten
minutes. We send
If-None-Match/If-Modified-Since, so an unchanged feed costs you a tiny 304 response. We use at most eight connections at once across all outlets, and never more than one to a given feed at a time. - An article page, once, for some stories, to read the page's own
social-preview metadata (its description and
og:image), the same data a link preview on a social network reads. We don't store or republish article text. - Readers' own feeds. Signed-in readers can add feeds to their personal "My Sources" panel. We fetch those with the same user agent, only for public addresses, with a short timeout and a size cap.
- We occasionally check that a link to a long-standing website is still alive, using
the user agent
Mozilla/5.0 (compatible; CRAYON link check).
What we show
The headline, the outlet's name, a link to the original, and the thumbnail from the feed where one is provided. We don't copy images to our servers: the reader's browser loads them directly from your site. For some outlets we ask your image server for a smaller size that your own site already publishes. Nothing here replaces visiting your site; it sends readers there.
Not a fit? Tell us
If you would like us to stop, to slow down, to fetch from a different address, or to allow CRAYONBot through a bot filter that is blocking it, write to [email protected] (Dave Maher, who runs CRAYON). A person reads it. We'll stop fetching your feed and remove your outlet from the site promptly on request, and we want to hear it if our traffic is causing you any trouble.
Looking for how we rank stories? See our methodology and editorial standards.