The Risk Signal crawler

If RiskSignal/1.0 is in your logs, it was us. Risk Signal tells companies which security events touch them, and we read public sources to know what is happening.

What it sends

Every request names us: RiskSignal/1.0 (+https://www.risksignal.eu/crawler/; contact hello@risksignal.eu)

What it fetches

Public feeds, public data services, and published advisories and filings. To find a site's feed, it may read the front page and try the usual feed addresses, such as /feed and /rss, a handful of requests at most. Never pages behind a login, never forms.

How often

Most feeds are read once an hour or less often. A feed is read about four times between its new items, so one that publishes once a day is read every six hours.

We send one request at a time to each host, with two seconds between them. We send back the validators your server gave us last time, so an unchanged feed costs you a 304 and nothing more.

What we keep

From each item: its title, link and date, and the text the feed gives with it, so we can tell our customers what happened and link back to you. A fetched feed itself we keep only when it brought something new, and then for 90 days.

How to stop us

Write to hello@risksignal.eu with the address of the feed or the site, and we stop within two working days. A 403 from your site is a no to us: we go through the feeds that refuse us and turn them off, and we never pretend to be a browser to get past it.