RoleDawnBot: our crawler

RoleDawnBot is the crawler behind RoleDawn. It reads public job postings that companies publish on their own applicant tracking system (ATS) job boards, and nothing else.

How to recognise it

Every request carries this User-Agent:

RoleDawnBot/1.0 (+https://roledawn.com/bot)

The robots.txt product token is RoleDawnBot.

What it fetches

Public job-board endpoints and pages on Greenhouse, Lever, Ashby, Workable, Recruitee and Workday: the list of open roles on a company's board, and for a small number of roles the public job description.
The robots.txt file of each host, before anything else on that host.
It does not log in, does not submit forms or applications, does not fetch candidate or employee data, and does not collect personal contact details.
Each tracked board is read about once a day.

The rules it follows

robots.txt first. Before requesting anything from a host, RoleDawnBot reads that host's robots.txt and follows it (RFC 9309): a path that is disallowed for RoleDawnBot or for * is never requested. robots.txt is cached for at most 24 hours. If robots.txt cannot be read because of a server error, the host is not crawled until it can.
Rate limits. At most 1 request per second to Lever, and at most 2 requests per second to any other domain, or slower when the site's robots.txt sets a Crawl-delay. A 429 Too Many Requests answer pauses requests to that domain for the time given in Retry-After (60 seconds when none is given).
Sources we do not use. We do not crawl SmartRecruiters, whose API robots.txt disallows crawlers, and we do not crawl LinkedIn or other social networks.

How to opt out

Block it in robots.txt. Add the lines below to the robots.txt of your careers host. RoleDawnBot picks up the change within 24 hours and stops requesting the disallowed paths.
```
User-agent: RoleDawnBot
Disallow: /
```
Or ask us. If you cannot change robots.txt (for example, your board is hosted by your ATS), send us a message with your company name and careers page. We remove the board from our tracking list and stop showing new signals for it, usually within 2 business days.

Contact

Questions, a crawling problem, or a removal request: contact form. We reply by email within 1–2 business days. How we turn postings into signals is described on the methodology page.