Skip to content

For website owners

ReachLeadBot, the crawler that reads our customers' websites

Do you see ReachLeadBot in your server logs? A ReachLead customer gave the address of their own website to describe their offer. Here is what this crawler does and how to keep it away.

Crawler signature

ReachLeadBot/1.0 (+https://reachlead.io/bot)

What it does

It reads at most 12 public pages of a website, the home page and the pages that describe the services, the company or its references, at the moment a ReachLead customer types in that address. It fills in no forms, signs in nowhere and does not leave the site it was given.

Its limits

Each page must respond within 5 seconds and weigh less than 2 MB, otherwise it is skipped. The crawler only reaches public addresses, on ports 80 and 443. Never an internal network.

How to block it

ReachLeadBot respects the robots.txt file. It follows the ReachLead or ReachLeadBot group if it finds one, otherwise the general group, and it also respects the nofollow and noindex meta robots tags as well as the rel=nofollow attribute on links.

To close your whole site to it, add these two lines to your robots.txt file. That is all.

User-agent: ReachLeadBot
Disallow: /

If your robots.txt file does not respond because of a server error, the crawler prefers not to read anything at all.

What happens to the text it reads

The text it reads is used to suggest to the customer a description of their offer and their target. The customer confirms or corrects it, and nothing is saved before that confirmation.

A question

Write to hello@reachlead.io and give us your website address. We will get back to you.