Lieu crawls fast; it seems to crawl at a rate of 5req/sec by default.
Normally, this wouldn't bee too worrying; however, the types of sites Lieu crawls are small hobbyist sites. Some peoples' sites may be self-hosted on limited hardware.
Respecting a crawl-delay robots.txt directive could help avoid overwhelming smaller sites.
Lieu crawls fast; it seems to crawl at a rate of 5req/sec by default.
Normally, this wouldn't bee too worrying; however, the types of sites Lieu crawls are small hobbyist sites. Some peoples' sites may be self-hosted on limited hardware.
Respecting a
crawl-delayrobots.txt directive could help avoid overwhelming smaller sites.