Slowing down Yahoo! Slurp (Yahoo! Search Bot)

A few weeks ago one of our servers started to crash every few days.

After investigating this issue for a while we found 100s of IP addresses from Yahoo! and Inktomi Corporation which is the company who developed Yahoo! Slurp.

So basically Yahoo! was launching DOS attacks against our server although unintentional but very annoying. It also shows that their technology is not as advanced as Google or other search engines or it’s buggy, Whatever you think, crashing people’s servers with your search bot is not cool at all and shows that you have to work on it a little more…

If you have the same problem here is a simple solution Yahoo! suggest; open your robots.txt (create it if you don’t have it and place it in your root folder) and add these lines to it:

User-agent: Slurp
Crawl-delay: 1

Only Yahoo! will understand the line Crawl-delay: X and Yahoo! suggests that you use a small number between 0.5 and 1.