Yesterday our servers were being hammered from IP address : 67.195.37.172 which appears to be the Yahoo indexer. Then today I read a tweet from Ben Nadel he was having the same thing happen. So I thought I would write a quick post in case some others were having the same issue.
On the Yahoo SLURP! page they suggest adding :
Crawl-delay:10
to your robots.txt file. Where the number is the seconds of delay before it hits you with another request.
This seemed to stop the problem unless SLURP! just gave up :)
In doing more research there appears to also be a
Visit-time: 1800-2330
entry you can make but nothing supports it right now. I added it just in case.