HTML and CSS Reference
Then check your server logs to see which IPs have actually loaded that file. Also check to see what other files
those IPs have loaded. If it's just a few files, widely separated in time, I'd ignore it. But if I see that IP address
has been hitting every other page on my site, I'll block it in my Apache .htconfig file, like so:
Allow from all
Deny from 220.127.116.11
Deny from 18.104.22.168
Deny from 22.214.171.124
This prevents it from hitting any page on my site, not just the protected ones. However, chances are that spider
is up to no good, so I don't mind doing this.
You can also use mod_rewrite to block robots by User-agent. However, it's so easy to change the User-agent
string that I rarely bother with this. I find it hard to believe that a spider that's ignoring robots.txt is not going
to fake its User-agent string to look exactly like a perfectly legitimate copy of Firefox or Internet Explorer.
Some people have automated this procedure or used other means of detection. In particular, if anyone is hitting
your site more than 12 times per minute, he may well be up to no good. However, this requires quite a bit more
server intelligence than simple IP blocking.