revjim.net

Mark Pilgrim writes regarding how to block spambots, ban spybots, and tell unwanted robots to go to hell. Even if you aren't having trouble with evil website robots (programs that download every page of your website in search of various pieces of information that are rarely of any value to you) or don't intend to do anything to fight back it's worth reading just to see how many of them are out there. Mark has done a great job of compiling a list of such offenders and giving detailed instructions for forbidding them to access the data on your webserver. After all, why should we pay for the bandwidth they use to eventually provide services that hurt us.