The Internet is Rude
I've been pondering the internet of late in the context of my web server with the various others thinking they can go anywhere and everywhere despite the content of robots.txt in the mix.While I would certainly be naive to presume that the stated rules would be followed in the context of my server there is this notion of retaliatory firewalling as a last resort against these interlopers ... I mean, seeing how I'm not interested in any search engine standings, results, or anything else artificial in my life.
 |
So ... I have began capturing the IPs of those offending search engines and others like the various homespun bots and those scrapers on amazon's EC2 den of iniquity out there seeking to go places I would rather they didn't.
Failure to do what they want to do is not one of those omnipotent scenarios I would prefer ... like nuking some renegade ISP or two or three; but it will have to suffice. Besides, in all fairness the technical folks at amazon are fine and dandy and cannot really help the exceeding low to absent character of some of those calling themselves 'customer'.
In the sweet by and by it will be of little consequences, but I now know better to clear firewall rules then reaccumulate them as I have done with the web server.
Suffice it to say that I am seriously considering a restore from backup and clearing out the riff raff almost totally yet again as it was before I grew all open circuit warm and fuzzy around the Christmas holidays.
I suppose a web server that would enforce robots.txt to a tee would suit me just fine. I would really like a web server that encountered a client in violation of some disallow and shut them down with extreme prejudice. Such a feature rich server environment would really get my attention. It would also be a stupendous hoot.
To hell with all of the rude spiders, bots, search engines, scrapers and BOY am i getting more firewall inclinations as I speak!