Fed Up with Search Engines in General

I maintain a blog which is for a subset of the population. This affords me the luxury of holding search engine companies accountable if they incur my wrath by thinking they are welcome anywhere and everywhere on my site.

Take google. They think it's such an honor for you to have them show up at your node that they may simply do whatever they choose on your site. My sitemaps outline the only places they may go but I sincerely doubt they even read them.

What about robots.txt ?

Ha ... what a joke.

While this may work for that multitude who values their traffic, alas I do not and when they tresspass into areas I deem off limits I try to fend them off politely.

Unfortunately, unable to take a hint they persist to the point that I just firewall as much of their network as I can.

I even include those collateral networks they bought and use for revenue streams and as alternate routes. You either stick to my sitemap or I'll make you go away. It's as simple as that.

You do not have my permission to run your custom code all over the place. Your bot is nothing to me and would just as soon firewall you as look at you in my logs walking all over my intellectual property.

Luckily, the really important stuff is password protected in the web server software. It seems that google has made contributions to Apache for which they are granted back doors in the product necessitating additional filtration scripting periodically.

However, it would seem that IP Tables hasn't received such contributions and thereby remains viable in the war against snooping and tresspass on the public domain.

I have had a number of people tell me how they can't find me in the search engines to which my reply is simply "excellent. glad to hear it". You see, some of us are private entities with content not meant for just anyone.

So if you don't like it good. It's a big internet out there.