Using robot.txt

For decades, website owners could control how search engine crawlers and other software bots interact with their sites. This powerful tool, known as robots.txt, has been an integral part of HTML since its early days. Let's dive into what robots.txt is and how you can use it effectively.

What is robots.txt? It is a simple text file placed in the root directory of your website. It provides instructions to web crawlers about which parts of your site they're allowed to access and index.

How does it work?

 The file typically contains at least two types of instructions:

1. User-agent: Identifies the crawler being addressed
2. Disallow: Specifies which parts of the site the crawler shouldn't access

 A basic example might look like this:

User-agent: *                                    
Disallow: /                                        

The downside of this approach is that Google’s will not Index your website and therefore it won’t be found in any Google searches. Probably not what we want. An alternative is to identify a folder on your website (if you have folders). In this case my images are all stored in the subdiretory /images. 

Disallow: /images/                           

It is easy to make your robot text file much more specific. For example I don’t want chat gtp’s web scraper to go through my website you can stop so I need to know the agent name that OpenAi are using and I got this information from 20i article How to stop AI scraping your website (see see video below) 

User-agent: GPTBot                    
Disallow: /                                    

At this point in time I not sure this approach is reliable because I suspect that a lot of Bots that are scraping the Internet are just ignoring robot text.  it’s clearly a voluntary exercise. However it is one step that demonstrates you want to be opted out of the datasets collected by specific AI developer, which may come in handy in future.

A more regulated way to opt-out os such indiscriminate content scrapping would be nice?

Avoiding the Black Hats & Spammers

I have become aware that at least one of my blog posts has again been “scrapped” and reposted as an exact copy of my original post, including picture and layout, on a spam site (I will not give you the link because you should not go there, and specially never every click on links you find on these dodgy looking “content farms”). At worse they will contain links to other spam and potentially malware. At best these lazy spamdexers are just trying to attract innocent links, thus page views and a higher google ranking, to their site using lots of plagiarized (aka stolen) content. It is a dishonest practice, and sorry I can do nothing about it.
What disappoints me most about all this black hat SEO trickery and blog scrapping is the fact that Google in many cases is paying the very spammers and thieves for the stolen content! The content farms have mostly been greedy & also monetized their site using Google’s own adsense. So try reporting this Google??? I have found there is a massive blankness, no response from anyone. You may remember my previous post on a lesson of theft versus sharing. Well that site, which looks a lot like a legitimate camera review site, still exists, it just keeps changing its name slightly, its ISP, its country of origin and its owner name, so that DMCA notices to cease and desist also go unanswered. At least that low life now acknowledges my posts and links back to my site (he may think technically he doesn’t breach my Creative Common Licence, but what he does is still not legal because he is expecting Google to still pay him for my content and my CC license is for NON commercial).  I can not find him, but Google obviously still can because he still displays an adsense badge. Am I being too cynical about Google’s apparent hypocrisy and will to fix this? Rant over for now.

As I indicated above there is little I can do, but there might be something you can do if you use chrome as your browser, Google has released an add-in that gives you a personal blocklist of such sites as you come across them. It will also transmit your blocking patterns back to Google, in theory they will then get a good crowd sourced blacklist of such websites and thus downgrade these undesirables in their famous search algorithms. Let us hope Google will follow through on this.