How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag

Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it's a problem that is easily rectified.

Your primary weapon of choice against duplicate content can be found within "The Robot Exclusion Protocol" which has now been adopted by all the major search engines.

There are two ways to control how the search engine spiders index your site.

1. The Robot Exclusion File or "robots.txt" and

2. The Robots < Meta > Tag

The Robots Exclusion File (Robots.txt)
This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. www.yourwebsite.com/robots.txt. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site's content.

The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.

Creating your robots.txt file

Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:

User-agent: *
Disallow:

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. By leaving the "Disallow" blank all parts of the site are suitable for indexing.

Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.

Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation

By naming the particular search spider in the "User-agent" you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.

That's all there is to it!

As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it's probably necessary to use a combination of the robots.txt and the robots tag.

The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the section of a web page. A simple example would be as follows;

In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.

In this second example I don't want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the "noarchive" directive.

What could be simpler!

Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.

Should you require further information about our search engine marketing or optimization services please visit us at http://www.e-prominence.co.uk ? The search marketing company

---

Search Engine Ranking... Oh, the Mystery!

Rankings, Rankings, Rankings!How do you get your website ranked well... Read More

Why is Search Engine Optimisation Expensive?

Many of our customers (http://www.internet-marketing-australia.com) find search engine... Read More

Link Building for Hilltop

Hilltop is one of the major concepts underpinning Google's search... Read More

Increase Your Link Popularity for Free

Among the many things you need to worry about for... Read More

SEO Trade Secrets - 8 Great Tools for Search Engine Optimization

About 80% of website traffic comes through search engines. And... Read More

Submit All Of Your Pages And Watch Your Traffic Grow

Everyone is looking for "secrets" about how to get more... Read More

Secrets on Website Promotion: How You Can Get a #1 Ranking for Your Website Name Within 30 Days

Launching a new website with enough acceleration to rise above... Read More

Are You Losing The Battle For Search Engine Traffic?

Search engine traffic should be a priority for any online... Read More

How To Measure Search Engine Marketing ROI

According to the Search Engine Marketing Professional Organization (SEMPO), advertisers... Read More

9 Ways to Keep Google Happy

A recent Google patent application has the SEO community buzzing.... Read More

Whats Link Popularity?

Link popularity is just one of the ways you can... Read More

How Do I Submit My Site To Search Engines?

You've built up a good site, put in a lot... Read More

Marketing Articles: Getting A Better Search Engine Rank For All Of Your Pages!

In one of my articles, I discussed how to market... Read More

Search Engine Optimization - A Beginners Guide

Getting your site listed in the top search engines, such... Read More

SEO -- Tips to Optimize Your Webpage to Compete for a High Ranking

Search engine optimization (SEO) is very important to websites. If... Read More

Linking for Traffic not Positioning!

With more and more experts and search engine enthusiastsclaiming the... Read More

Link Horse Trading For The PR Challenged

After 105 days Google finally updated PR. And it's about... Read More

The Google Sandbox Explained

Introduction The Google Sandbox is a metaphorical term... Read More

SEO Made Easy!

Search engine optimization remains a minefield of old advice, outdated... Read More

SEO Tips for Google

Getting a high ranking on Google is a big achievement.... Read More

Search Engine Optimization for Beginners

If you are confused about terms like "search engine optimization"... Read More

SEO Expert Guide - Page Optimization (part 5/10)

In parts 1 - 4 you learnt how to develop... Read More

How to Improve Your Search Engine Rankings

When people think of search engine optimization, they immediately think... Read More

Sitemaps 101 - Back to SEO School

Sitemaps are without doubt one of the most often ignored... Read More

SEO for CEOs ? Search Engine Optimization Unmasked for CEOs

If you're like most other CEOs, the term "search engine... Read More

Search Engine Marketing: Are You Accidentally Hiding From Potential Customers?

You may be hiding if the search engines can't 'see'... Read More

The Business Case for SEO

It's interesting how potential clients have preconceived notions about which... Read More

Quick Keyword Research that Works!!!

Keywords are terms or words related to particular topic. Keyword... Read More

MLM and SEO - Bad Business! No Business!

MLM has been around way before the Internet. It is... Read More

How Search Engines Work

Before anyone can start optimizing a web site, you must... Read More

How To Get Listed In Google For Free

Google does not accept payment for inclusion in their main... Read More

Google Page Rank Is Dead - Part II

In part I - Google Page Rank Is Dead -... Read More

Optimizing For Visitors Not Search Engines

Most people feel that optimizing is to target the search... Read More