The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

---

The (Not So) Shocking Truth About Getting A High Search Engine Ranking

I have seen my site hit #3 at Google, and... Read More

Get A Top Position In Google In Three Days!

So how do you get a top listing in Google... Read More

Get Personal With Google

Anyone who's been on the net any time at all... Read More

Look Out MSN Search, Here Comes Gbrowser

It is official, the search engine wars are in full... Read More

What All Top Google Websites Have in Common

While it is okay to admire and envy all those... Read More

How to Get Listed in Yahoo Within 48 Hours, Without Paying $299

First of all you need to get a blog. If... Read More

Search Engine Optimization: Creative Ways To Acquire Natural Back Links

Search engines use algorithms calculate the order in which the... Read More

Are You Losing The Battle For Search Engine Traffic?

Search engine traffic should be a priority for any online... Read More

Google Rankings ? Achieving a Top 10 Position in Google ? Part 1

Achieving a top ranking position in Google is every webmasters... Read More

The Google Sandbox Explained

Introduction The Google Sandbox is a metaphorical term... Read More

Search Engines Are Going to Love You for This

What's the most potent free traffic source on the web?***... Read More

Get More Cosmetic Surgery Patients From The Web

It's no surprise that dominant cosmetic surgery practices also have... Read More

Search Engine Optimisation - Getting Targeted Traffic

Getting SEO right is an art and it's based on... Read More

Over Optimization and the OOP - Does a Penalty Exist?

If you have questions about whether or not the Over-Optimization... Read More

Organic SEO: Patience For Long Term Ranking Results

When does long term SEO show ranking results? It takes... Read More

One Way Linking Campaigns II

There is a way to generate links with the content... Read More

Ten Steps To A Well Optimized Website - Step 5: Internal Linking

Welcome to part five in this search engine positioning series.... Read More

Marketing Articles: Getting A Better Search Engine Rank For All Of Your Pages!

In one of my articles, I discussed how to market... Read More

Determining the Value of Your SEO Service

Every once in a while--and probably more often than we... Read More

Cache in the Bank: Understanding Googles Advanced Operators

If you would like to know when your site was... Read More

Google: The Ultimate Web Writer?s Style Guide

Indulge me for a moment.Forget that Google is a search... Read More

The Wonders of Wordtracker: Its More than a Hunt for Keywords

For me personally, Wordtracker.com is not just a... Read More

Every Search Engine Robot Needs Validation

Your website is ready. Your content is in place, you... Read More

See No Google, Hear No Google, Speak No Google

That's right - I dreamt of a World Wide Web... Read More

Good Content: The Key To Search Engine Ranking

Good Content: The Key To Search Engine RankingWhat drives visitors... Read More

The Long And Short Of It Is That These Two Sales Techniques Are The Same

With the Internet beginning to stand up and be counted... Read More

Top Search Engine Ranks, Part 2- Mastering the Secret- Explained

In the first part of this series on ranking at... Read More

Optimum SEO Keyword Density ? A Real-Life Case Study

So you've built your website, you know what keywords you... Read More

Finding the Right SEO Company

I often talk to people who have lost faith in... Read More

Getting Noticed by Search Engines

We all, meaning us webmasters want to have the best... Read More

Creating a Google SiteMap For Your Work At Home Business Web Page

Search engine traffic is the best traffic You can get... Read More

Search Engine Optimization for Everyone

There are many facets to SEO and the search engines... Read More

High Search Engine Rankings - A Long Term Strategy

The last 1.5 years have shown major changes in search... Read More