The Role of the Robots.txt File to Improve Site Ranking!

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

---

Surviving the Search Wars ? Local Directories

The pursuit of online information has become an increasingly dynamic... Read More

The Latest Craze: Local Search, 7 Steps to Being #1 in Your Local Market

Anyone would agree that it is much easier to be... Read More

Website Optimization, Good Overall Optimization is Key

Good overall optimization, the right keyword phrases and quality content... Read More

Submitting Your Site To The Open Web Directory: Some Dos And Don?ts

One of the most important steps in any site's publicity... Read More

Beyond Search Engines

Some webmasters report that search engines account for 75% or... Read More

DIY SEO

Part 1. Wordtracker for keywords.A problem for all new webmasters... Read More

A Way for Search Engines to Improve

Wouldn't it be nice if the search engines could comprehend... Read More

DIY SEO - Part 2 Tags n Things

Part 2 Tips 4 tagsMeta tags are the descriptive tags... Read More

SEO = Search Engine Optimization, tips on successful page ranking

One of the key things to remember when developing your... Read More

Importance of Keywords in Anchor Text or Title Text

Keywords are indisputably, the single most important element of an... Read More

Torpedo and Sink the Ship SS Search Engine Ranking

I was recently contacted by one of my best clients... Read More

Boost Your Website Rankings with Expert Content

The chase for a high web ranking is constantly on... Read More

Part I : Getting Free Hits Using These Simple Tips & Tricks

Search Engine OptimizationSearch engines still remain the #1 tool to... Read More

Why Search Engine Traffic Should be Your Top Priority

Most Internet marketing methods are risky and many will not... Read More

All You Need to Know About Your Website Traffic is Contained in 5 Key Stats

You built that content rich web page optimized for your... Read More

Tips On Optomizing Your Website For Search Engines

I wish to give a few little tips, about optimizing.... Read More

HTML Title Tags Dictate Your Rankings

There are a lot of things in Search Engine Optimization... Read More

Googles Trap, DMOZs Nap, And Yahoo!s Crap

On November 16th, 2003, Google commenced an update (the Florida... Read More

How To Stay One Step Ahead of the Google Dance

It's taken you 6-months of hard work, constant changes, reading... Read More

Why You Need a SEO Maintenance Plan

A search engine optimization maintenance service plan will ensure that... Read More

Local Search Optimization - A Guide to Getting Started

While searching the web these days, it's hard not to... Read More

SEO Hints and Tips and Free SEO Tools

Do you realize that if you manage your website, SEO... Read More

Is This A ?DMOZ Killer? In The Making?

Is building a Human-Edited Fully "Democratic" Search Directory Possible?The independent... Read More

The Google Sandbox Explained

Introduction The Google Sandbox is a metaphorical term... Read More

What is Google Talk ?

The advent of Internet has affected lives of almost the... Read More

Ten Steps To A Well Optimized Website - Step Seven ? Website Submissions

Welcome to part seven in this ten-part search engine positioning... Read More

The Simple Formula To Search Engines

Search engines are one of the best tools to bring... Read More

An SEO Checklist

Search engine optimization is on every webmaster's mind these days.... Read More

How To Get Your Web Site Banned ? or Search Engine Promotion, How Not to do it!

Search engines constantly strive to advance their technology and algorithms... Read More

Is New MSN Search More Precise? Just Ask Google

MSN finally unleashed its new search technology to the world... Read More

The Importance of Search Engine Optimization

Search Engine Optimization is a key to any successful internet... Read More

Search Engine Updates vs. SEO

Webmasters always anxiously wait for a search engine update. Those... Read More

10 Ways to Make Sure Your SEO Goes Out of Its Way for You

If they want to have success, companies should do everything... Read More