Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

---

Hiring An SEO Constultant - 10 Reasons Why You Should

It crosses every webmaster's mind anytime they see an ad... Read More

SEO #2: On-page Optimization

Yesterday you should have read the first course out of... Read More

Google Page Rank Is Dead - Part III

HELP! My PR page rank is grey, call the development... Read More

7 Search Engine Optimization Mistakes and Solutions

To many websites, webmasters discover that major sources of website... Read More

Internet Directory Submission Tips

Internet Directories and their ImportanceThere are two very pertinent reason... Read More

Future of SEO - Making Money Online from Your Home and Building Homes from Making Money Online

Everyone seems to want the benefits from working at home:... Read More

SEO Expert Guide - Paid Site Promotion (Marketing) (part 7/10)

In parts 1 - 6 you learnt how to develop... Read More

Local Search Optimization - A Guide to Getting Started

While searching the web these days, it's hard not to... Read More

Have You Heard Of Website Optimization

Have you heard of website optimization ? If you are... Read More

The Golden 5: Steps to Google Success

The Dream: You wake up one morning and notice your... Read More

Find Best Keywords For Your Site

Keyword optimisation is probably the most important thing that you... Read More

10 Ways To Indirectly Get To The Top Of Search

There are millions of web sites trying to get listed... Read More

How Do I Improve My Web Site Conversion Rate? Part 2

Question 1Does it help to track visitor behavior on websites... Read More

Free Search Engine Advertising: 10 Secret Ways To Indirectly Race To The Top Of Search Engines

Do you have a website that has little or no... Read More

SEO Expert Guide - Free Site Promotion (PR) (part 6/10)

In parts 1 - 5 you learnt how to develop... Read More

The 7 Points of Do-It-Yourself SEO

Ever felt intimidated at the convoluted, jargon-ridden information about Internet... Read More

Google Groups

Some very early users of the Internet - not the... Read More

Are Your Keywords Making Money for You?

I built my website, it's perfect. My chosen subject of... Read More

Search Engine Spam

Running an online business relies to a greater or lesser... Read More

Website Optimization, Good Overall Optimization is Key

Good overall optimization, the right keyword phrases and quality content... Read More

Banned By Google And Back Again

The date: 29th July 2005. The time: early morning. I... Read More

10 Quick Ways To Kick-Start Your Profit Pulling Keywords

First, you must realize that targeting the right keywords or... Read More

The Great Search Engine War, Where Content is King

When search engines first appeared, they were simple affairs consisting... Read More

Googles Back Door... Get Your Website Indexed Quicker!

It's the buzz word's going across the internet marketing guru's... Read More

Search Engine Optimization - A Beginners Guide

Getting your site listed in the top search engines, such... Read More

7 Free Search Engine Resources You Should be Using Now

Ask any business person who's website is at the top... Read More

Googles Trap, DMOZs Nap, And Yahoo!s Crap

On November 16th, 2003, Google commenced an update (the Florida... Read More

Meta Tags - What Are They and Which Search Engines Use Them?

Defining Meta Tags is much easier than explaining how they... Read More

Feed me - Satisfy the Search Engines and Your Sites Visitors With Keyword-Rich Content

Search engines love content. Graphics may make your site look... Read More

A Classified Way To Drive Business To Your Web Site

There are more than 105 million of them in the... Read More

New MSN Search Engine: How Good is it?

If you have an online business or you just use... Read More

Shopping Carts and SEO

Shopping and the Web. They go together like Mr. and... Read More

Keep Your Web Site Content Relevant

Visitors and search engines love content-rich web sites, but just... Read More