Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

In The News:

pen paper and inkwell


cat break through


Has Google Indexed Your Site ?

So has Google found your site yet?Over the last 12... Read More

Duplicate the Exact Steps Used to Get a Number 1 Yahoo Ranking in Less than 30 Days

If you have ever been into a McDonalds you will... Read More

Which SEO Company/Firm to Choose for SEO Services?

In the last 2-3 years many new companies have mushroomed,... Read More

What Makes The Perfect SEO Firm?

SEO companies come in all shapes and sizes. You've got... Read More

Beating the New Google AdWord Rules with Blogs and RSS

When Google Adwords first came on scene, it was not... Read More

Building Link Popularity with Topical Articles

One of the important factors in ranking high in search... Read More

Yahoo Dopey, MSN Goofy, Google is Mickey Mouse Lost in a Sandbox

Seventy-two days ago Googlebot first showed up and crawled over... Read More

SEO - Get Your Site Out of the Google Sandbox Fast!

Is your new site sitting in the infamous Google "sandbox"?... Read More

Submitting Your Site To The Open Web Directory: Some Dos And Don?ts

One of the most important steps in any site's publicity... Read More

Does Google Hire Mad Scientists?

Online search giant, Google, often tests the waters for new... Read More

Help Your Visitors Zero in with Site-Flavored Google search

As Google has gained in their search reputation the past... Read More

The Budget Webmaster?s 6 Step Guide to Improving Existing Rankings in Google

The Budget Webmaster's 6 Step Guide to Improving Existing Rankings... Read More

Optimize Your Site Pt2

6: The hidden benefits to having the right links. Every... Read More

Content Management Systems Eyeball SEOs

Content Management Systems and search engine optimization (SEO) used to... Read More

How MSN and Yahoo Sells Your Traffic

Yes, it really happens. Now you might find it hard... Read More

Search Engine Tips & Techniques

As you are building your site or getting your site... Read More

9 Ways to Keep Google Happy

A recent Google patent application has the SEO community buzzing.... Read More

SEO Blues

SEO, not again!, you may groan. The webmaster world is... Read More

SEO and the Outsourcing of Inbound Link Building

Search Engine Optimization nowadays has a lot to do with... Read More

Get a Number One Google Ranking With This Simple Technique

You probably do this already - complete regular searches in... Read More

Can Invisible Text in CSSs Slip Under Search Engine Radar?

I'm literally inundated with questions on the subject of invisible... Read More

How To Get Listed In Google For Free

Google does not accept payment for inclusion in their main... Read More

Soliciting Search Engines

As your guide operator through the web, search engines are... Read More

Analyzing Googles Backlinks Is Close To Worthless

... Read More

What is the Google Dance?

As with any good web developer, the ability to time... Read More

Search Engine Optimization SEO Made Easy

SEO is a never ending battle! So is SEO... Read More

Is Something Missing From Your Keywords Research? (Part 2)

In my previous article, I raised the issue that proper... Read More

Keep Your Content Fresh with this Quick and Easy Tip

Some of the Search engines want only original content. So... Read More

7 Steps to Highly Relevant, Search Engine Friendly, and People Useful Web Site Content

The task is to build a Content Rich Search Friendly... Read More

Search Engine Updates vs. SEO

Webmasters always anxiously wait for a search engine update. Those... Read More

Page Rank - A Quick Overview for Beginners

Page Rank (PR) is a specific value for a website... Read More

Search Engine Optimization: Get the Low-down

Been hearing the words Search Engine Optimization lately? I know... Read More

Top Search Engine Ranks- The Only Secret You Need- Explained: Part 1

The top three search pages- the only place you'll be... Read More