Automated search engine robots, sometimes called "spiders" or "crawlers", are the seekers of web pages. How do they work? What is it they really do? Why are they important?
You'd think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can't do certain things. Robots don't understand frames, Flash movies, images or JavaScript. They can't enter password protected areas and they can't click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation. How Do Search Engine Robots Work?
Think of search engine robots as automated data retrieval programs, traveling the web to find information and links.
When you submit a web page to a search engine at the "Submit a URL" page, the new URL is added to the robot's queue of websites to visit on its next foray out onto the web. Even if you don't directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of the reasons why it is important to build your link popularity and to get links from other topical sites back to yours.
When arriving at your website, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files the robot doesn't need to concern itself with.
Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow the links from one page to another. The entire World Wide Web is made up of links, the original idea being that you could follow links from one place to another. This is how robots get around.
The "smarts" about indexing pages online comes from the search engine engineers, who devise the methods used to evaluate the information the search engine robots retrieve. When introduced into the search engine database, the information is available for searchers querying the search engine. When a search engine user enters their query into the search engine, there are a number of quick calculations done to make sure that the search engine presents just the right set of results to give their visitor the most relevant response to their query.
You can see which pages on your site the search engine robots have visited by looking at your server logs or the results from your log statistics program. Identifying the robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google's "Googlebot"; others are bit more obscure, like Inktomi's "Slurp". Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.
Along with identifying individual robots and counting the number of their visits, the statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In the resources section of the end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them. How Do They Read The Pages On Your Website?
When the search engine robot visits your page, it looks at the visible text on the page, the content of the various tags in your page's source code (title tag, meta tags, etc.), and the hyperlinks on your page. From the words and the links that the robot finds, the search engine decides what your page is about. There are many factors used to figure out what "matters" and each search engine has its own algorithm in order to evaluate and process the information. Depending on how the robot is set up through the search engine, the information is indexed and then delivered to the search engine's database.
The information delivered to the databases then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page.
The search engine databases update at varying times. Once you are in the search engine databases, the robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have the latest info. The number of times you are visited depends on how the search engine sets up its visits, which can vary per search engine.
Sometimes visiting robots are unable to access the website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, the robot may not be able to access your site. When this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.
Resources
*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html
*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/
*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/
*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/
*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.
Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.
Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.
|
|
|
|
|
|
|
|
|
|
|


As the undisputable leader in search engines, Google places a... Read More
If you have a Web site, have you ever wondered... Read More
Many of our customers (http://www.internet-marketing-australia.com) find search engine... Read More
So you built a website, you have a product, but... Read More
If they want to have success, companies should do everything... Read More
Being listed in search engines and ranked high on searches... Read More
Search engine optimization experts are having fun with Google. Experts,... Read More
It seems I now do this rant every single year... Read More
The META tags are used to provide extra information about... Read More
Getting your website returned by the search engines as a... Read More
Uniquely built web sites can create unique issues when being... Read More
Every webmaster knows that the free search engines are a... Read More
If your site isn't found in the search engines, it... Read More
The attraction of Pay Per Click (PPC) online advertising is... Read More
"Dear David: I just created a website on baby toy... Read More
Unfortunately, we don't live in a perfect world. You may... Read More
Among the many things you need to worry about for... Read More
Web users turn to search engines for answers to their... Read More
Introduction The Google Sandbox is a metaphorical term... Read More
A higher search ranking is what many website owners dream... Read More
Meta tags are an absolute must from a search engine... Read More
I was recently contacted by one of my best clients... Read More
Launching a new website with enough acceleration to rise above... Read More
Does Creating a Google Site Map and Telling Google About... Read More
This is the first of a series of articles about... Read More
Long before the days of researching phrases with the helpful... Read More
The Internet may have started as the fervent brainchild of... Read More
With tons of competition and copycats online, you need a... Read More
The first task most netizens do when they log on... Read More
Listen. Some make submitting pages to search engines sound like... Read More
If you are a webmaster and you review your logs,... Read More
If you are a webmaster, then you've probably submitted your... Read More
This article will cause many companies to stir, but it's... Read More


Firstly, let me say that I am sure all our... Read More
You've seen the ads: Guaranteed #1 Ranking! There are no... Read More
We know that each and every website page is assigned... Read More
Ever felt intimidated at the convoluted, jargon-ridden information about Internet... Read More
When shopping online for Link Management Software that can... Read More
There are many techniques that SEM/SEO experts use to optimize... Read More
Google is the undisputed heavyweight champion of search engines. Most... Read More
Am I alone as I survey that vast wilderness outside,Sat... Read More
With search engines like Google currently indexing over 8 billion... Read More
A search engine optimization maintenance service plan will ensure that... Read More
IntroductionThe Google Sandbox is a term applied to the phenomenon... Read More
The robots.txt file is an exclusion standard required by all... Read More
There are millions of web sites trying to get listed... Read More
Google has many ways to help you find want you... Read More
Search engines are constantly tweaking their ranking algorithms and when... Read More
Internet Directories and their ImportanceThere are two very pertinent reason... Read More
I admit it, our SEO services are pretty bad.Speaking of... Read More
If you want to increase traffic to your website and... Read More
It's a marketing dream come true: A potential customer, looking... Read More
Visit Google, Yahoo, MSN or one of the lesser search... Read More
In this third article, we continue to dig into the... Read More
Lets face it. Search Engines rule the World Wide Web... Read More
The task is to build a Content Rich Search Friendly... Read More
Being listed in search engines and ranked high on searches... Read More
Creating a well-designed website is the first step in your... Read More
I knew things were bad at DMOZ. But I guess... Read More
Every once in a while--and probably more often than we... Read More
If you would like to know when your site was... Read More
A Brief Introduction To Link PopularityAs webmasters research various ways... Read More
"Owning your business category online is more than just a... Read More
Before using keywords for your site, it is always better... Read More
Among the many things you need to worry about for... Read More
The first step to developing any search engine optimization effort... Read More
Search Engine Optimization (SEO) |