If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.
Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.
If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.
Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?
If you know what to "feed" the spidering robots you will help yourself with search engine ranking.
Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.
Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.
A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.
Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility
To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi
You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line
When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.
Resources
*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html
*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/
*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/
*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/
*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.
Copyright © 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.
Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.
Uniquely built web sites can create unique issues when being... Read More
Let me preface this report by citing advertisers in 2004... Read More
Becoming an SEO expert, or a search engine optimization expert,... Read More
Google's calling your name...Hi, Google here. We want to index... Read More
IntroductionWhy does a FLASH intro need SEO? Flash, the great... Read More
In parts 1 - 5 you learnt how to develop... Read More
When shopping online for Link Management Software that can... Read More
Keywords have turned into big business on the internet. If... Read More
Is your new site sitting in the infamous Google "sandbox"?... Read More
Keyword optimisation is probably the most important thing that you... Read More
Stealth as in Spying. But Knowing What your Top Competitors... Read More
Many internet marketers blow mountains of start-up cash on their... Read More
There are methods to increase your search engine rankings which... Read More
Unfortunately, we don't live in a perfect world. You may... Read More
As the economy begins to recover in certain parts of... Read More
The chase for a high web ranking is constantly on... Read More
There are many Black Hat techniques that people use to... Read More
Everyone wants to increase their rankings with the search engines... Read More
As you have seen throughout the guide, search engine optimization... Read More
1. Google love ...Google and all its programming is not... Read More
Previously...In our article on Understanding Google's Algorithm,... Read More
In order to get a top 10 google ranking it... Read More
Search Engine Optimization is a widely misunderstood industry. Many webmasters,... Read More
Why, you ask? Mainly, because search engines want to provide... Read More
Did you know that you can dramatically increase the number... Read More
Sitemaps are without doubt one of the most often ignored... Read More
Being listed in search engines and ranked high on searches... Read More
With the help of MSN, recently I've been reminiscing about... Read More
Yahoo! has a developed a new concept in search engine... Read More
For the uninitiated, searching for web pages can seem a... Read More
It crosses every webmaster's mind anytime they see an ad... Read More
When Paypal's official Web site no longer ranked #1 in... Read More
As you may already know, keywords are an essential part... Read More
Get Indexed FastWhat does getting indexed mean?The search engines keep... Read More
MSN finally unleashed its new search technology to the world... Read More
SEO's Relationship With Website ArchitectureSearch engine optimization for today's search... Read More
Search engines love content. Graphics may make your site look... Read More
You have heard the phrase LOCATION LOCATION LOCATION. But wait,... Read More
Search engine optimization remains a minefield of old advice, outdated... Read More
When it comes to search engine optimization there is a... Read More
You've got a cool new website with all the works:... Read More
Once the keywords have been decided for the site one... Read More
Ever see a website that seems to speak a foreign... Read More
Want to drive more targeted traffic to your web site?Want... Read More
It used to be that designing an attractive website to... Read More
One of the most overlooked strategies to improve search engine... Read More
Getting top search engine listing is perhaps one of the... Read More
Smartpages are highly optimized pages which draw stampeded of search... Read More
In the following article I will give you some basic... Read More
Making effective use of the Internet is increasingly about creating... Read More
Where is the need to worry? Who needs a PR... Read More
Don't put the cart before the horse.You can't do SEO... Read More
Search Engine OptimizationSearch engines still remain the #1 tool to... Read More
Web site owners need to have realistic expectations when it... Read More
... and why I 'm dying to get finally in... Read More
Online search giant, Google, often tests the waters for new... Read More
Running an online business relies to a greater or lesser... Read More
All the great efforts made for making your own site,... Read More
Some of the Search engines want only original content. So... Read More
Part 1. Wordtracker for keywords.A problem for all new webmasters... Read More
Search Engine Optimization is emerging as the most powerful form... Read More
Yesterday you should have read the forth course out of... Read More
You finally have a website and you are ready to... Read More
Getting listed in Yahoo! (The Very Top Search Engine/Index) is... Read More
Keywords are terms or words related to particular topic. Keyword... Read More
If you're a webmaster, you've probably spent almost as much... Read More
Search Engine Optimization (SEO) |