Do Search Engines Like Your Web Site?

Between 75% and 98.8% of visitors to Web sites come from searches made at search engines. If you're going to get high levels of traffic - and hence the levels of ROI you're looking for - it's very important that the search engines can access all the information on your Web site.

Do the search engines know about all of your pages?

You can find out which pages on your site the search engines know about by using a special search. If you search for 'site:' and your Web site address, the search engine will tell you all of the pages on your Web site it knows about.

For example, search for: site:webpositioningcentre.co.uk in Google. Yahoo or MSN Search, and it will tell you how many pages they know about.

If the search engines haven't found some of the pages on your Web site, it is probably because they are having trouble spidering them. ('Spidering' is when the search engine uses an automated robot to read your Web pages.)

Spiders work by starting off on a page which has been linked to by another Web site, or that has been submitted to the search engine. They then read and follow any links they find on the page, gradually working their way through your whole Web site.

At least, that's the theory.

The problem is, it's easy to confuse the spiders - especially as they are designed to be wary of following certain kinds of link.

Links which confuse spiders

If your links are within a large chunk of JavaScript code, the spider may not be able to find them, and will not be able to follow the links to your other pages.

This can happen if you have 'rollovers' as your navigation - for instance, pictures that change colour or appearance when you hover your mouse pointer over them. The JavaScript code that makes this happen can be convoluted enough for the spiders to ignore it rather than try to find links inside.

If you think your rollovers are blocking your site from being spidered, you will need to talk to your Web designers about changing the code in to a 'clean link' - a standard HTML link, with no extra code around it - that is much easier for the spiders to follow.

Links like these will look something like this:

Home Page

Page addresses to avoid

Spiders will also ignore pages if they don't like the URL (the address needed to find the page).

For example, a Web site that has URLs containing several variables can cause spiders to ignore the page content. You can spot pages like these as they have a ? in them, and &, for instance:

http://webpositioningcentre.co.uk/index.php?page=12&cat=23&jib=c

This URL has three variables, the parts with the = in them, between the ? and &s. We find that if a page has one variable, or even two, the top search engines will spider them without any problems. But if a URL has more than that, often the search engines will not spider them.

Spiders particularly avoid URLs that look like they have 'session IDs' in them. They look something like this:

http://webpositioningcentre.co.uk/index.php?page=12&id=29c8d7r2398jk27897a8

The set of numbers and letters do not make much sense to humans, but some Web sites use them to keep track of who you are, as you click through their Web site.

Spiders will generally avoid URLs with Session IDs in them, so if your Web site has them, you need to talk to the people who developed the site about re-writing it so they do not use these IDs, or at least that you can get around the Web site without them.

Clean links = happy spiders

If you use clean, easy to follow links without several variables in them, your Web site should be spidered without problem. There are, of course, many other facets to successful Search Engine Optimization, but if the search engines can't spider your content, your site will fall at the first hurdle.

Paul Silver and David Rosam are Head of Technical SEO and Head of SEO Copywriting at Web Positioning Centre (http://webpositioningcentre.co.uk). Paul has been involved with the Web commercially since 1996 and David has been writing marketing copy for 20 years, and writing for the Web for a decade.

---

Search Engine Traffic Domination - How to Turn the Search Engines into Your 24/7 Salesmen

Getting your website returned by the search engines as a... Read More

Google vs. Yahoo -- How To Rank High On Each One

Google likes incoming links, especially links from high-ranking, on-topic pages... Read More

Martial Arts Webmasters: Time to Optimize Your Site!

A few months ago I was looking through the search... Read More

21 Search Engine Terms Every Web Marketer Should Know Part 1

1. Search Engine - Is a database of web sites... Read More

How To Improve Your Search Engine Ranking

With search engines like Google currently indexing over 8 billion... Read More

What is Google Pagerank?

PageRank is one of the factors that Google uses to... Read More

Google Rankings ? Achieving a Top 10 Position in Google ? Part 1

Achieving a top ranking position in Google is every webmasters... Read More

7 Search Engine Resources You Should Be Using Now

Ask any business person who's website is at the top... Read More

The 7 Best Stealth Techniques to Improve Your Google Rankings

Stealth as in Spying. But Knowing What your Top Competitors... Read More

An SEO Checklist

Would you like a checklist of the important steps to... Read More

What Is Search Engine Marketing?

It's in our genes, we're driven to seek. We 'hunt'... Read More

What Constitutes a Complete and Effective SEO Campaign?

Unfortunately, not many Search Engine Optimization companies know what this... Read More

Developing A List Of Keywords For Marketing

Keywords aren't just some words that allow search engines, like... Read More

Search Engine Visibility - The Mantra of Corporate Profitability

The corporate fundamentals are par excellence! The product is unsurpassable... Read More

Have You Been Dropped By Yahoo?

Yahoo has fairly recently updated its guidelines. You can read... Read More

Six Reasons Why Your Alexa Rating Is Still Important

1. Additional Exposure For Your Site.At the end of 2003,... Read More

Directories and Their Importance for Search Engine Rankings

About directories:A directory is simply a web site that contains... Read More

How To Measure Search Engine Marketing ROI

According to the Search Engine Marketing Professional Organization (SEMPO), advertisers... Read More

Google Sitemaps: 7 Benefits You Cant Ignore

Google Sitemaps enables Webmasters to Directly Alert Google to Changes... Read More

Basic Search Engine Optimization For Dummies

OK, you published your site, now you just sit by... Read More

Using Meta Tags Wisely to Attract Search Engines and Visitors

There are certain tricks and techniques in SEO industry, which... Read More

Search Engine Optimisation ? Why And How To Do It

In the following article I will give you some basic... Read More

What In The World Is This Google Sandbox Theory Thing? And How Do I Beat It?

Ok, so over the past month or so I've been... Read More

LSI and Link Popularity

When Paypal's official Web site no longer ranked #1 in... Read More

Owning Your Category Online - Is Our Website Broken?

"Owning your business category online is more than just a... Read More

SEO #3: Getting Listed In Google in Under 24-Hours!

Yesterday you should have read the second course out of... Read More

Search Engine Rankings for Beginners

Search engine optimization is best left in the mystical land... Read More

Marketing to Search Engines AND Humans

When you were just a young and precocious student of... Read More

STOP Writing for Search Engines

Back when I was starting out with my first internet... Read More

London Bombings and Number Patterns

Firstly, let me say that I am sure all our... Read More

Search Engines The Masters Of The Internet Universe ? Part 3

This is part three of the article series and deals... Read More

Hens Lay Eggs (HLE) by Silke Stahl

"HLE" is a bit of a joke term, (or possible... Read More

Search Engine Optimization Tips For 2005 - Part Two

Welcome to the second part of our series of articles... Read More