If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?
Not quite sure what I am talking about? Here is a few examples of various bots searching my website:
207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.
207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.
66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.
What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.
What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.
Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.
How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.
How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.
What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.
Ken Dennis
http://KenDennis-RSS.homeip.net/
Welcome to part three of our series of articles on... Read More
This article will cause many companies to stir, but it's... Read More
There's an assumption that the higher a ranking or positioning... Read More
Sitemaps are without doubt one of the most often ignored... Read More
Utilizing effective search engine optimization techniques will improve the page... Read More
Since search engines are the first stop for people on... Read More
SEO consultants will tell you that you need to be... Read More
When people think of search engine optimization, they immediately think... Read More
Link building is a waiting game. Many clients have asked... Read More
Most Internet marketing methods are risky and many will not... Read More
One of the key things to remember when developing your... Read More
Have you heard of the SETI Project? SETI stands for... Read More
Targeted traffic is the lifeblood of any online business. The... Read More
When most people want to find something on the web,... Read More
Webmasters can spend most of their waking hours doing everything... Read More
The first months my website was online, I was constantly... Read More
Google uses PageRank to rank your pages. To maximize your... Read More
There is a lag time between the indexing or updating... Read More
I seem to have created quite a stir, on a... Read More
So you have built a nice web site with good... Read More
Google, the most popular, and many say best, search engine,... Read More
Thanks to a unique algorithm that produces most relevant results... Read More
If you've been involved in SEO (search engine optimization) for... Read More
How many years did you register your domain name for?... Read More
In this third article, we continue to dig into the... Read More
Google Labs is currently testing Google Print, which returns results... Read More
A higher search ranking is what many website owners dream... Read More
There are many websites that fail to target their required... Read More
Search Engine traffic accounts for nearly 80% of the Internet... Read More
Back in November, when the Google Dance began, Barry Lloyd... Read More
Google has many ways to help you find want you... Read More
Search engine traffic should be a priority for any online... Read More
In recent times, I have been closely studying keywords that... Read More
Google is currently one of the dominant entities on the... Read More
It's no secret that Search Engine marketing can drive significant... Read More
Don't put the cart before the horse.You can't do SEO... Read More
Getting links from other websites pointing to your site can... Read More
Do you ever wonder how people search for things on... Read More
Search engine optimization (SEO) is very important to websites. If... Read More
I just read an article at SitePro News that really... Read More
What we are hunting for?Unique Domains,One Way Links,Text Link copy... Read More
Just for a change, rather than a technical article, I... Read More
Anyone would agree that it is much easier to be... Read More
Search Engine Optimization (SEO) is a very complex process. It... Read More
Search Engine Listing And OptimizationIn the world of website marketing,... Read More
Why, you ask? Mainly, because search engines want to provide... Read More
Google now checks the year your domain name was first... Read More
Okay I admit it. I was kidding. I can't guarantee... Read More
Welcome to part five in this search engine positioning series.... Read More
One of the most frequent questions I get asked is... Read More
On the internet, competition is stronger than ever. There was... Read More
Over the last few months, search engine submissions have changed... Read More
Lets face it. Search Engines rule the World Wide Web... Read More
Want to drive more targeted traffic to your web site?Want... Read More
Among the many things you need to worry about for... Read More
IntroductionThe Google Sandbox is a term applied to the phenomenon... Read More
You've got a website. You've put countless hours into it,... Read More
Use text links, avoid image links. Anyhow, if you... Read More
When you were just a young and precocious student of... Read More
Get Indexed FastWhat does getting indexed mean?The search engines keep... Read More
Achieving a top ranking position in Google is every webmasters... Read More
Search Engine Optimization is emerging as the most powerful form... Read More
Search engine optimization is one of most popular online marketing... Read More
With the ever evolving internet market for just about anything... Read More
I set out with the intention of writing a self... Read More
With all the recent publicity given to Google as the... Read More
Search Engine Optimization (SEO) |