Making effective use of the Internet is increasingly about creating better and more intelligent applications and search engines. Here is a brief introduction into how search engines work:
01) Define the corpus, search space/data;
02) Separate the corpus into documents;
03) Generate features for each document;
04) Generate a representation of each document;
05) Study the feature/vector space;
06) Cluster documents;
07) Reduce dimensionality;
08) Accept input Queries;
09) Find the cosine angles against the query vector;
10) Find the sought vector column;
11) Output results to user in some way;
Each document in a corpus (database) is described by a set of keywords called index terms. We assign weights to index terms according to their relevance (frequency of occurrence for instance), this is how we go about creating the index, that we can then search.
Corpus preparation:
Web pages of interest are analysed and cleaned by removing hypertext tags or any other hyper language; Pages are then broken down into documents where each document is scanned through searching for words/terms of interest: those which make a document unique, not standard words.
Extract terms of interest:
Bear in mind that terms of interest must be invariant, that is be characteristic of a document, not generic and easy to find in any corpus/document. The idea is to find a signature per document.
Build term-by-document matrix:
The search space is defined by N dimensions where the chosen terms/features of a document is a point in the N term space, this allows conceptual/semantic searches.
Each document becomes a column vector, each row represents a term. Each row identifies the frequency of a term across the analysed corpus, at first we simply build the matrix by counting the terms for each document.
Compress the matrix:
There are two basic techniques/methods, Compress Row Storage (Scans matrix row by row) and Compress Column Storage (Scans matrix column by column) Both use three arrays.
Normalis the matrix:
Normalisation implies transforming column vectors to unit vectors: i.e. vectors of unit length
Unit document vectors contain frequency of terms; the normalisation is applied because the semantic content of a document is generally determined the relative frequency of terms.
Singular Value Decomposition:
This simplifies a symmetric matrix into three matrices Two are identical and represent the eigenvectors: the new dimensions. The third is diagonal and represents the eigenvalues, that is the spread of the corpus along these new dimensions.
A geometric interpretation:
The corpus is first formated, stemmed and is then stored in a compact term-by-document matrix. Each column of such matrix is then normalised to produce the likelihood of a term across the corpus, or, equivalently, the frequency of terms in a document.
The term-by-document matrix is then decomposed to calculate eigen values and vectors. Eigen vectors represent a new Cartesian coordinate frame spanning the same search space, BUT, they indicate the most important dimenions/axis along which documents mainly lie. Eigen value do quantify the spread of documents along these new axes/eigen vectors.
Queries:
Queries must be based on defined features/terms within the term-by-document matrix, matching in a vector space such as this is implemented by multiplying the query vector against the terms by document matrix,ie matching a query vector q against the documents of the matrix.
© I am the website administrator of the Wandle industrial museum (http://www.wandle.org). Established in 1983 by local people determined to ensure that the history of the valley was no longer neglected but enhanced awareness its heritage for the use and benefits of the community.
Google's calling your name...Hi, Google here. We want to index... Read More
You've got a cool new website with all the works:... Read More
Time is a factorTo obtain positive results is not very... Read More
Keyword density. When it comes to SEO copywriting, this has... Read More
In the process of designing a website for clients the... Read More
The biggest problem that most are running into seems to... Read More
In this three part article, you'll find many tools that... Read More
As the undisputable leader in search engines, Google places a... Read More
If you are one to pay attention to what happens... Read More
Google has always been the search industry's innovator and that's... Read More
Achieving a top ranking position in Google is every webmasters... Read More
If you have any interest in getting high search engine... Read More
What is the Google Algorithm?An algorithm is a mathematical equation... Read More
OK. So you've created a nice website with lots of... Read More
Everyone is looking for "secrets" about how to get more... Read More
If you have questions about whether or not the Over-Optimization... Read More
Over the past few years, there have been many debates... Read More
I set out with the intention of writing a self... Read More
Have you ever seen any email offers of getting you... Read More
It's interesting how potential clients have preconceived notions about which... Read More
Everyone wants to increase their rankings with the search engines... Read More
Many search engine optimization companies will sell you a search... Read More
The chase for a high web ranking is constantly on... Read More
First of all, the Google directory is really just the... Read More
Welcome to part three of our series of articles on... Read More
You've selected an appropriate Online Business Opportunity. That is not... Read More
Watching a recent football game, I imagined two very different... Read More
Keyword Research will reveal answers to 3 critical questions:1. Is... Read More
If you want to develop a successful search engine optimization... Read More
Trying to improve search engine rankings is just like a... Read More
Back when I was starting out with my first internet... Read More
If you're looking for an SEO firm, we recommend that... Read More
My Grandfather ran a small Grocery Store and when you... Read More
If you are confused about terms like "search engine optimization"... Read More
Every webmaster knows that the free search engines are a... Read More
PageRank is one of the factors that Google uses to... Read More
As keyword marketing becomes more and more expensive and competitive,... Read More
"The use of black hat SEO techniques are completely unethical."... Read More
Google uses an algorithm to determine the search engine results... Read More
There are a lot of operators (commands you can enter... Read More
There's no denying it: Search engines are a dominating force... Read More
Search Engine Optimization (SEO) is something you should be aware... Read More
Like all things, theft is theft, misery loves company, negative... Read More
The point of optimizing your website is so that you... Read More
Ok, here's the deal, follow these steps and shoot me... Read More
For so many web surfers, it's almost automatic to type... Read More
It's taken you 6-months of hard work, constant changes, reading... Read More
Open Wordtracker [ http://www.wordtracker.com/ ] and you'll see... Read More
Lets get things straight. SEO is a very competitive market.... Read More
The days of basing a successful link building strategy on... Read More
Search engines determine their rankings based on two things: ... Read More
At the beginning of the web era, users would go... Read More
Some webmasters report that search engines account for 75% or... Read More
In parts 1 - 4 you learnt how to develop... Read More
Search Engine OptimizationSearch engines still remain the #1 tool to... Read More
It's in our genes, we're driven to seek. We 'hunt'... Read More
Your Website visitor reports are a goldmine of information. If... Read More
As the economy begins to recover in certain parts of... Read More
Strategic search engine optimization involves far more than keyword research,... Read More
Search engine optimization (SEO) is a long and complicated process... Read More
With the help of MSN, recently I've been reminiscing about... Read More
Getting indexed in Yahoo has become very difficult in the... Read More
Keyword Research is the first task in optimizing your web... Read More
Once you have your web site complete and ready to... Read More
With search engines like Google currently indexing over 8 billion... Read More
It doesn't matter what kind of marketing you are involved... Read More
Search Engine Optimization (SEO) |