From Corpora to Matching

Making effective use of the Internet is increasingly about creating better and more intelligent applications and search engines. Here is a brief introduction into how search engines work:

01) Define the corpus, search space/data;
02) Separate the corpus into documents;
03) Generate features for each document;
04) Generate a representation of each document;
05) Study the feature/vector space;
06) Cluster documents;
07) Reduce dimensionality;
08) Accept input Queries;
09) Find the cosine angles against the query vector;
10) Find the sought vector column;
11) Output results to user in some way;

Each document in a corpus (database) is described by a set of keywords called index terms. We assign weights to index terms according to their relevance (frequency of occurrence for instance), this is how we go about creating the index, that we can then search.

Corpus preparation:
Web pages of interest are analysed and cleaned by removing hypertext tags or any other hyper language; Pages are then broken down into documents where each document is scanned through searching for words/terms of interest: those which make a document unique, not standard words.

Extract terms of interest:
Bear in mind that terms of interest must be invariant, that is be characteristic of a document, not generic and easy to find in any corpus/document. The idea is to find a signature per document.

Build term-by-document matrix:
The search space is defined by N dimensions where the chosen terms/features of a document is a point in the N term space, this allows conceptual/semantic searches.

Each document becomes a column vector, each row represents a term. Each row identifies the frequency of a term across the analysed corpus, at first we simply build the matrix by counting the terms for each document.

Compress the matrix:
There are two basic techniques/methods, Compress Row Storage (Scans matrix row by row) and Compress Column Storage (Scans matrix column by column) Both use three arrays.

Normalis the matrix:
Normalisation implies transforming column vectors to unit vectors: i.e. vectors of unit length

Unit document vectors contain frequency of terms; the normalisation is applied because the semantic content of a document is generally determined the relative frequency of terms.

Singular Value Decomposition:
This simplifies a symmetric matrix into three matrices Two are identical and represent the eigenvectors: the new dimensions. The third is diagonal and represents the eigenvalues, that is the spread of the corpus along these new dimensions.

A geometric interpretation:
The corpus is first formated, stemmed and is then stored in a compact term-by-document matrix. Each column of such matrix is then normalised to produce the likelihood of a term across the corpus, or, equivalently, the frequency of terms in a document.

The term-by-document matrix is then decomposed to calculate eigen values and vectors. Eigen vectors represent a new Cartesian coordinate frame spanning the same search space, BUT, they indicate the most important dimenions/axis along which documents mainly lie. Eigen value do quantify the spread of documents along these new axes/eigen vectors.

Queries:
Queries must be based on defined features/terms within the term-by-document matrix, matching in a vector space such as this is implemented by multiplying the query vector against the terms by document matrix,ie matching a query vector q against the documents of the matrix.

© I am the website administrator of the Wandle industrial museum (http://www.wandle.org). Established in 1983 by local people determined to ensure that the history of the valley was no longer neglected but enhanced awareness its heritage for the use and benefits of the community.

In The News:

pen paper and inkwell


cat break through


DIY SEO - Part 2 Tags n Things

Part 2 Tips 4 tagsMeta tags are the descriptive tags... Read More

Boost Your Website Rankings with Expert Content

The chase for a high web ranking is constantly on... Read More

Search Engine Essentials

What are Search Engines?There are numerous different search engines, and... Read More

A Simple 4 Step Approach To Have Google Crawl Your Site Today Guaranteed!

Extra, Extra Google Now Gives you feedback. By Creating a... Read More

Getting Specific With Local Search Engine Optimization

Localized search engine optimization is often overlooked as an excellent... Read More

Guide to Getting Linked

Getting links from other websites pointing to your site can... Read More

Google Rankings -- Achieve Top 10 Rankings with Free Tools

In order to get a top 10 google ranking it... Read More

7 Search Engine Optimization Mistakes and Solutions

To many websites, webmasters discover that major sources of website... Read More

Attack Of The Killer Google Zombies!

Don't share this with people of a nervous disposition because... Read More

Boost Your Search Engine Ranking And Generate Free Traffic With Reciprocal Links

Reciprocal links are an important step in your overall plan... Read More

Is Google Having a Tough Time with Their Website Limit?

If you are one to pay attention to what happens... Read More

No Cost Search Engine Marketing

As a matter of fact, I recommend NOT wasting money... Read More

Complete Web-Site Optimization For Search Engines (Part 1)

SEO or search engine optimization strategy now becomes widely popular... Read More

Secrets on Website Promotion: How You Can Get a #1 Ranking for Your Website Name Within 30 Days

Launching a new website with enough acceleration to rise above... Read More

The Need of Popularity

In very simple words, the link popularity of your site... Read More

New MSN Search Engine: How Good is it?

If you have an online business or you just use... Read More

Beyond Hits!

Your Website visitor reports are a goldmine of information. If... Read More

Agressive Linking Strategies Help To Get Your Site Noticed

A good link building strategy has become an essential part... Read More

The Sandbox Effect

What once many people thought they had a penalty, is... Read More

The Life Blood of Internet-Based Home Businesses...Search Engines

Anyone involved in an Internet-based home business will soon come... Read More

Search Warrant

So, you've just paid £300 - £10,000 for a new... Read More

SEO Expert Guide - Black Hat SEO - Activities to avoid (part 8/10)

In parts 1 - 7, you learnt how to develop... Read More

How To Design A Search Engine Friendly Website

There are many websites that fail to target their required... Read More

Why Pay-Per-Inclusion Search Engines are Dying

A Pay-Per-Inclusion search engine is a service in which a... Read More

SEO Hints and Tips and Free SEO Tools

Do you realize that if you manage your website, SEO... Read More

How to Get Non-stop Free Traffic to Your Website

Yet the simple truth is that without traffic a website... Read More

Stay In The Know With Google SMS

The Short Messaging Service (SMS) from Google sends short, quick,... Read More

The Importance of Search Engine Optimization

Search Engine Optimization is a key to any successful internet... Read More

Introduction to Google Page Rank (PR)

For anyone looking to enhance their Google Page Rank (PR)... Read More

Search Engines Are Doomed to Fail

A very provocative title, isn't it? Based on the latest... Read More

How To Start An Internet Business ? Meta Tags and Keyword Density

Okay, you have a domain name, layout and content. Now... Read More

An Introduction to Google Sitemaps

... and why I 'm dying to get finally in... Read More

Ten Critical Website and Search Engine Promotion Mistakes

When it comes to promoting your website, you might think... Read More