Web Crawlers

To me they sound like something from a horror film: 'the dreaded web crawlers!' but despite their 'creepy' name these are what retrieve the information from your website for indexing.

Web Crawlers are often also referred to as an internet bot, web spider, web scutter, ant or an automatic indexer. I'm sure their must be more names but these are the ones I have actually heard people refer to them as. Web crawlers are NOT something to be feared, in fact you want them visiting your site as much as possible!

Depending on what search engine your referring to and or system you use you will usually start using one name and stick to it, I prefer the 'Spider' analogy as the whole 'web and spider' scene works in my mind!

So what does a web crawler do and why? Well a web crawler visits websites and sends back information to the search engine for the purpose of indexing (page ranking) your site. They recover meta data, key words, the last time of update, the frequency of usage etc.

Depending on the search engine it is thought that as the amount of world wide web content that is indexed is somewhere between 40%-70%.........which I find shocking. If you think of some of the past things you have searched for on-line and the search engine has returned hundreds, maybe thousands of results, now think that at least 30% of results aren't showing! This is why it is important to get your website indexed and indexed properly.

With the shear scale of the world wide web it is understandable that not all of it is indexed, and as accessibility to it becomes easier more and more content appears. Search engines must work out what is popular and relevant as well as what isn't and what has stopped being used altogether.

This is why when a company begins SEO (search engine optimisation) work it can take up to 3 months top be web crawled, but once your site has been 'crawled' for the first time the web crawlers will return to see what changes have happened. Once your site is really up and moving it could be crawled as much a once a day but at first it will be more like once a month.

If the results returned by the web crawlers suite the search engines algorithms then you get to watch your site move up the page ranking, which in turn will drive more traffic to your site and so on! If your site isn't returning what the search engine wants then don't expect any positive page ranking movement.

(I have heard that they recover text in a ratio of somewhere between every 3-7 words and that they only really concentrate on the first 100 characters found on your site. This can be very useful information but I would always recommend providing a website full of useful, relevant information and not to try to second guess the web crawlers activities).

Hopefully now you'll have a better understanding of what web crawlers are and how they work. If you have a website that you feel is not getting the noticed then please visit www.abovewebsitedesign.co.uk. We specialise in small businesses and offer affordable Search Engine Optimisation packages that will get your site indexed and moving up the page rankings.

Website Indexing?

Also referred to as web indexing or internet indexing is quite a complex topic. However I feel it is one of the easier on-line topics to explain on a basic level as many of the terms used conjure up mental images that help me remember how it all works.

As website indexing implies it is about getting a website indexed or listed on search engines and the results they provide.

The usual goal is to try to get a website indexed at the first position available when the search engine returns the results back to you after search for whatever you have typed in search box. Quite often this doesn't mean that you will be listed on the top of the page as many search engines will have paid for advertised links at the top of the page, however it does usually mean that you will be found on the first page (directly under the paid results).

So how does the search engine decide which page gets listed where? Well there is no straight forward answer to this as search engines have unique 'algorithms' that set the criteria as to what makes a good website.

When you try to get your website on top of the page listings or index it requires a bit of trial and error, and not everything that works for one website will work for the next, which can be frustrating and opened the debate of whether search engine optimisation of a website is an art or a science? (but that's for another blog!)

There are however certain things we know about how a website gets 'indexed' and if you have a website these are the least things that you need to do to ensure your site gets indexed and moves up the 'listings'

Here's what we know for sure:

Search engines like Google, Bing, Internet Explorer, Zafari, and so on....Use something called 'Web Crawlers' to gather information relating to a website. These web crawlers may gather information from many areas of your website but tend to use your websites 'keywords' and 'meta data' to add to the search engines search criteria (or to match to what someone searches for).

So it is important that your website has the correct terms and keywords in use enable the search engines to list or index your site where it should be when it is searched for.

Although search engines take into account many other factors including site age, site popularity, up to date coding, site usage (content management), and so on, to insure your site gets positioned on the index or page rankings correctly, the meta data and keywords are something you should like to get right from the very beginning.

(If your getting lost with the terms already don't worry because we will explain and link from them in future posts.)

If you require a website why not visit our commercial website Above Website Design. We are specialists in getting small businesses on the World Wide Web.