Web Crawlers

To me they sound like something from a horror film: 'the dreaded web crawlers!' but despite their 'creepy' name these are what retrieve the information from your website for indexing.

Web Crawlers are often also referred to as an internet bot, web spider, web scutter, ant or an automatic indexer. I'm sure their must be more names but these are the ones I have actually heard people refer to them as. Web crawlers are NOT something to be feared, in fact you want them visiting your site as much as possible!

Depending on what search engine your referring to and or system you use you will usually start using one name and stick to it, I prefer the 'Spider' analogy as the whole 'web and spider' scene works in my mind!

So what does a web crawler do and why? Well a web crawler visits websites and sends back information to the search engine for the purpose of indexing (page ranking) your site. They recover meta data, key words, the last time of update, the frequency of usage etc.

Depending on the search engine it is thought that as the amount of world wide web content that is indexed is somewhere between 40%-70%.........which I find shocking. If you think of some of the past things you have searched for on-line and the search engine has returned hundreds, maybe thousands of results, now think that at least 30% of results aren't showing! This is why it is important to get your website indexed and indexed properly.

With the shear scale of the world wide web it is understandable that not all of it is indexed, and as accessibility to it becomes easier more and more content appears. Search engines must work out what is popular and relevant as well as what isn't and what has stopped being used altogether.

This is why when a company begins SEO (search engine optimisation) work it can take up to 3 months top be web crawled, but once your site has been 'crawled' for the first time the web crawlers will return to see what changes have happened. Once your site is really up and moving it could be crawled as much a once a day but at first it will be more like once a month.

If the results returned by the web crawlers suite the search engines algorithms then you get to watch your site move up the page ranking, which in turn will drive more traffic to your site and so on! If your site isn't returning what the search engine wants then don't expect any positive page ranking movement.

(I have heard that they recover text in a ratio of somewhere between every 3-7 words and that they only really concentrate on the first 100 characters found on your site. This can be very useful information but I would always recommend providing a website full of useful, relevant information and not to try to second guess the web crawlers activities).

Hopefully now you'll have a better understanding of what web crawlers are and how they work. If you have a website that you feel is not getting the noticed then please visit www.abovewebsitedesign.co.uk. We specialise in small businesses and offer affordable Search Engine Optimisation packages that will get your site indexed and moving up the page rankings.