Index
Intro
Services
Pricing
Order
About Us
Privacy Policy
Contact
[01.09] 2002-02-11
All search engine spiders (crawlers, searchbots) have to be recognized by our on site software for this strategy to work.
To this purpose we have created the world's largest and most comprehensive search engine botBase, our fantomas spiderSpy™ service, currently covering 8.000+ entries including IP addresses, URLs and UserAgent variables as maintained by the search engines. Hundreds of entries are being added weekly: our network of “spider trap” domains dedicated to trap spiders' data and integrating it into our recognition database before they hit a client's sites; massive third party support from all over the world; extensive evaluation and verification of all available spider data resources, etc. have helped to make this into the most ambitious and effective search engine spider reference currently available.
The fantomas spiderSpy™ service makes a special point of monitoring international search engines on a global scale.
This is of great importance because global networks being what they are, many major search engines have implemented time and bandwidth sharing plus load balancing in assigning crawling tasks to their vast range of spiders.
Hence, it is not advisable to restrict such a project (as all of our competitors seem to be doing) to “the major search engines generating 90% of the traffic anyway”. While it is true that a mere handful of major engines accounts for the lion share of overall search traffic, spidering and indexing activity (which is, after all, what search results are based on) is a totally different story.
Two examples to illustrate why it is of utmost importance to monitor spider activity on an international scale: A web site submitted to a US based major engine like Go (formerly Infoseek) may well be crawled by their Japanese spider. If you submit your site to AltaVista (Netherlands), the spider crawling your site will come from AltaVista (UK).
Hence, only a comprehensive and global database will ensure that these search engine spiders are actually recognized for what they are in order to feed them with appropriate content for optimized rankings.
Index | Intro | SE Marketing | FAQs | Services | TechSpecs | Pricing | Order | About Us | Privacy Policy | Contact
