CatchBot is the web crawler for Catch, the online division of Reed Business Information Australia. Reed Business Information is Australia’s leading and largest business to business publisher and information provider.
CatchBot investigates websites for publicly available information about companies, such as a company’s name, address, telephone number and keyword data about a company’s products and services. CatchBot is not designed to access or index any personal information or any information about individuals.
Information gathered by CatchBot is stored on our password protected servers and the security of this information is of the highest importance to us.
Information gathered by CatchBot may be used for business activities that are undertaken by Catch. Examples of this include publishing and maintaining business directories in various countries around the world, industry specific websites and online portals
Is CatchBot a well behaved crawler?
CatchBot obeys the standard web protocols, including robots.txt. If you want to find out more about robots.txt please visit www.robotstxt.org.
How many pages from my site will CatchBot typically crawl?
Normally CatchBot will only access a few pages of any website. It does not attempt to crawl the entire web or every page of your website.
Will my servers or site be affected by CatchBot?
It is unlikely that your servers or site will be affected by CatchBot. As Catchbot typically only crawls a few pages of any website slowly, CatchBot has no material negative impact on the load on your servers or the availability of your website. In fact, unless a webmaster investigates its server logs, most webmasters may not be aware that CatchBot has accessed their site at all.
How do I identify CatchBot in my server logs?
CatchBot identifies itself using the user agent of ‘CatchBot/1.0; +http://www.catchbot.com’ and if you investigate your server logs you will be able to see if it has visited your site or not.
Please note: it is possible for malicious bots, or bots that are not as well behaved as CatchBot, to spoof their user agent and claim to be CatchBot when they are not. Catch cannot take any responsibility for the actions of bots that are impersonating CatchBot and we actively discourage this behaviour.
How do I prevent CatchBot from accessing my site or following links on my pages?
If you do not want CatchBot to access pages from your site include the following two lines in your robots.txt file:
If you do not want CatchBot to index a specific page of your site place the following meta tag into the head of your html page:
<META NAME="CatchBot" CONTENT="NOINDEX">
If you do not want CatchBot to follow any links on a specific page of your site place the following meta tag info the head of your html page:
<META NAME="CatchBot" CONTENT="NOFOLLOW">
How do I contact Catch to enquire about CatchBot?
Please email firstname.lastname@example.org.