research crawler · CatDBBot/1.0
CatDBBot: research crawler contact
CatDBBot is an automated research crawler operated by Subvert Labs Limited (company no. 9461332, NZBN 9429053983107), Aotearoa New Zealand.
What it crawls, and why
CatDBBot fetches public New Zealand web pages for website classification research: working out what kind of organisation or service a site represents, so that aggregate, non-personal statistics about the NZ web can be produced. It is not a search engine. It does not build a public index and does not republish page content.
User-Agent
Requests are sent with this User-Agent string:
CatDBBot/1.0 (+https://research.subvertlabs.nz/)Crawl policy
- robots.txt is honoured. Disallowed paths are not fetched, and Crawl-delay directives are respected.
- Rate-limited. Requests are spaced out per host, well below anything that would load a site's servers.
- Targeted. Only pages the classifier needs are fetched. No link-following sprawl.
- No login, paywall or form access. Forms are not submitted and authentication is not attempted.
- Stop on request. If you ask us to stop crawling your site, we stop and remove that site's data from the research set.
Contact
For questions, crawl complaints, opt-outs or data removal:
Include the site's domain name. Crawling of that site is paused as soon as the request is received.