Operator of Claude and of three documented crawlers: ClaudeBot for training data, Claude-User for user-initiated page fetches and Claude-SearchBot for search quality. It publishes robots.txt blocking instructions and crawler IP ranges, making it one of the AI institutions whose crawler policy SEO and content teams must configure for explicitly.
Perplexity
AI answer engine that crawls the web with PerplexityBot and cites sources inline in its responses. It is a central subject of generative engine optimisation work because its visible citation panel makes source selection observable in a way most AI assistants do not.
Why it matters to search
Cited AI answers; PerplexityBot crawler and publisher revenue-sharing programme.
Perplexity is a search engine based in United States.
Related organizations
6 other search engines, plus others based in United States.
Baidu's webmaster platform (百度搜索资源平台), providing submission APIs, indexing diagnostics and Chinese-language crawling guidance. It is the required entry point for any site attempting visibility in mainland China's largest search engine, where Google is unavailable.
Search engine built on a fully independent index of billions of pages rather than syndicated results, operated by Brave Software. Its independence makes it one of the few alternative indexes SEOs can test against, and it also sells index access via an API used by AI products.
Vietnamese browser and search engine built specifically for Vietnamese users, with local-language handling and a search API for developers. It holds a meaningful share of Vietnamese search and browser usage, so sites targeting Vietnam optimise for it alongside Google.
Privacy-focused search engine that combines its own crawling with results licensed from Bing, and publishes its DuckDuckBot crawler documentation. Its refusal to build user profiles removes personalisation from its results, which makes it a useful neutral reference SERP for rank checking.
Not-for-profit search engine founded in Berlin in December 2009 by Christian Kroll, dedicating all profits to tree planting and operating as a steward-owned certified B Corporation. It represents a meaningful share of privacy- and values-motivated European search traffic and, with Qwant, co-founded a European index initiative.