Search Engine

Operator of Claude and of three documented crawlers: ClaudeBot for training data, Claude-User for user-initiated page fetches and Claude-SearchBot for search quality. It publishes robots.txt blocking instructions and crawler IP ranges, making it one of the AI institutions whose crawler policy SEO and content teams must configure for explicitly.

Baidu's webmaster platform (百度搜索资源平台), providing submission APIs, indexing diagnostics and Chinese-language crawling guidance. It is the required entry point for any site attempting visibility in mainland China's largest search engine, where Google is unavailable.

Search engine built on a fully independent index of billions of pages rather than syndicated results, operated by Brave Software. Its independence makes it one of the few alternative indexes SEOs can test against, and it also sells index access via an API used by AI products.

Vietnamese browser and search engine built specifically for Vietnamese users, with local-language handling and a search API for developers. It holds a meaningful share of Vietnamese search and browser usage, so sites targeting Vietnam optimise for it alongside Google.

Privacy-focused search engine that combines its own crawling with results licensed from Bing, and publishes its DuckDuckBot crawler documentation. Its refusal to build user profiles removes personalisation from its results, which makes it a useful neutral reference SERP for rank checking.

Not-for-profit search engine founded in Berlin in December 2009 by Christian Kroll, dedicating all profits to tree planting and operating as a steward-owned certified B Corporation. It represents a meaningful share of privacy- and values-motivated European search traffic and, with Qwant, co-founded a European index initiative.

Google's official channel for site owner documentation, covering crawling, indexing, structured data, Search Console and spam policies, plus the Search Central Blog announcing algorithm and core updates. It is the primary normative source for SEO practice, and its guidance changes are treated by the industry as de facto rules.

Subscription-funded, ad-free search engine that blends its own crawling with third-party indexes and lets users up-rank or block domains. Its explicit rejection of advertising revenue and its user-controlled ranking make it a live experiment in what SERPs look like without SEO-driven commercial incentives.

Independent open-source search engine that deliberately ranks non-commercial, text-heavy and low-JavaScript pages, run as a small project rather than a company. It is widely referenced as a counterexample to commercially optimised SERPs and as a discovery tool for content that mainstream SEO practice suppresses.

Microsoft's site owner programme for Bing, providing indexing controls, the IndexNow submission protocol, crawl diagnostics and performance reporting. Its importance has grown because Bing's index also feeds Copilot and several third-party search and AI products.

Independent UK search engine based at the Sussex Innovation Centre in Brighton, which built its own crawler, indexing stack and ranking algorithms and indexed over 9 billion pages as of 2025. It is one of a small number of genuinely independent Western indexes and is frequently cited in search plurality debates.

Naver's webmaster tool and documentation for South Korea's leading search portal, covering site registration, sitemap submission and the Naver-specific crawler. Korean search behaviour is heavily oriented toward Naver's blog, cafe and knowledge verticals, so Naver's guidance differs substantially from Google's.

Operator of ChatGPT and of four documented web crawlers: OAI-SearchBot for ChatGPT search results, GPTBot for model training, ChatGPT-User for user-initiated fetches and OAI-AdsBot for ad landing page checks. Its published robots.txt policy, which lets publishers allow search visibility while blocking training, is now a core technical decision for every site owner.

AI answer engine that crawls the web with PerplexityBot and cites sources inline in its responses. It is a central subject of generative engine optimisation work because its visible citation panel makes source selection observable in a way most AI assistants do not.

French privacy-focused search engine that has invested in building a European web index rather than relying wholly on syndicated results. It matters to SEO as part of the EU's push for search sovereignty and as a distinct index for French-market visibility.

Czech portal and search engine operating since 1996 with its own index and the SeznamBot crawler. It retains a meaningful share of Czech search queries, making it one of the few national search engines outside Asia and Russia that SEO teams must optimise for separately.

Yandex's site owner platform and documentation set, covering indexing management, robots.txt and sitemap handling, snippet and favicon control, and the Wordstat keyword demand tool. It is the authoritative guidance for optimising for the dominant search engine in Russia and several CIS markets.

Search company that pivoted to supplying web search, content extraction and research APIs for AI agents and LLMs, serving over 10 million queries a day. It is one of the small set of vendors whose index decides which pages AI agents can actually retrieve, making it an indirect but real ranking surface.