Tool category
Crawler
Site crawlers fetch and parse your pages the way a search engine would, surfacing broken links, redirect chains, duplicate titles, blocked resources and rendering problems at scale.
17 tools · 10 free or with a free tier · 8 more cover this area as a secondary feature
| Tool | What it does | Pricing | From | HQ |
|---|---|---|---|---|
| Audisto audisto.com | High-performance crawler for large websites with a focus on link graph analysis, PageRank-style internal link metrics and reproducible technical audits. | Paid | — | Germany |
| BotifyAI botify.com | Enterprise SEO platform combining crawling, log analysis, keyword data and an activation layer that renders and submits pages to search engines. Has extended reporting to AI search surfaces. | Enterprise | — | France |
| Conductor Website Monitoring (ContentKing) conductor.com | Real-time website monitoring and auditing product, originally ContentKing, that continuously tracks pages for changes and technical SEO issues and alerts on regressions. | Enterprise | — | Netherlands |
| Crawlee crawlee.dev | Open-source web scraping and crawling library for JavaScript and Python that handles request queues, proxy rotation, blocking mitigation and headless browsers, usable for building custom SEO crawlers. | Open Source | — | — |
| JetOctopus jetoctopus.com | Cloud crawler and log analyser aimed at large websites, with fast crawling, log file processing and Search Console data joins. | Paid | — | Ukraine |
| Katana github.com | Open-source Go crawling and spidering framework from ProjectDiscovery with both standard and headless-browser modes, used for URL discovery and JavaScript-rendered link extraction. | Open Source | — | — |
| LibreCrawl Technical SEO Audit MCP github.com | Open-source MIT-licensed MCP server built on LibreCrawl that exposes 37 crawling and auditing tools with more than 50 technical SEO checks, unlimited page counts and ephemeral storage, for use from Claude, Cursor or Codex. | Open Source | — | — |
| Lumar lumar.io | Enterprise website intelligence platform, formerly DeepCrawl, providing large-scale crawling, technical SEO monitoring, accessibility testing and automated protection against site regressions. | Enterprise | — | United Kingdom |
| Netpeak Spider netpeaksoftware.com | Desktop SEO crawler for on-page audits, detecting broken links, duplicate content, redirect chains and other technical issues, with multi-domain scraping support. | Paid | — | Ukraine |
| Oncrawl oncrawl.com | Technical SEO platform combining a large-scale crawler, server log file analysis and integration with ranking and analytics data for enterprise sites. | Enterprise | — | France |
| Python SEO Analyzer github.com | Open-source Python tool and CLI that crawls a site, analyses its structure, counts body-copy terms and warns about common technical SEO issues. | Open Source | — | — |
| Scrapy scrapy.org | Open-source Python framework for extracting data from websites at scale, commonly used to build bespoke SEO crawlers and SERP or content data pipelines. Maintained by Zyte with hundreds of contributors. | Open Source | — | — |
| Screaming Frog MCP github.com | Open-source headless MCP server that wraps the Screaming Frog SEO Spider command line, letting AI agents start crawls and export or analyse crawl data for scheduled audits and CI use. Community project, not published by Screaming Frog. | Open Source | — | — |
| Screaming Frog SEO Spider screamingfrog.co.uk | Desktop website crawler for technical SEO audits, covering broken links, redirects, duplicate content, directives, structured data, JavaScript rendering and integrations with Search Console and analytics. Free up to 500 URLs. | Freemium | $279 | United Kingdom |
| Sitebulb sitebulb.com | Website auditing tool available as a desktop application and a cloud server product, producing prioritised technical SEO hints with explanations and visualisations of site architecture. Sitebulb Cloud plans start at £95/month; the desktop licence is priced separately. | Paid | — | United Kingdom |
| Siteliner siteliner.com | Free website scanner from the makers of Copyscape that reports duplicate content, broken links and page structure for small sites. | Freemium | — | United States |
| Visual SEO Studio visual-seo.com | Windows desktop crawler for technical SEO audits, aimed at consultants and small agencies, with structured data checks and sitemap tooling. | Freemium | — | Italy |
1 of 17 vendors in this category publish a starting price. Where the column is empty, the vendor quotes on request or the price was not published at the time of compilation.
Also covers crawler
These tools are catalogued under another primary category but include crawler functionality.
Other categories
GEO AI Visibility (47)All In One (32)Analytics (15)Content Optimization (15)Keyword Research (15)Page Speed (15)Rank Tracking (14)SERP API (13)Browser Extension (12)Free Utility (12)Local SEO (10)Reporting (10)AI Writing (9)WordPress (9)Digital PR (8)Technical SEO (7)Backlinks (6)Schema (6)Log Analysis (5)Ecommerce SEO (4)Internal Linking (4)Enterprise (3)