Open-source Python library for SEO and SEM workflows, including crawling with Scrapy, robots.txt and sitemap parsing, SERP APIs, log file parsing and text analysis.
advertools.readthedocs.io
CrawlerFreemiumFree tierMCP server
Screaming Frog SEO Spider
Desktop website crawler for technical SEO audits, covering broken links, redirects, duplicate content, directives, structured data, JavaScript rendering and integrations with Search Console and analytics. Free up to 500 URLs.
What it is
Desktop website crawler for technical SEO audits, covering broken links, redirects, duplicate content, directives, structured data, JavaScript rendering and integrations with Search Console and analytics. Free up to 500 URLs.
It sits in the crawler category, and also covers technical seo. It is built by Screaming Frog Ltd, based in United Kingdom, and has been around since 2010. The vendor also ships an MCP server, so the data can be queried directly from an AI assistant.
The Atlas does not rank tools. No scores, no "top 10", no affiliate links. Categories are descriptive, and two tools in the same category may solve very different problems. Trial the shortlist against your own site before deciding.
Alternatives & neighbours
9 tools sharing at least one category with Screaming Frog SEO Spider.
SEO platform built on its own web crawler and link index, offering Site Explorer, Keywords Explorer, Site Audit, Rank Tracker and Content Explorer. Brand Radar adds AI-answer visibility tracking.
AI visibilityahrefs.com
Chinese webmaster tool site providing composite SEO diagnostics, Baidu weight and keyword ranking analysis, backlink and reciprocal-link checks, keyword mining, dead-link detection and historical site data, plus a paid data API.
aizhan.com
High-performance crawler for large websites with a focus on link graph analysis, PageRank-style internal link metrics and reproducible technical audits.
audisto.com
French link intelligence platform with its own crawl index, providing semantic value and induced strength metrics for pages and domains.
babbar.tech
Microsoft's free site owner console for Bing, providing search performance data, index and crawl diagnostics, URL submission, IndexNow support and keyword research.
bing.com
Enterprise SEO platform combining crawling, log analysis, keyword data and an activation layer that renders and submits pages to search engines. Has extended reporting to AI search surfaces.
AI visibilitybotify.com
Technology lookup service and extension reporting the software, hosting, analytics and advertising stack used by a website, with lead list exports.
builtwith.com
Chinese webmaster tool suite from Chinaz covering SEO overview queries, Baidu weight estimation, app and site ranking lookups, WHOIS, ICP filing checks, reciprocal-link verification and a data API.
chinaz.com