Open-source Python library for SEO and SEM workflows, including crawling with Scrapy, robots.txt and sitemap parsing, SERP APIs, log file parsing and text analysis.
advertools.readthedocs.io
CrawlerOpen SourceFree tier
Scrapy
Open-source Python framework for extracting data from websites at scale, commonly used to build bespoke SEO crawlers and SERP or content data pipelines. Maintained by Zyte with hundreds of contributors.
What it is
Open-source Python framework for extracting data from websites at scale, commonly used to build bespoke SEO crawlers and SERP or content data pipelines. Maintained by Zyte with hundreds of contributors.
It sits in the crawler category, and also covers technical seo. It is built by Zyte.
The Atlas does not rank tools. No scores, no "top 10", no affiliate links. Categories are descriptive, and two tools in the same category may solve very different problems. Trial the shortlist against your own site before deciding.
Alternatives & neighbours
9 tools sharing at least one category with Scrapy.
SEO platform built on its own web crawler and link index, offering Site Explorer, Keywords Explorer, Site Audit, Rank Tracker and Content Explorer. Brand Radar adds AI-answer visibility tracking.
AI visibilityahrefs.com
Chinese webmaster tool site providing composite SEO diagnostics, Baidu weight and keyword ranking analysis, backlink and reciprocal-link checks, keyword mining, dead-link detection and historical site data, plus a paid data API.
aizhan.com
High-performance crawler for large websites with a focus on link graph analysis, PageRank-style internal link metrics and reproducible technical audits.
audisto.com
French link intelligence platform with its own crawl index, providing semantic value and induced strength metrics for pages and domains.
babbar.tech
Microsoft's free site owner console for Bing, providing search performance data, index and crawl diagnostics, URL submission, IndexNow support and keyword research.
bing.com
Enterprise SEO platform combining crawling, log analysis, keyword data and an activation layer that renders and submits pages to search engines. Has extended reporting to AI search surfaces.
AI visibilitybotify.com
Technology lookup service and extension reporting the software, hosting, analytics and advertising stack used by a website, with lead list exports.
builtwith.com
Chinese webmaster tool suite from Chinaz covering SEO overview queries, Baidu weight estimation, app and site ranking lookups, WHOIS, ICP filing checks, reciprocal-link verification and a data API.
chinaz.com