AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
Automated, non-human requests to a site — crawlers, scrapers, monitoring, agents — which increasingly outnumber human visits and distort analytics if not filtered.
The category has broadened with agentic AI: alongside classic bots there are now retrieval fetchers, agent browsers acting for users, and scrapers rebuilding indexes. Client-side analytics usually miss them because they do not run JavaScript, so logs and CDN analytics tell a different story from GA4. Distinguishing valuable automated traffic from parasitic load is now a real capacity-planning question for large sites.
CDN logs show automated agents accounting for a majority of requests while GA4, being JavaScript-based, records almost none of it.
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.
AI systems that navigate websites on a user's behalf — reading pages, filling forms, comparing options and sometimes completing…
Google's event-based analytics platform, which replaced Universal Analytics in July 2023 and models sessions and users from event streams.
Analytics platforms estimating results from a subset of data, or suppressing small values for privacy, both of which distort reporting if…