Myriam Jessier asked Google about what would be good attributes of a web crawler. In which both Martin Splitt and Gary Illyes gave some responses to.

First, Cloudflare accused the AI company of bypassing no-crawl directives. Now Perplexity says Cloudflare has it all wrong.

In a debate likely to get louder as AI agent usage grows, some people say Perplexity crawling blocked websites isn't a simple matter.

Publishers and other providers of creative content now have the option to block AI crawlers from accessing and scraping their intellectual property with new tools from Cloudflare.

Internet firm Cloudflare has started blocking AI web crawlers to prevent them from "accessing content without permission or compensation," by default according to an announcement on Tuesday.

Pay up or stop scraping: Cloudflare program charges bots for each crawl Cloudflare now beta testing pay-per-crawl feature to stop endless AI scraping.

Cloudflare reshapes the AI landscape by blocking Big Tech AI bot crawlers by default, allowing websites to demand payment for content access.

Google’s “What’s New in Web UI” Talk: Less Custom Component JavaScript, More Web Standards Jun 08, 2025 2 min read ...

Google is rolling out Project Mariner, an AI agent born out of Google DeepMind that browses and uses websites on a user's behalf.

Google's AI crawler, Google-Extended, the crawler it uses for Gemini and other related AI services, does render itself and can render JavaScript, just like Googlebot, Google's main web search crawler.