This week’s Media Briefing looks at how AI crawlers now reach publisher content, one year after Cloudflare’s default bot changes.
- Why the black market scraping economy still wins.
- Netflix strikes video licensing agreements with Penske Media digital brands including Variety and Rolling Stone.
From default blocks to ‘mixed-use’ bots
The AI crawling environment for publishers is fragmenting. On one side is a visible layer of declared “good bots” that go through infrastructure gatekeepers like Cloudflare, Fastly and Akamai. On the other is a much larger gray crawling economy that still sits largely outside meaningful oversight.
Last week, Cloudflare rolled out new rules for “mixed crawlers” — bots that can be used both for basic search and for AI training or agentic tasks — declaring a “deadline to end the free pass” for these kinds of bots, which it says disadvantages site owners and more transparent AI companies. On Sept. 15, the defaults will be set to allow for search but block training and agent use for pages with ads.
Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.