Category: In the News

Robots.txt got some much-needed TLC last week, courtesy of Cloudflare’s latest update. 

Cloudflare’s new Content Signals Policy effectively upgrades the decades-old honor system and adds a way for publishers to spell out how they do (and perhaps more importantly – how they don’t – want AI crawlers to use their content once it’s scraped.) 

For publishers, that distinction matters because it shifts the robots.txt file from a blunt yes-or-no tool into a way of distinguishing between search, AI training and AI outputs. And that distinction goes to the heart of how their content is used, valued and potentially monetized. 

Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.

,Read More

AI Mode isn’t just the name of Google’s newest search product. It also describes the present period of digital media — and the theme of this September’s Digiday Publishing Summit.

AI is no longer the next epoch of the platform era. It’s the present. Publishers are seeing generative AI chatbots siphon search traffic at the same time as they are having to defend against AI crawlers scraping their sites. They are generating revenue from the chatbots’ parent companies while developing new revenue models for the potential day when the open web gives way to the agentic web. They are incorporating AI technologies into virtually every facet of their businesses, from ad sales to editorial products, from subscription paywalls to programmatic sales.

This is a member-exclusive article from Digiday. Continue reading it on digiday.com and subscribe to continue reading content like this.

,Read More