Why publishers are questioning the effectiveness of blocking AI web crawlers

A number of publishers — including Bloomberg and The New York Times — were quick to block OpenAI’s web crawler from accessing their sites, to protect their content from getting scraped and used to feed the artificial intelligence tech company’s large language models (LLMs). But whether this tactic is actually effective is debatable, according to conversations with five publishing executives.

“It’s a symbolic gesture,” said a senior tech executive at a media company, who requested anonymity to speak freely.

In August, OpenAI announced that publishers can now block its GPTBot web crawler from accessing their web pages’ content. Since then, 26 of the 100 most-visited sites (and 242 of the top 1,000 sites) have done so, according to Originality AI.

Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.

,Read More