European publishers are getting hit harder than North American sites by AI bot scraping – and no one really knows why.
A recent report from TollBit found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules – which websites use to communicate to crawlers what they can and cannot scrape – ignored more often.
Here are the main findings from that report, based on TollBit’s analysis identified AI bots from 40 scraping vendors across 3,906 publishers:
Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.