“Stealth crawler” sounds like something from a sci-fi spy story, but the term has gained momentum with recent legislation trying to crack down on these web crawlers.
A bill passed in New York last month and legislation introduced in the U.S. House of Representatives in July are pushing to prohibit bots from masking their identity.
The timing tracks. Bot traffic is no longer a fringe problem. Cloudflare data shows more than half of all web traffic is now bot based. And a growing share of that is crawlers pulling content off publisher sites without identifying themselves or their purpose. Publishers are left in the dark on both counts: who’s taking their content, and what it’s being used for.
Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.