Cloudflare is pushing to create a market for licensed AI content, offering a compliant crawler aimed at giving publishers more control and reducing inefficient site crawls — even as some question the irony of the company that once blocked scrapers now building one itself.
Last week, the company released a crawl API (its crawl endpoint within Cloudflare’s browser rendering API) that can scrape an entire website with one request. (You give it a URL, it crawls the whole site and returns the content in HTML, Markdown, or structured JSON). The announcement raised eyebrows and left some publishers confused and unsettled by the move to provide its own version of what it has so far protected publishers from.
That sentiment was exacerbated by some publishers noticing how they couldn’t block Cloudflare’s own scraper when they tried to adjust their settings. One independent publisher Thomas Baekdal was particularly vocal, posting on LinkedIn that Cloudflare had “betrayed every single publisher” with this tool.
Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.
Ready to launch your brand’s affiliate program? This Affiliate Marketing Step-by-Step guide for brand marketers covers everything from setting goals and choosing a platform to recruiting partners and optimizing for growth, including the benefits of affiliate networks and advanced technology. Contact us to get started!,