Internet firm Cloudflare has started blocking AI web crawlers to prevent them from “accessing content without permission or compensation,” by default according to an announcement on Tuesday.
EPA-EFE/WU HAO
July 1 (UPI) — Cloudflare announced it will begin blocking AI web crawlers to prevent them from “accessing content without permission or compensation,” from all of its clients beginning on Tuesday.
Cloudflare blocking AI crawler bots builds off the tool launched in September last year that allowed publishers the ability to block crawlers with one click but announced Tuesday the option to block them will be implemented by default for all of its clients.
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” said Matthew Prince, co-founder and CEO of Cloudflare. “This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
Cloudflare is a content delivery network which helps businesses and applications by caching the data closer to end users and it estimated that 16% of global internet traffic goes directly through it’s service in a 2023 report.
The company also announced it will implement a Pay Per Crawl program that will allow some publishers to set a price that can be viewed by companies to decide whether they want to pay the fee for its content.
AI crawlers are automated bots with the intent to extract large amounts of data from websites, to train large language models from companies such as OpenAI and Google.
AI crawlers are typically seen as more invasive and selective when it comes to the data they consumer. They have been accused of overwhelming websites and significantly impacting user experience,” Matthew Holman, a partner at U.K. law firm Cripps, said.
“If effective, the development would hinder AI chatbots’ ability to harvest data for training and search purposes,” he added. “This is likely to lead to a short term impact on AI model training and could, over the long term, affect the viability of models.”