Major Internet Material Distribution Network (CDN), CloudflareAI has announced war on companies. From July 1, Cloudflare now blocks by default ai web crawler Access the content from your websites without permission or compensation.
The change addresses a real problem. My own small site, where I track all my stories, Practical technologyThe AI Craler has been dramatically slowed down several times. This is not just me. Many website owners have reported that AI Craler, such as Openai’s GPTBOT And CloudboatYield Automatic requests large scale volumes that turn off websites So they are slow as mud. Googlebot Reports alone that cloud-hosting service Workel It bombs the sites that host with more than 4.5 billion requests a month.,
These AI bots often crawl more aggressive sites than traditional search engine crawlets. They sometimes see every few hours or even the same page again Hit sites with hundreds of requests per secondWhile AI companies deny that their bot has to blame, evidence tells a different story.
Also: Senate lifted the ban on state AI rules from Trump’s tax bill
Thus, from its side More than two million customers20%of the web, Cloudflare now blocks AI Craler. For any new website signing up for your services, the AI Craler will be automatically blocked by accessing its content unless the owners of the site give clear permission. Additionally, Cloudflare promises to detect “shadow” scrapers – bots that attempts to find out by using behavior analysis and machine learning. What is good for AI Hans, good for Gander.
This step reverses the previous position, where the website owners had to get out of AI crawling. Now, the blocking is default, and AI vendors should request to access and clarify their intentions, whether for model training, search, or other uses, before they allow them.
This change arises not only due to the owners of the disappointed website. Many publication companies, such as Associated Press, Konde Nast and ZDNET’s own original company, Zif DavisIt is disappointed that AI companies have been “strip mining” on the web for materials. All often, it is done without compensation or consent, and sometimes, such as ignoring standard protocols Robots.txt Those who are for blocking the crawler.
(Disclosure: ZDNET’s original company Ziff Davis filed a case of April 2025 against Openai, alleging that it violates Ziff Davis copyright training and operating its AI system.)
In addition, recent court cases have ruled in favor of meta and anthropic, finding that the use of copyright work was legal under the principle of appropriate use. Needless to say, writers, artists and publishers do not like this a bit. The publishers are still concerned that the federal government will free AI as it wants with their content. The government continues to listen to AI training on copyright data as appropriate use to AI powerhouses such as Openai and Google.
It is also worth noting that its pre-convention version was released after the Copyright Office. 108-Page Copyright and AI ReportWhich supported these two world class industries and attacked a medium ground which contributes a lot to our economic and cultural progress. However, it has been stated that while some generative AI possibly forms a “transformative” use, the mass of all data did not qualify as proper use. The next day, Trump administration fired Copyright Office Head And took her place with a lawyer with no prior experience in the copyright law.
Also: The new ruling of the US Copyright Office on AI Art is here – and it can change everything.
Given all this, it is no wonder that the publishers demanded a colleague in technology.
As CEO Matthew Prince of Claudflare said in a statement, its new policy is to “give publishers to control the controls they deserve and create a new economic model that works for everyone – manufacturer, consumer, tomorrow’s AI founders and the future of the web.”
To complement the move to block AI Craler, Claudflair has also launched it “Payment Per Crol” ProgramThis enables publishers to determine its rates for AI companies that want to earn their content.
Also: AI -Janit images are a legal mess – and still a very human process
The system is currently in private beta and its purpose is to make a framework, where AI firm can pay for access, or refuse to refuse. Technically, this will be done by dusting an old, most unused web server response, dust, Http 402Which reacts with the “payment required” error message. This means that it should be applied and compatible with existing websites and their infrastructure.
Overall, this is a big thing. Thanks to Cloudflare, a significant amount of web materials can be inaccessible for AI companies, thanks to Cloudflare until they pay access or license fees. As Nicholas Thompson, K CEO Atlantic“So far, AI companies do not need to pay for material licenses as they can simply take it without results. They will need to negotiate.,
At this point, most AI companies have been actively against paying for content. As the Sir Nick Cleag, Prime Minister of former Deputy Chief Minister and Meta Executive said, “Recently, ask only before allowing artists. Scrap copyright content “will originally kill the AI industry.”
Also: Cloudflare blocks the biggest DDOS attack – here is how to protect yourself
Cloudflare’s new policy is a direct response to this approach and the increasing amounts and infiltration of the AI crawler coming with it. It is also an attempt to prevent traffic siphoning which will go to the publishers otherwise.
Since the rise of AI, traffic has fallen on news sites. For example, Business Insider’s traffic fell more than half55%from April 2022 to April 2025. Left, Thompson recently predicted that, thanks to AI, Atlantic staff should expect Traffic from Google at zero from traffic.
What will happen next? Other CDNs, such as AkamaiTo follow? Stay For now, the era of unrestricted AI crawling is ending, well, at least for the fifth of the Internet which flows through the pipe of the cloudflair.
Get top stories of morning with us in your inbox every day Tech Today Newsletter.
