Cloudflare changes the rules of AI bots to help website owners

Nearly 30 years ago, an internet model was born based on a mutually beneficial exchange: search engines (primarily Google) were allowed to index content, and in return directed traffic to websites. This allowed authors to earn from advertising, subscriptions, or at least gain recognition. But recent years have changed everything. More and more people seek answers in AI tools, such as ChatGPT, rather than Google.

The latter, in turn, has itself started “answering” instead of redirecting to third-party resources: initially through the Answer Box, and now through AI Overviews. Consequently, more content is copied, and less traffic goes to sources. And if previously Google at least maintained some balance, modern AI models, like OpenAI or Anthropic, practically do not return traffic to authors.

“The web has become a ‘raw resource’ for AI – without compensation to those who create this content.”

Cloudflare changes the rules of AI bots to help website owners - image 1

Cloudflare changes the game

As of July 1, Cloudflare announces the start of a new era on the internet. Henceforth, AI bots attempting to access content without permission or compensation are blocked by default. Together with leading publishers, social networks (such as Reddit and Pinterest), and AI companies, Cloudflare is setting new rules: artificial intelligence must pay for learning from others’ content. Site owners gain control: they can allow or deny indexing, as well as set a price for access through the ‘pay per crawl’ model.

Cloudflare – a global leader in cybersecurity and internet resource optimization, offering protection from DDoS attacks, bot management, traffic optimization, and global corporate network creation. iIT Distribution – the official distributor of Cloudflare in Ukraine, is engaged in promoting its solutions in the market.

New internet: less traffic – more value

Cloudflare offers to rethink the value of content. Instead of a simple traffic metric, the company wants to introduce an evaluation based on the “Swiss cheese” principle: if your content fills “gaps” in AI models’ knowledge, it should be especially valuable. This approach will support the creation of original and high-quality content, rather than mass duplication of information. Cloudflare is already working with leading economists and scientists to create a market where AI companies can buy precisely the content they truly need.

A new chance to change the internet

The traditional internet model, namely indexing in exchange for traffic, no longer works. Cloudflare initiates a transformation: instead of depleting the content resource – reward the authors. This is not only a chance for a new economy for publishers but also an opportunity to make AI better and more responsible. And if everything goes according to plan, we will witness a new golden era of digital creativity under conditions of fairness and transparency.

NEWS