Creative Commons announces temporary support for ‘pay-to-crawl’ AI systems

After announcing it earlier this year frame for the open AI space, not for profit Creative Commons has come in favor of “pay-to-crawl” technology – a flexible way to return content to a website when accessed by machines, such as AI crawlers.

Creative Commons (CC) is best known for leading the licensing community that allows creators to share their work while retaining copyright. In July, the organization he announced the process of providing a legal framework and the ability to share data between companies that control data and AI providers who want to train.

Now, the non-profit organization is supporting the pay-as-you-go system, saying it “supports care.”

“Rightly implemented, paid-crawling can represent a way for sites to continue to create and share their content, and to manage content, keeping content public where it cannot be shared or can end up behind restrictive paywalls,” said CC. blog post said.

Under the direction of companies like Cloudflarea pay-to-crawl idea would be to charge AI bots every time they crawl a page to gather sample training content and updates.

In the past, websites allowed browsers to enter their content into search engines like Google. They benefited from this system by seeing their site listed in the search results, which drives visitors and clicks. With AI technology, however, the dynamics have changed. When a customer gets their answer through an AI chatbot, they can’t click through to the source.

This change has already happened damaging to publishers by killing non-hunters, and shows no tolerance.

A paid-to-crawl approach, on the other hand, can help publishers recover from the AI ​​hit they’ve had. In addition, it can work well for small publishers who are not interested in negotiating with AI providers. Great deals have been made between companies like OpenAI and Condé Nast, Axel Springer and others; and between Confused by Gannett; Amazon and the New York Times; and Meta and various publishers, in others.

The CC issued a number of warnings against supporting pay-to-crawl, noting that such practices can put pressure on the Internet. It would also limit access to “researchers, non-profits, civil society organizations, educators, and others engaged in the public interest.”

It made several important recommendations for paid-crawling, including not making crawling standard across all websites and avoiding vague rules on the web. In addition, it said that pay-to-crawl systems should be flexible, not restrictive, and to protect public access. They must also be open, coherent, and built with stable components.

Cloudflare isn’t the only company that sells paid hosting.

Microsoft is also developing software for AI market for publishers, and smaller startups like ProRata.ai and TollBit are starting to do so as well. Another group called the RSL Collective he announced owns a new standard called Really Simple Licensing (RSL) that dictates which parts of a website they can access but stops short of crawling. Cloudflare, Akamai, and Quickly took over the RSL, which is behind and Yahoo, Ziff Davis, O’Reilly Media, and others.

CC was also one of them he announced his support for the RSL, along with CC symbolsits larger project to develop technology and tools for the AI ​​era.