The New York Times Updates Terms of Service to Prevent AI Scraping Its Content

Among the early use cases of AI within newsrooms appears to be fighting AI itself.

The New York Times updated its terms of services Aug. 3 to forbid the scraping of its content to train a machine learning or AI system.

The content includes but is not limited to text, photographs, images, illustrations, designs, audio clips, video clips, “look and feel” and metadata, including the party credited as the provider of such content.

The updated TOS also prohibits website crawlers, which let pages get indexed for search results, from using content to train LLMs or AI systems.

Defying these rules could result in penalties, per the terms and services, although it’s unclear what the penalties would look like. When contacted for this piece, The New York Times said that it didn’t have any additional comment beyond the TOS.

“Most boilerplate terms of service include restrictions on data scraping, but the explicit reference to training AI is new,” said Katie Gardner, partner at Gunderson Dettmer.

AI models rely on content and data, including journalism pieces and copyrighted art, as a main source of information to output results. In some cases, this content is replicated verbatim. Publishers, especially those with paywalls and healthy subscription businesses, are concerned that AI models will undermine their revenue streams by publishing repurposed content without credit, and contribute to misinformation, degrading people’s trust in news.

The confusing case of creepy crawlers

LLMs like ChatGPT work similarly to website crawlers which scan content on publishers’ sites and feed their information to inform search results.

While publishers can see crawlers visiting their sites, they cannot know their exact purposes, whether for search engine optimization or training AI models. Some paywall tech companies are looking at ways to block crawlers, according to Digiday’s reporting.

Crawlers like CommonCrawl, with a data set of 3.15 billion web pages, have brokered deals with OpenAI, Meta, and Google for AI training, per The Decoder.

Earlier this week, OpenAI launched GPTBot, a web crawler to improve AI models. This will let publishers control GPTBot’s access to their website content. Still, significant players in the field, namely Microsoft’s Bing and Google’s Bard, have not added this functionality to their bots, leaving publishers struggling to control what the crawlers scrape.

While tech companies like OpenAI are reticent to disclose what they train their AI models on, The Washington Post analyzed Google’s C4 data set, a smaller version of the CommonCrawl dataset, to understand what was training the models. It found evidence that content from 15 million websites, including The New York Times, have been used to train LLMs such as Meta’s LLaMAa and Google’s T5—an open-source language model that helps developers build software for translation tasks.

All this has spurred other publishers to reevaluate their terms of services, according to Chris Pedigo, svp for government affairs at trade body Digital Content Next, whose members include The New York Times and The Washington Post.

More licensing deals to come

While it’s unclear how AI companies will respond to these updated terms of services, they have a vested interest in shielding themselves from legal repercussions.

As a result, discussions are underway between AI companies and major publishers to establish licensing agreements, according to Pedigo, such as the deal between OpenAI and The Associated Press.

These deals are primarily set for AI companies to compensate publishers for their content. However, there’s a desire from publishers to go beyond just financial matters.

Ongoing negotiations look at how to cite publishers for their content, including aspects like footnotes. Simultaneously, there is a focus on establishing mechanisms such as guardrails and fact-checking processes within AI companies to prevent the generation of factually inaccurate content by the LLMs.

“Publishers would not want to be associated with that, especially if they’re going to have a licensing deal,” said Pedigo. “Publishers want to make sure that information meets the brand level.”