This story was originally published in On Background with Mark Stenberg, a free, weekly newsletter that explores the key themes shaping the media industry. You can sign up for it here.
Stop me if you have heard this before, but it bears repeating: When OpenAI debuted ChatGPT in November 2022, it introduced an existential problem that, even after more than three years, has yet to find a solution.
In short, answer engines rely on the copious availability of quality data to fuel their responses, which they draw from content creators, like publishers. Yet in answering users’ questions without sending those users back to the websites that supplied the data, they starve these publishers of revenue, creating a system that is, quite obviously, unsustainable. Over time, the publishers will wither, the answer engines will lose their data providers, and the model will collapse.
This fundamental paradox has risen from a concern to a crisis for the digital media industry, which has seen its collective search traffic, and its attendant revenue, plummet in recent years. It has spurred countermeasures from content creators, such as diversifying their traffic sources or signing one-off deals with answer engines, but these are stays of execution, at best.
For the open web to survive, a new payments infrastructure needs to emerge that allows publishers to be compensated by AI firms for use of their data. And so far, the early results on this front have been uninspiring: Most of the initiatives involve publishers restricting web crawlers’ access to their content—an effort to create scarcity and, eventually, increase the value of their archives.
Outside of the many technical challenges involved in setting up such a system, a far more basic roadblock has so far proven impassable: The AI firms have little need to pay. Like any good boycott, publishers’ withdrawal of access only works if no one crosses the picket line, which is to say that every last website and content creator in the world would have to link arms in unison to make such a withholding work. Naturally, that has not happened.