Google’s Rare Privacy Sandbox Test Highlights More Questions for the Cookieless Future

Despite the protracted timeline for deprecating third-party cookies scheduled for next year, there is still little data to help marketers understand what that future might look like.

Google Ads is trying to add some clarity to this murky picture.

This week, Google released results of testing interest-based audience solutions, a suite of tools that are designed to be more privacy-safe than third-party cookies.

Essentially, the results found that these solutions only do a slightly worse job in scale, ad quality, and ad relevance than third-party cookies, digital advertising’s decades-old workhorse, which regularly offered inaccurate representations of real audiences online.

In a blog post, vp of Google Ads Dan Taylor called the results “encouraging.” A white paper that further details the findings notes that without third-party cookies, the internet will likely have coarser, less precise data signals. So, the fact the findings show just a slightly weaker performance than cookies might be promising.

Still, broadly, marketers are lagging in testing alternative IDs. Supply-side platforms aren’t testing Google Privacy Sandbox retargeting protocol Fledge (which Google renamed Protected Audience API this week), even though industry experts agree that it’s one of the more promising privacy tech solutions.

Crucially, the utility of the findings lies in the degree to which the industry trusts them.

The fact that Google has introduced Privacy Sandbox solutions like Topics into its browser, Chrome, and that this test was run by separate business units Google Ads and Demand & Video 360, has some people uneasy. Some are wary to take Google’s word that life without cookies will be just fine.

“I find these numbers unverifiable and heavily in Google’s interests,” said Robert Webster, global vp of strategy at marketing tech consultancy CvE. “We have still yet to see independent tests show such results.”   

A Google spokesperson said that the company is committed to using the Privacy Sandbox and other privacy-preserving signals, so it makes sense for the company to test these signals just as any other ad-tech firm might.

“We are conducting this work under the purview of the U.K.’s Competition and Markets Authority and privacy regulators, so there is independent regulatory oversight,” the spokesperson added. The spokesperson added that Google’s aim with Privacy Sandbox is to be transparent and collaborative, and it will continue to do more tests.

What we know: performance is only slightly worse

Google tested the efficacy of three types of interest-based audience solutions (dubbed IBA); contextual signals, Privacy Sandbox protocol Topics API and Publisher Provided IDs, which lets brands transact on first-party data within a particular publisher.

The study did not include publisher-provided signals, a tool Google introduced last fall that lets publishers transact contextual data across the open web according to seller-defined audiences protocol.

Google compared status quo traffic, which includes some IBA signals and third-party cookies, with traffic where third-party cookies had been removed.

Google looked at whether advertisers spent more money bidding on the IBA traffic, compared to traffic with third-party cookies, to understand whether campaigns using privacy-preserving signals would allow marketers to run the same size campaigns. It found advertiser spend on privacy-preserving IBA was 2%-7% less than on third-party cookies.

Google also measured the average number of conversions on an ad compared to the dollars spent on that ad, as a measure of how well the ad worked for marketers. Google found that on the IBA inventory, conversions per dollar were 1% to 3% lower than the conversions generated for status quo traffic powered by third-party cookies.

Finally, click-through rates on IBA traffic were within 90% of the status quo.

Google said click-through rates were a proxy for ad relevance, though Webster questioned the reliability of this metric.

“CTR is a fairly meaningless metric,” Webster said, noting that the IAB U.K. has called on brands to stop using the measure to judge ad effectiveness. “Most clicks on mobile (ie, most clicks) are accidents due to annoying placements.”

What we don’t know: efficacy of each protocol

Google’s test grouped together three different types of signals, showing all their results together, making it hard to isolate the efficacy of any one in the post-cookie landscape.

Of note, observers can’t pull out the results on Topics API, which was rejected by trade body W3C earlier this year and has faced criticism from the wider industry.

Taylor said on a briefing call with reporters this week that not testing Topics API in isolation was intentional.

“We did this because the Topics API is not being designed as a standalone offering,” he said.

The experiment also only tested a set of privacy-preserving targeting protocols and not other current use cases for cookies, like remarketing and attribution, which Privacy Sandbox protocols Fledge and Attribution API aim to respectively solve.

“We definitely want to be able to see how the other APIs perform both in isolation and then as a total package,” Taylor said on the call, on the choice to not test Fledge or Attribution API.

Finally, the test showed that the IBA traffic produced better results for advertisers when they also used some of Google’s AI-driven ad-buying tools. Machine learning has been advantageous in filling gaps amid signal loss.

But what is unknown is how buyers using other demand-side platforms outside of Google’s ad-tech tools might replicate those results.

“Other companies provide their own solutions in this space …. based on their own proprietary technology,” Taylor said. “Google built solutions that are focused on leveraging Google’s AI to deliver results.”