Day 7: We're building a wall, and AdX is paying for it

The Monopoly Report is going daily during the trial. Our intrepid founder is in a windowless room without a phone or laptop to get you the good stuff.

Day 7: We're building a wall, and AdX is paying for it

The majority of today was spent with Google’s buy-side engineering leader Nirmal Jayaram. Despite emails claiming there was a “Chinese Wall” between Google’s buy-side and sell-side products, plenty of evidence was shown where Google Ads demand was strategically supporting AdX’s business goals.

Here’s what happened.

Stuff we learned

• The acronym “IIUC” stands for “If I understand you correctly”

• Project Poirot was named after the Belgian detective because it involved detecting bad auctions.

• In addition to Project Poirot, there was a Project Marple. Agatha Christie FTW.

Plaintiff’s witness 20: Nirmal Jayaram, Engineering Director, Google

Thus far most of the Google testimony has been on the sell-side, so Nirmal shed light on the various interactions between Google Ads and DV360 with those sell-side products. At the heart of the DOJ’s case is the flow of ad dollars from Google Ads to AdX, and the way that flow allegedly locks publisher customers in to the DFP-AdX combination.

We also hear from Nirmal about the buy-side’s approach to header bidding, Open Bidding, and Project Poirot. He was on the stand for over five hours, so there’s a lot here.

Google Ads demand and AdX

Previous testimony has established that Google Ads demand is very valuable to publishers and hurts their ability to move off of DFP and AdX. The DOJ now is making the argument that the Google Ads demand was being artificially funneled for this purpose, potentially against the best interests of Google Ads.

An internal document labeled “GDN as an AdX Buyer” laid it all out:

This subsidization of AdX was quantified by a January 2014 simulation run by Nirmal in which he estimated that removing GDN demand would reduce AdX revenue by 70%.

In a previous newsletter we also noted that Google Ads is giving AdX a 20% pricing subsidy as well. That is not considered here.

AWBid’s long, tortured history

In previous newsletters we have discussed AWBid, Google’s program to allow a small sliver of demand from Google Ads to bid on third party exchanges. Nirmal worked on this project from the start of his time at Google. From the first document discussion “Adwords cross-exchange bidding” (roughly 2012-13) there was already a consideration of the effect on the sell-side. An early document raised concerns about “eliminating the AdSense and AdX USP… that is the only access point for dynamic AdWords demand.” And that “we expect many publishers would terminate their AdX relationship” without this demand.

In December 2015 a document was created to develop a joint buy/sell side “unbiased view” on some of these issues. This document noted that DV360 spends ~53% on external exchanges (this number later skyrockets above 70%), while AWBid only spends ~7%. The document also suggests the sales team can “use AWBid data as a lead list to have conversations with publishing partners”, which must make the folks at Rubicon, Pubmatic, et al very happy.

The risks of expanding AWBid to the sell-side are clearly described in this 2015 document:

• Loss of revenue / inventory

• Publishers put inventory on other exchanges

• Sales value proposition weakens

• Revenue share reduced

On cross-examination, the witness pointed out the many difficulties in expanding AWBid, including spam, latency/timeouts, discrepancies, billing, and others. (As a former CEO of a bidding company, I am playing a very small violin while listening to this testimony). Some of the mitigating facts about AWBid’s slow roll-out:

• At one point 98% of clicks from Pulsepoint were deemed fraudulent.

• Google did not have a system for reconciling discrepancies or refunding customers for fraud.

• Early in AWBid they missed their latency window ~50% of the time. By 2019 the 95% latency measurement was 107ms, vs a goal of 80-100ms.

• 50% of AWBid auctions did not have a cookie match.

• They ran out of servers (seriously, this was in the testimony).

The DOJ countered this litany of woes with documents from 2014 saying “inventory quality meets AdX policy ratings” and a 2018 document stating “Spam rates are comparable to AdX.”

Was AWBid slowed down by normal course-of-business issues, or internal politics meant to protect AdX? We may never know. But we do know that as of today, Google Ads is still only bidding with a fraction of demand. Google wanted to include testimony of the current amount spent by AWBid, but the judge did not allow it.

The buy-side perspective on header bidding

In a document with exactly the above name from September 2016 the buy-side folks at Google are pretty psyched about header bidding.

This document advocates for DV360 getting inventory directly from the header, though the exact mechanism is not discussed. The document also undermines some of the sell-side arguments against header bidding by describing the common criticisms of latency, spam, etc as “manageable or misperceptions.”

The buy-side authors weigh in on Jedi, AKA Open Bidding with a comparison I’ve reproduced below:

In the same doc, the reasons buyers want header bidding are enumerated clearly:

• Lower margin

• Better access

• Transparent pricing

• First party cookies

• Not subject to exchange policies

Sounds like the buy-side of Google should have joined Prebid.org! The buy-side enthusiasm for header bidding mostly serves to reinforce the conflict of interest within the organization.

Project Poirot

Poirot has been covered previously. In brief, it was DV360’s bid shading project, that had the effect of moving a great deal of auctions off of external exchanges to AdX. Google argues they were doing what was right for buyers in the face of “dirty auctions”, the plaintiff’s allege this was just another attempt to “dry out” header bidding.

The witness, Nirmal, was involved in Poirot in a number of ways. First, Nirmal ran an experiment where all DV360 buying on external exchanges was turned off entirely for 0.5% of traffic. The hypothesis was that if all traffic was duplicative and external exchanges were not incremental, then there would be no negative effect on buyers. This hypothesis was thoroughly disproven when DV360 buyers saw a 35% decline in impressions and a 47.94% decline in revenue.

The DOJ kept coming back to this experiment as some kind of smoking gun, with the implication that Google would shut off external exchanges if they could.

Google did not shut off external exchanges, but they did move forward with reducing the bids on those exchanges. Evidence was shown about how Google determined that an exchange was running “dirty auctions.” A slide was shown where the cost/bid ratio for different exchanges was shown. In a true second-price auction this ratio should be low, as bids are reduced. In a first price auction this ratio would be 1.00. Here are some of the numbers presented by Google:

In previous testimony we have seen the amount Poirot reduced revenue per exchange and the number align roughly with the order shown here, with AdX barely effected and AppNexus, OpenX, and Pubmatic taking it on the chin (see below for OpenX’s perspective).

In a document about Poirot, product manager Tobias Maurer made a comment “in the case of RPO [Reserve Price Optimization] being enabled this will tend towards charging 95% of the fixed bid.” In other words, Tobias is saying that in some cases AdX is running a dirty auction. The witness said Tobias was wrong, and a response to his comment in that same document said “Poirot doesn’t detect AdX as treatworthy.” [“Treatworthy” is a new one for me, too, presumably it means worth treating.]

After the Poirot launch in August, 2017 AdX’s share of DV360 impressions rose from 43% to 48% and external exchanges saw a 10% decline in share.

One interesting tidbit in the testimony that hadn’t been previously disclosed is that Poirot’s bid reduction factors were extremely crude, and were not at the auction level, or the publisher level, but were just at the exchange level, and only updated daily.

In sum, Nirmal’s testimony was helpful for the government, as it clearly showed conflicts of interest and the lack of a “Chinese Wall” between buy- and sell-sides. For whatever reason, the AWBid project to move demand to external exchanges has been very slow, and that demand remains only available on AdX. The Poirot facts are a lot less clear, and the argument that program was good for buyers remains a strong one.

Plaintiff’s witness 21: Tim Cadogan former CEO, OpenX

Tim’s testimony was pretty much what you would expect:

• They used to run an ad server, but couldn’t compete with DFP because of the AdX demand.

• The exchange was very successful, then suddenly in the Fall of 2018 DV360 demand dropped off a cliff (see Poirot, above).

• As a result of the 2018 changes, he had to lay off 200 people, and the company has still not regained the spend levels prior to this.

On cross examination the picture got more nuanced and Google scored some points:

• The 2018 revenue decline was not just DV360, but also TTD.

• The ad server was losing customers for a bunch of reasons, including poor VAST support and poor sales and service.

• By 2014 OpenX only had two engineers working on the ad server.

• OpenX had a feature called “Dynamic Allocation” and another feature called “First Look”.

On its own Tim’s testimony had some issues, but since it matches very clearly with other competitors, like Equativ and Kevel, it probably holds more weight.

Plaintiff’s witness 17: Brian O’Kelley continued (video deposition)

They are shoving little bits of BOK’s testimony in whenever they have a second. Today we got the last of it. There was one interesting moment in the testimony shown today, when a blog post BOK wrote was quoted as saying publishers should “use hard floors” and those floors should be “consistent across exchanges.” Sounds like BOK would be in favor of Unified Pricing Rules!

Plaintiff’s witness 22: Jeremy Helfand, formely of Disney (deposition only)

Sorry, I wasn’t able to stay for this. Will update you tomorrow if anything interesting happened.

What’s next?

Tomorrow I believe we’ll hear from long-time AdX product lead Scott Spencer.