Ad Exchanges

State of Ads.txt Adoption

September 16, 2017

What is ads.txt?

Ads.txt is an IAB Tech Lab project that was created to fight inventory fraud in the digital advertising industry. The idea is simple; publishers put a file on their server that says exactly which companies they sell their inventory through. The file lists partners by name, but also includes the publisher’s account ID. This is the same ID buyers see in a bid request, which they can use as a key for campaign targeting.

Buyers use a web crawler to download all the ads.txt files and the information contained within on a regular basis and use it to target their campaigns. This means buyers know that if they bid on request that comes from an authorized ID, it’s coming from a source the publisher trusts or has control over. Buyers seem to be taking the idea seriously, too. Just a week ago Digitas published an open letter on Digiday saying they won’t buy from any publisher without an ads.txt file.

Ads.txt isn’t a silver bullet for all inventory quality woes, but it is a dead simple solution. You’d be stupid not to lock the door to your house, even if it’s not a guarantee of safety, right? The important bit is that for the first time publishers have a tool against inventory fraud instead of relying on the programmatic tech alone.

Are you a developer or patient person? Then try the ads.txt crawler yourself

As part of the program’s release, Neal Richter, a long time ad technology veteran and one of the authors of the ads.txt spec wrote a simple web crawler in Python. The script takes a list of domains and parses the expected ads.txt format into a database, along with some other error handling bits.

Developers will probably find it a piece of cake to use and non-developers will struggle a bit, like I did. That said, I got it running after pushing through some initial frustration and researching how to get a small database running on my computer. I wrote a detailed tutorial / overview of how to get it working for anyone interested in a separate post.

12.8% of publishers have an ads.txt file

[amcharts id=”pie-1″]

At least, among the Alexa 10K global domains that sell advertising. To get this stat, I took the Alexa top 10,000 domains, removed everything owned by Google, Amazon, and Facebook – which don’t sell their inventory through 3rd parties and therefore don’t need an ads.txt file – and removed the obvious pornography sites. After filtering, I had 9,572 domains to crawl. I sent all those through Neal’s crawler and found 1,930 domains selling ads, and 248 with an ads.txt file. 248 / 1,930 = 12.8%, voila!

Update: Nov 1, 2017

In the less than 6 weeks or so since I published my first analysis, ads.txt adoption has continued to mushroom and now stands at 44%. I’m astonished to see the adoption rate triple over such a short time frame and I have to imagine this sets a record for publisher embrace of any IAB standard. So what’s driving it? My own opinion is the primary driver is Google’s Doubleclick Bid Manager’s announcement that they’d stop buying unauthorized supply paths at the end of October, which had led to a big grassroots push among the major exchanges with their publisher clients.

[amcharts id=”chart-1″]

(more…)

What is Holistic Ad Serving?

September 24, 2012

Certainly one of the biggest opportunities in ad tech today is integrating real time bidding (RTB) systems to core ad serving platforms such that ad serving decisions are made from a single system. This vision of a fully integrated monetization stack is known as holistic ad serving, and it’s going to be big.

Holistic ad serving consolidates what is today a fragmented marketplace, modernizes the publisher ad serving stack, and lays the groundwork for advertisers and publishes to transact guaranteed campaigns over RTB infrastructure. In other words, it provides a way for publishers to transition from a world of manual campaign implementations to accepting and trafficking campaigns programmatically without having to manage the balance between two systems.

Tactically, holistic ad serving is a seems like a basic change – instead of filling direct campaigns first and then letting the exchange try to fill whatever is left, the idea is for publishers to call to the exchange marketplace and get a bid for every single impression, thereby allowing RTB demand to compete directly with the traditionally sold campaigns with guaranteed goals. By at least getting a bid for every impression, the publisher’s ad server can understand the benefit or cost of filling an impression with a direct campaign – it has all the information. Holistic ad serving also opens the possibility, on an impression by impression basis, for an RTB campaign to trump a direct campaign. (more…)

History of the Ad Exchange Landscape Part IV: The Ad Network Is Dead, Long Live the Ad Exchange

May 3, 2011

Part IV: The Ad Network Is Dead, Long Live the Ad Exchange

I’ve written a few times about Ad Exchanges on this site, as they are one of the more exciting areas to think about in digital advertising right now. Ad Exchanges such as Google Exchange, Right Media Exchange, or Microsoft Exchange are an elegant solution to this mess of redirects and inefficient monetization for the industry at large and offer the revolutionary opportunity to do true audience targeting. Via an ad exchange, sellers can auction their inventory to the highest bidder through a single redirect, and buyers can evaluate and bid on that inventory impression by impression using rich pools of their own or 3rd party data. This rich targeting gives ad exchanges a big advantage over ad networks in terms of transparency and targeting capabilities and since ad exchanges aggregate the inventory from thousands of publishers in the same place, exchanges also offer exponentially more reach than any ad network ever could. Combine that with the fact that the exchange is also highly transparent in terms of pricing and doesn’t mask a markup on the media like an ad network, and you can see why both advertisers and publishers are pretty excited about the ad exchange opportunity.

Of course, if your business is optimizing ad network spend, the exchange is actually a threat, since the exchange aims to do exactly the same thing as a network optimizer, but just on a larger scale and therefore, with greater efficiency. But unlike many of the ad networks out there, the network optimizers weren’t just an office of sales people, they had real technology, teams of developers, and knew how to build product. So instead of trying to push publishers away from the ad exchanges, they’ve for the most part embraced ad exchanges and real-time-buying (RTB) systems, and have worked to aggressively assimilate demand from the ad exchanges into their auction and optimization algorithms. RTB is certainly a growth area for digital advertising, but since ad networks still provide about 70 – 80% of the demand dollars out there most publishers can’t walk away from that source of revenue just yet.

So where does digital advertising go from here? Well, there are a few other issues lurking in the background where Supply Side Platforms will play a key role. The first issue is data leakage, where advertisers seek to data-mine publisher audiences though dropping their own cookies out of ad buys, or even using javascript tags to scrape the page content the ad serves on as well. In my experience, publishers drastically underestimate the risk of data leakage to their business, mostly because there aren’t many tools in place to help them manage the problem. Currently there are only blunt-instrument approaches, but the SSPs are hard at work building new tools to expose the problem to publishers and help them take on this issue head first. Also on the horizon is the concept of audience futures, or guaranteed RTB, which combines the targeting benefits of RTB with the placement and priority guarantees of a premium ad buy, and by the way, can bypass the ad exchange altogether. Data Management Platforms like Demdex and RedAril have already built a model for audience profiling systems, but it’s such a natural place for the Supply Side Platforms to go because it provides a new premium sales product to publishers and uses most of the same infrastructure as the ad exchange monetization plumbing they already provide.

Hard to say exactly where this will all land, but it should be interesting to find out!

Read the prior sections of this series:

Part I: Rise of the Ad Networks

Part II: Network Fragmentation and the Ad Ops Problem

Part III: Network Optimizers to the Rescue (?)

History of the Ad Exchange Landscape Part III: Network Optimizers to the Rescue (?)

May 2, 2011

Part III: Network Optimizers to the Rescue (?)

Network Optimizers weren’t created to stop the bleeding of publisher sales to network sales, they were built to solve the latency and user experience problem by figuring out which network to serve what ad and outsource the implementation and management process for publishers. They accomplished this feat with raw manpower to start, and then got their hands on some venture cap money to build out technology and algorithms to figure it out and scale their operations.

As I mentioned in Part I, no one ad network wanted all the impressions a publisher could provide, they just wanted a little, or up to a certain frequency, or within a certain geography, or during a certain time of day, or with some other specific characteristic. If you were a publisher you could setup this kind of targeting to point users with certain characteristics to certain networks, but after a certain level of scale, things started to fall apart. it was much easier to just traffic one redirect to a network optimizer, and let them figure out which network was most likely to take the impression. Usually publishers would pull a weekly report to figure this out for themselves, but a network optimizer with an API connection to the ad network’s reporting server could pull data every few minutes and start to learn what impression the network wanted, and which ones it didn’t. By figuring out which ad network was most willing to take the impression on the first try and not keep redirecting users through the so-called daisy-chain of ad calls, network optimizers could reduce latency from the ad server, improve user experience, and increase revenue to the publisher. Optimizers could also handle the operational hassles of managing an advertiser blocklist through multiple ad networks, enforcing ad quality guidelines to keep tobacco or alcohol advertising off a publisher’s site, and could even do the bill collecting if need be. It was like outsourcing an entire back office of Ad Operations, Reporting, and Billing team all at once for a small revenue share.

As a business strategy, this worked pretty well – major digital publishers signed contracts to manage billions of unsold impressions with network optimizers like Collective Media, Pubmatic, AdMeld, and Rubicon Project. It was a giant step forward for publishers, but another industry force was just starting to emerge that would present publishers with a new opportunity as well as new challenges: the ad exchange.

SSP to DSP Cookie Syncing Explained

May 1, 2011

The matching process of the SSP cookie ID to the DSP cookie ID happens through a parallel process to serving ads called cookie syncing. A cookie sync is necessary because as a standard security process, web servers of any kind can only request cookies that are set to their own domain. Since the SSP sits between the end-user and all the DSP bidders in a real-time auction however, the DSP needs a way to identify the users it is looking for.

Why Cookie Syncing is Necessary

So let’s take a simple example on a store trying to retarget their users. Let’s say that you run storeABC, and user123 drops in and adds a pair of $150 shoes to their shopping cart, but never makes it to the checkout. You want to retarget that user and serve them an ad directing them back to your store to try and close the sale. Since you work with DSP456, you have a 1×1 pixel sitting on your shopping cart page, which forces the user to call out to DSP456′s web server as they load the shopping cart page, giving DSP456 a chance to drop a cookie on user123. That cookie ID is DSPcookie789. Now, user123 is surfing around the web, and lands on awesomesite.com, which is using SSP123 to monetize their ad inventory. awesomesite.com serves a 3rd party redirect to SSP123, which drops a cookie on user123. That cookie ID is SSPcookieXYZ. SSP123 now requests a bid from DSP456 among other bidders for the impression that SSPcookieXYZ is about to view. But wait, how does DSP456 know that SSPcookieXYZ = DSPcookie789? On this first ad, it doesn’t, so your DSP doesn’t bid on the impression. Bummer. Remember, the SSP can only read and pass its own cookie ID to bidders.

Piggyback Scripts Power Cookie Syncs

After SSP123 selects a winning bidder though, it runs one last piece of javascript that forces user123 to call out to a handful of regular bidders, including DSP456. In that redirect, using a query string, the SSP passes its cookie ID on user123 (SSPcookieXYZ). Now the user IS calling DSP456′s web server and the DSP can request its own cookie from user123 in a process the industry refers to as “piggybacking”. Eureka! SSPcookieXYZ also has DSPcookie789 – it’s user123! The DSP knows the SSP’s cookie ID because of the query string in the piggyback call, and it can read its own cookie ID because that user called its web server as the end destination with the piggyback call. DSP456 now writes into its database that DSPcookie789 = SSPcookieXYZ for bid requests from SSP123. The next time user123 hits a page that SSP123 is helping to monetize, DSP456 will know it is the same user that abandoned their shopping cart and can bid appropriately. The process repeats for sites using other SSPs.

It sounds complex, but all it means is that for every ad served through RTB, about 10x as many technology companies are involved in the cookie-sync process. The reason the syncing process occurs after the auction happens and not before it is because of latency and user experience. If user123 had to call 10 DSPs, wait for those DSPs to cookie their machine, write the matching SSP id into their database, and then bid, it would dramatically slow down the entire auction process for everyone. If the cookie sync fails on the other hand, well there will be many more opportunities for that.

Cookie Syncing Step-by-Step

Below is a simplified diagram of the cookie sync process, where user123 visits the Marketer’s website first (1), is then redirected to the Marketer’s DSP (2), calls the Marketer’s DSP (3), receives the DSP’s cookie (4) and is simultaneously redirected to various SSPs that the DSP is cookie syncing with (5). When the user calls those SSPs, the call passes DSP456’s cookie ID on that user in a key value parameter. The process completes when the DSP456’s ID is logged in the match table for each SSP, and receives the SSP’s cookie in return.

As an interesting post-script to this, the SSP might redirect the user back to DSP if the DSP is hosting the match table instead of the SSP. This would be the same process as 4 / 5, just in the reverse order, with the SSP passing its ID to the DSP so the DSP can log both IDs in its own match table. Who hosts the match table is a commercial arrangement that DSPs and SSPs / Exchanges make with each other, with the match table host typically paid to shoulder the cost of maintaining the database.

Example Cookie Sync Scripts

If you are interested, here are the URL’s that run the piggyback script for each major SSP – these pages look blank, so you’ll have to ‘view source’ to see the code.
Pubmatic: http://ads.pubmatic.com/AdServer/js/syncuppixels.html It looks as though ‘vcode’ is the item passing the cookie ID.
Rubicon Project: http://tap2-cdn.rubiconproject.com/partner/scripts/rubicon/emily.html It looks as though ‘nid’ is the item passing the cookie ID.