Gambling on Ad Tests

Posted on February 26, 2012

“Set ad serving to rotate” is a regularly heard piece of AdWords advice. I do not use this setting for all campaigns and in this post I hope to explain why.

Imagine that you are in a casino and that you want to make money as quickly as possible playing the bandits. Also assume that you are in the kind of imaginary casino where it is actually possible to make money playing the slots. There are 10 different machines you could play, what is your strategy?

To begin with you have no data on any of the machines so you will have to play all of them at least once. After that strategies get more complicated. Your strategy must decide between expoiting the machine that you think has the best payout and exploring the other machines in case the machine you think is the best actually isn’t

Here are some example strategies:

  1. Pick the first machine in line and only play that machine. Obviously flawed in that no exploration is done at all.
  2. Play each machine once and then play the machine with the highest payout. Flawed in that the machine you think pays out the highest may only seem the best because of random chance.
  3. Play each machine n times and then play the machine with the highest average payout. Slightly better. But if n is too low then this strategy suffers from the same weakness as 2. If n is too high then you have wasted money exploring when you should have been exploiting.
  4. Play each machine equally until one machine has reached “statistical significance” and then play only that machine. It is difficult to monitor an ongoing test without ruining the test from a statistical standpoint. Basically you should be fixing the sample size in advance which makes this strategy the same as 3. If you don’t, either your test is not as significant as you thought (pretend for this discussion that this is a DISASTER) or you have avoided repeated significance testing errors but your test has run for longer than it could have done meaning you spent too much time exploring and not enough time exploiting.

There is a better way. Statisticians and decision theorists have come up with many strategies to beat the “multi armed bandit” problem. The wikipedia article on the topic has a good overview.

Enough gambling. Back to PPC

You probably know where I’m going with this - an split test as run by most people is strategy 3 or 4 from above but to make the most money one of the bandit optimization strategies should be used. When you choose “Optimize for Clicks” or “Optimize for Conversions” Google uses their own bandit optimization strategy on your behalf. Personally, I think their algorithm is too conservative when it comes to giving impressions to a new ad, but I still believe that they make my clients more money than even rotation.

My Ad Serving Strategy

  • Head campaigns are run on even rotation ad serving. Volumes are high enough and stable enough here that I can run valid tests quickly. I may lose a bit of revenue by not using a more perfect strategy but I gain in knowledge. Split testing allows for testing of hypotheses which is not possible with bandit optimization.
  • Body/Tail campaigns are run on “Optimize for Conversions”. I am not really testing here (but maybe I should be), I just want to make money. Results and knowledge about the audience from the head campaign experiments filter down and are applied here.

This seems to be striking a good balance between exploration, exploitation and making everything to damn complicated. I hope it has given you some ideas.

Thanks to Untyped for inspiring this post with their article Stop A/B Testing and Make Out Like a Bandit