Determining the true value of paid search

Posted on October 2, 2013

1 The experiment you never dare run

The esteemed Matt Van Wagner wrote a very interesting article on Search Engine Land titled The PPC Experiment You Never Dare Run. He describes the PPC nuclear option of turning off all adverts and then observing the effect on overall revenue.

This does help answer the question “Why are we paying for this traffic? Aren’t we going to get that traffic anyway?” but a couple of problems spring to mind straight away (I’m sure Matt is aware of these):

  1. The nuclear option is not a good one to take; there is a massive opportunity cost involved in figuring stuff out this way.
  2. It is not a true experiment because some external event forced the change: “Campaigns that had been running for a few years were taken offline abruptly three months ago”. This means that the results of the experiment could be confounded by whatever external factor caused the campaigns to go offline in the first place. For example, some companies cut marketing budgets during a recession and restore them afterwards; in this case we would see site revenue drop whilst PPC was turned off even if PPC had no overall impact on the site.

2 Ebay and paid search effectiveness

Both of these problems were avoided by Ebay in their March 2013 study Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment. Which, in paid search circles, went down only a little bit better than a house on fire because of the poor impression it gives of the channel.

Regardless of whether or not Ebay use PPC well I think the methodology behind the study is excellent. Here are the basic ideas (details in section 4.1 of the linked document):

  1. Divide the target geography into separate areas. Ebay used the Neilsen Designated Market Areas for this.
  2. Randomly select a subset of these areas to be candidates for the test. This is done to reduce risk and make this less of a nuclear option.
  3. Out of the candidates pair up the areas based on how similar they are.
  4. Randomly select one from each pair to be the control and one from each pair to be the experiment.
  5. Stop all advertising in the experiment areas.
  6. Compare the results for these areas with the controls.

For Ebay this resulted in them stopping paid search activity in roughly 30% of regions which, to me, still seems like quite a high number; perhaps not the full nuclear option but definitely a dirty bomb or chemical attack on revenue.

3 Can we get the rigour without the opportunity cost?

There will always be some opportunity cost to running this type of experiment (so weigh the value of the knowledge gained carefully). But it is possible to run a similar experiment with lower risk.

The variable that determines the risk is the percentage of people who end up in and experiment area - if you only turn off advertising for a small mountain village then the opportunity cost is very low. This must be balanced with getting enough data to be confident about any conclusions.

Here are the levers we can pull:

  • We control the number of candidates for the experiment. Increasing this number increases the exposure to risk and decreasing it reduces the opportunity cost of the experiment.
  • Instead of pairing up candidate areas the candidates can be partitioned into larger groups from which only one is assigned to the experiment. If candidates are grouped into fives than only 20% of the candidates end up in the experimental group. This method also has some extra robustness built in as during the experiment it is possible to check that the control areas are still similar.

4 The experiment you should consider running?

AdWords makes it easy to prevent ads from running in selected areas so as far as I can tell these are the only reasons not to:

  1. There are more important things to do.
  2. You are confident in the value of paid search.
  3. The risk of losing even a small amount of revenue is too much
  4. Partitioning the areas is hard

Point number four is probably the only one I can help with - look out for more information on time series clustering in the future.