The esteemed Matt Van Wagner wrote a very interesting article on Search Engine Land titled The PPC Experiment You Never Dare Run. He describes the PPC nuclear option of turning off all adverts and then observing the effect on overall revenue.
This does help answer the question "Why are we paying for this traffic? Aren’t we going to get that traffic anyway?" but a couple of problems spring to mind straight away (I'm sure Matt is aware of these):
Both of these problems were avoided by Ebay in their March 2013 study Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment. Which, in paid search circles, went down only a little bit better than a house on fire because of the poor impression it gives of the channel.
Regardless of whether or not Ebay use PPC well I think the methodology behind the study is excellent. Here are the basic ideas (details in section 4.1 of the linked document):
For Ebay this resulted in them stopping paid search activity in roughly 30% of regions which, to me, still seems like quite a high number; perhaps not the full nuclear option but definitely a dirty bomb or chemical attack on revenue.
There will always be some opportunity cost to running this type of experiment (so weigh the value of the knowledge gained carefully). But it is possible to run a similar experiment with lower risk.
The variable that determines the risk is the percentage of people who end up in and experiment area - if you only turn off advertising for a small mountain village then the opportunity cost is very low. This must be balanced with getting enough data to be confident about any conclusions.
Here are the levers we can pull:
AdWords makes it easy to prevent ads from running in selected areas so as far as I can tell these are the only reasons not to:
Point number four is probably the only one I can help with - look out for more information on time series clustering in the future.