"Test everything" is a PPC and SEO truism that everyone says is important but that no one actually does. In PPC at least most people test some things (I'm not sure if this can be said of most SEOs) but anyone who sets out to test everything is on the fast track to analysis paralysis. So if no one is testing everything then what are they testing? Based on my experience the following list comes close:
Basically, people test things where the cost of testing is low or where the expected impact of new knowledge is high.
What exactly do I mean by the expected impact of new knowledge? I will try and explain with an example.
I am extremely confident that separating the search and content networks into different campaigns is a good thing. Learning otherwise wouldn't make a big difference to how I manage accounts because I'd want to separate these two impression types anyway for reporting purposes. So if I were to test this out I'd be running a test that will almost certainly confirm what I already know and in the unlikely event of a combined campaign performing equally well it wouldn't change any of my actions. This is an example of knowledge with low impact.
Alternatively consider using the conversion optimiser bid management tool. I have had mixed results with this in the past, sometimes it works well and sometimes it completely bombs. So I am much less certain about the likely outcome of any test. In the cases where the conversion optimiser works it saves a lot of time and can increase the number of conversions. My initial levels of uncertainty combined with the high impact outcome make this test worth running; if I don't test then I could make a bad decision.
If you can correctly guess the outcome of a coin flip then I will pay you $1. You also have the opportunity to perform a simple test; pay my assistant $0.75 and she will tell you (with 100% accuracy - no funny business) which way the coin landed.
Without testing your expected return is $0.50 per flip. With the test, you'll get the answer right each time but you can only expect to earn $0.25 each flip.
In this case the test costs too much given the value of the information it provides. In a situation where you have a lower probability of being correct then the information is relatively more valuable. For example if the setup was exactly the same except you had to tell me the score from rolling a six sided dice then paying my assistant $0.75 would be worthwhile.
For another example suppose I ask you to tell me whether or not the value of a dice roll is greater than one. My assistant offers to tell you whether or not the number is even. How much should you pay for this information?
Without any extra knowledge, you will say that the score is greater than one and expect to earn $(5/6)=$0.83 per turn.
If you know the number is even then you know the number is certain to be greater than one. If the number is odd then it is greater than one with probability 2/3 (because five and three are the only odd dice numbers greater than one). So either way, the rational thing to do is to guess that the number is greater than one. Hence my assistant's information is valueless to you and you should not pay for it.
Probabilities in the real world are never so easy to calculate as in the examples. Often it will be impossible (or the opportunity cost will be too high). So instead I offer the following rules for when not to test:
All of these concepts could do with being more precisely defined. Perhaps I will attempt that in a future post.