Beat Your Control by Testing the Big Things
Saturday, April 25, 2009 at 01:40PM When you test direct mail, for example, test the big things rather than tweaks.
“Tweak testing” pervades the risk adverse environments in many industries. This dramatically reduces their chances of competing with those firms that understand the need to compete aggressively with their testing programs.
Why? Because breakthroughs of 25% plus in costs per sale that displace controls do not happen with small changes. Big changes in results require big differences in your package tests.
The testing priorities are clear. The priority sequence is as follows. 
Typical direct mail test grid
1. The list
2. The offer
3. The format
4. The main selling proposition
So don’t try to structure the test so that you are testing paper stocks, envelope sizes, first class versus 3rd class and so forth. These all impact response to some degree, but by the time you figure it all out everything will have changed. So concentrate on the majors.
Besides, you have limited staff resources for analyzing test results as well as fixed test budgets. Spend your assets on the things that make a significant difference.
When testing offers, don’t try to make the two packages by forcing the same package contents or format. Test packages, not package elements. Every package possesses its own unique character. After all, you are looking for a breakthrough, not an analytical report that has too many variables to make sense out of anyway.
So in direct mail, test packages against each other and not package elements against each other. Your chances of beating your controls are greatly enhanced through this approach.