--
-- 		--
Home | About Us | Services | Articles | Online Demos | Contact Us
Key Word Search:
Bibliography
Web-based Research
Trade-Off Analysis
Other Quant. Methods
General Interest
Business Ethics:
Where did
they go?
Exploiting Human
Nature: How to
Successfully
Market High-Tech
New Products
Three and a half
Steps to
Statistical Success
Why Market
Research Is a
Waste of Money
(And What You
Can Do About It)
McCullough's Laws:
First Principles
of Commercial
Data Analysis
Non-Business Articles

--

Bring Your Survey Design Out Of The Dark Ages 1

By Paul Richard McCullough

Download This Article

Take a questionnaire written last week and place it side by side with one written 20, 30 years ago. Chances are they will look identical. Same logic. Same skip patterns. Same batteries and scales. Same limitations.

Back in the day, quantitative market research meant cross-tab decks with 20 point banners. Back in the day, that was rocket science, state-of-the-art, leading edge. I wrote those surveys (and analyzed their data) with suspender-snapping pride. Problem is, we are no longer back in the day. Back in the day, corporate main frames didn’t have the computing power of today’s smallest laptops. Marketing scientists and other brainiacs have had the last 30 years to develop new analytic techniques to take advantage of all this computing power. These new and not-so-new-anymore methodologies are designed to eliminate many of the biases and inaccuracies of traditional surveys. They deliver answers to questions we didn’t even dare ask “back in the day”.

But the analytics are just the engine. They need fuel to run. And they need high octane fuel to run at their optimum. Antiquated survey designs yield very low octane fuel. They keep these high-powered engines from blowing past the competition and hitting that checkered flag first. Bad survey design turns your Ferrari into a Model T. And it happens every day.

There are three main problem areas in old school surveys:

  • Missing data - Don’t knows and skip patterns are the primary culprits here. Generally speaking, both are entirely unnecessary. And both are devastating to advanced analytics.

  • Collinearity - Any two questions that are highly correlated contain essentially the same information. That is, they are wasting survey real estate. Test virtually any survey data set and you’ll find collinearity of epidemic proportions. 100 questions with the information value of 10. Again, entirely unnecessary.

  • Direct questions - Did you buy that sports car because you want to attract women (Yes/No)? Did you buy my product because of the ad you just saw (Yes/No)? You can bury these types of questions in a check all that apply battery (or whatever else) but you’re just putting a dress on a pig. Respondents will answer any question you ask them. But they won’t necessarily answer truthfully. Sometimes they don’t know. Sometimes they don’t want you to know. Advanced analytics can ferret out the truth that respondents may not want or may not be able to share. But you have to ask the questions differently.

All of these problem areas can be corrected in the survey design, even if you’re designing a paper-and-pencil survey, if you understand how modern analytic techniques work.

I’ve written a paper discussing common problems in survey designs and how to fix them so that you can take full advantage of the information you’ve paid for. If you’d like a copy of that paper, just send me an email and I’ll forward a copy to you.

1 A version of this Newsletter was sent to our subscription list in January, 2010.

MACRO CONSULTING
Intelligent Decisions