--
-- 		--
Home | About Us | Services | Articles | Online Demos | Contact Us
Key Word Search:
Bibliography
Web-based Research
Trade-Off Analysis
Other Quant. Methods
General Interest
Non-Business Articles

--

TRADE-OFF STUDY SAMPLE SIZE: HOW LOW CAN WE GO?[1]

By Dick McCullough

Download This Article

Abstract
The effect of sample size on model error is examined through several commercial data sets, using five trade-off techniques: ACA, ACA/HB, CVA, HB-Reg and CBC/HB. Using the total sample to generate surrogate holdout cards, numerous subsamples are drawn, utilities estimated and model results compared to the total sample model. Latent class analysis is used to model the effect of sample size, number of parameters and number of tasks on model error.

Introduction Effect of sample size on study precision is always an issue to commercial market researchers. Sample size is generally the single largest out-of-pocket cost component of a commercial study. Determining the minimum acceptable sample size plays an important role in the design of an efficient commercial study.

For simple statistical measures, such as confidence intervals around proportions estimates, the effect of sample size on error is well know (see Figure 1). For more complex statistical processes, such as conjoint models, the effect of sample size on error is much more difficult to estimate. Even the definition of error is open to several interpretations.

Figure 1.


Many issues face practitioners when determining sample size:

  • Research objectives
  • Technique
  • Number of attributes and levels
  • Number of tasks
  • Expected heterogeneity
  • Value of the information
  • Cost and timing
  • Measurement error
  • Structure and efficiency of experimental design:
    • Fixed designs
    • Blocked designs
    • Individual-level designs
Some of these issues are statistical in nature, such as number of attributes and levels, and some of these issues are managerial in nature, such as value of the information, cost and timing. The commercial researcher needs to address both types of issues when determining sample size.

Objectives
The intent of this paper is to examine a variety of commercial data sets in an empirical way to see if some comments can be made about the effect of sample size on model error. Additionally, the impact of several factors: number of attributes and levels, number of tasks and trade-off technique, on model error will also be investigated.

Method
For each of five trade-off techniques, ACA, ACA/HB, CVA, HB-Reg, and CBC/HB, three commercial data sets were examined (the data sets for ACA, and CVA also served as the data sets for ACA/HB and HBReg, respectively). Sample size for each data set ranged between 431 and 2,400.

Since these data sets were collected from a variety of commercial marketing research firms, there was little control over the number of attributes and levels or the number of tasks. Thus, while there was some variation in these attributes, there was less experimental control than would be desired, particularly with respect to trade-off technique.

[1]       The author wishes to thank Rich Johnson for his invaluable suggestions and guidance during the preparation of this paper.  The author also thanks POPULUS and The Analytic Helpline, Inc. for generously sharing commercial data sets used in the analysis.

Next Page

MACRO CONSULTING
Intelligent Decisions