Do They Really Want a Coffee Maker?

The center of any scientific endeavor typically features two smaller questions; a) are these two groups different and b) how would we know if any differences we observe are actually real? For example, if group A has a mean of 2.6 on a five point scale of interest in having a coffee maker in the office and group B has a mean of 3.2, we observe a difference.  But, how can we be sure that group B wants a coffee maker more than group A?  In other words, do we have enough statistical power to be sure about our conclusions?  How much data does one need?  The answer depends on the size of the effect we observe—smaller effects need more data points to illuminate stable group differences.

When conducting an experiment, for example, and wish to compare groups on a factor, such as gender, for example, you typically need at least 25 people per “cell.”  That is, 25 people in each level of the variable of interest. So for gender you’d need 25 men and 25 women.  If you want to compare men and women’s level of education (and there were four levels of education), then you’d need 200 respondents —4 (education) X 2 (gender) X 25. So when you introduce other variables into your model, it causes the required responses to rise.

There is a nifty power calculator here, courtesy of the University of Dusseldorf, for folks that want to check it out.  Tell us what you think.