statistical power analysis grows more and more popular... if the result of an analysis is - contrary to predictions - not statistically significant. but if it is significant there also seems to be something to worry about.
a reason why many (post-hoc) power analyses are conducted is the lingering hope that it "may have been significant had i only had more participants in the sample". this phenomenon points quite directly to a reason why power analysis is important and even should be done
before collecting data. if i know the effect to be expected is small, i should plan to have more participants beforehand as otherwise, assuming a fixed alpha-level, the beta-error (the probability of failure to find a significant difference even though in the porpulation there really is a difference) may "explode", become very large, so that i won't find the effect that is actually there.
but if i do find a significant effect (without worrying about power, which, if i find that asterisk, indeed seems to be a quite rare concern), there's also a "power problem": if systematic power analysis is omitted, the effect i may have found could be very small while the beta error is huge. "so what" could one say, "i still found the effect against the odds of the power. small effects, especially in basic research, are also interesting and may have a huge effect down the causal stream of some intricate process."
but there is one more problem arising where i find a significant difference with small N: originally, a small alpha is preferrable, as the smaller it is, the less i am confident that the empirical results "comes" from the null distribution. confidence of course "should not" increase linearily with decreasing alpha/p-value, but there should still be a monotonic relationship: if the likelihood that a test statistic (e. g. a t-value) from empirical data like that found or larger stems from a null distribution is "so small", i don't believe it comes from that null distribution anymore. but at the same time, i decide for the alternative hypothesis, that "there really is a difference". i do that believing that the likelihood of finding a value equal or larger to the empirical value of the test statistic under the assumption that there really is a difference is larger than the likelihood of it occuring if it comes from the null distribution. so, power is not only the probability of finding an effect if there really is one, but it is also something that should exceed alpha, if confidence in a decision based on a test statistic is desirable.
back to the situation: a significant small effect is found with a small N and, unbeknowst to the researcher, beta is huge and power is miniscule. i simply shouldn't be that confident that there really is a difference to the extent a small alpha by itself sometimes seems to convey: the likelihood that i find a t-value that large or larger under assumption of the null hypothesis is not very much higher than that to find the same empirical result under assumption of the alternative distribution.
an example: assuming a t-test with n1=n2=17, an effect found d=.1 (small according to Cohen's conventions) and an p-value of .049 (significant, yipieh!), power is .0855. if i had to bet money on whether that empirical result comes from the distribution suggested by the effect (alternative distribution) rather than from the null, i wouldn't put to much money on it. i would be a lot easier with the money to bet if n1=n2=266 and d still equal to .1, as then the probability of a t-value equal to or larger than that associated with p = .049 occurring under the null would be sustantially smaller (p = L(null) = .049) then under assumption of the alternative distribution: power = L(alt) = .20. for a basic-research-in-psychology-vanilla-ice two sample t-test, that is a huge N. and power is not so impressive either, given that we usually ask for power levels in the order of .80. and what's even more disturbing: an even smaller p value does not resolve the problem in a big way: with n1=n2=266 and an effect of d=.1 associated with p = .001, power is .0261. granted, it is more than 26 times the size of p. but would you bet money on an alternative hypothesis under which the likelihood of a test statistic equal or larger than that which you found is 2%? if someone told you she has two urns, one with 37 blue balls and one red ball, and another one with 999 blue balls and one red ball. she has drawn from one of the urns a red ball. surely if you had to bet a fixed amount of money, it's wiser to bet the ball comes from the first, but not the second urn. but how confident would you be? a small alpha does not indicate per se a "strong effect" and confidence that there really is a difference should be limited - one more reason to not only look at the little asterisks in the output and simply think smaller is better.
