Some Misconceptions in Statistical Hypothesis Testing

In this short paper we summarize and elaborate on a few well-known misconceptions that often occur when people conduct statistical hypothesis testing. For example, when the truthfulness of the null hypothesis is a random event and the formal hypothesis testing procedure cannot reject the null, then the chance that the null is really true may be substantially lower than what most people would expect. Such a "distortion'' has to do with the structure of type I error and type II error we use as a basis in designing the testing procedure. We also discuss the implications of the asymmetric treatment of the two types of errors, the effects of sample size on testing results, etc., that are often overlooked by students and researchers when reporting their empirical results.