Committing Errors in Hypothesis Testing

When testing hypotheses with sample data we always run the risk of committing an error.  Whether we choose to reject the null hypothesis (conclude there is a relationship) or fail to reject the null hypothesis (conclude there is not a relationship), there is always a chance that we could be wrong because we are only working with a sample, not the population.

Measures of significance only give us an educated guess about the likelihood of a relationship in the population, and we can never be 100% certain about our conclusion unless we are actually dealing with population data.

When making conclusions about our null hypothesis, there are four different possible outcomes, depending upon what our conclusion is, and ultimately what the "truth" is:

and HØ for the population is really: If our conclusion is:
Reject HØ Fail to reject HØ
True We've committed a Type I Error No Error
False No Error We've committed a Type II Error
Reject HØ = Conclude that there really is a relationship.
Fail to Reject HØ = Conclude that there really is no relationship.

The trick is, of course, that we don't really know what the "truth" is.  That's why we deal with probablities when we look at measures of significance.  Recall that the probability value given with these measures represents the probability that the null hypothesis (HØ) is true.

So, if we reject the null hypothesis, the probability value also represents the probability that we've committed a Type I Error.