Kijun Times

I regularly teach courses in statistics. And make no mistake about it, I LOVE teaching statistics. I’ve taught this class since 1996, have (with Sara Hall) co-authored a textbook on the topic (link is external), and I’d say that as much as 50% of my mental energy in a given day may be spent thinking about a variety of things in terms of statistics. I breathe this stuff!

Pretty much all psychology majors around the world have to take at least one basic statistics class – so as behavioral scientists, it’s pretty important that we get this stuff right when we teach it.

As a teacher of technical information, I am constantly working to make the concepts as intuitive and accessible as possible. And on this point, I have a nit to pick with traditional presentations of statistics. It has to do with hypothesis testing.

In “hypothesis testing,” we set up a test to see if some pattern found in a sample of data is likely to generalize to a broader population of interest. For instance, suppose that you wanted to see if border collies score higher on some measure of doggie intelligence compared with labs (just saying!). So you get five border collies and five labs and you test their canine IQs. And you find that the average for the border collie sample is 150 while the average for the lab sample is 99.

Now there are a few things that could be the case here. One is that your research hypothesis simply could be correct. Maybe the entire population of border collies is smarter than the entire population of labs. This would be, in hypothesis-testing parlance, a “correct decision.” However, you may actually be wrong. It may actually be the case that you happened to have (for no good reason) some particularly bright border collies and some particularly dull labs in your samples. In fact, at the population levels, the average IQs might be equal for these breeds. If this is actually how things are at the population level, but your results of your hypothesis test from your sample show (mistakenly, in this case) that there IS a difference, then you have made “Type-I Error” – an error  typified by essentially finding something that is not really there.

There are more possibilities out there. For instance, imagine a slightly different scenario in which the findings from your sample came out such that the mean IQ for your five border collies is 100 and the mean IQ for the sample of five labs is also 100. So you find no difference. BUT imagine that, in reality, the IQ of border collies, at the level of the full population, is higher than is the IQ of labs (at the full population). In this case, you would have made what we call a “Type-II Error” which is when you fail to find something that is really there.

Calling Things What They Are

As a teacher, I always tell my students that technical terms usually mean what they say. For instance, “natural selection” is simply the idea that some qualities are selected to exist by nature.

This said, “Type-I Error” and “Type-II Error” are clearly arbitrary labels! These names do not give the reader an intuitive sense of what you’re talking about! On many occasions, I have had students understand the concepts at hand, but get test items wrong, claiming (probably accurately) that they got the terms confused with one another.

Let’s Rename Type-I and Type-II Error

So here’s a radical proposal. And I may well be working uphill and alone on this one … but what if behavioral scientists worked together to change what we call these things?

Here’s my proposal: How about if we call Type-I Error something like the “Found Fool’s Gold” Error. This term makes it clear not only that an error was made, but it’s of the variety such that you report that you found a statistically significant effect even though, in actuality, you were wrong. Fool’s gold - this is a possibility. Some folks (such as my departmental colleague, Alison Nash) have suggested to me that we shift to calling this the relatively meaningful "false positive" - to them I say this, "that's a lot better than Type-I Error!"

And how about if we refer to Type-II Error simply as something like the “Failed to Find Something that's Actually Real” Error. This term simply reflects that there was something there to be found, but in your hypothesis-testing scenario, you failed to actually find it. Some folks have suggested that we could systemically call this one a "false negative" - and compared with "Type-II Error," I'd say this is much better.

The particular terms that we end up with may well be up for grabs - but, of course, my main point is this: Let's work as a community to change to something new!

Helping Statistics Students Understand the Concepts

As a statistics teacher, I’m always looking for ways to make the concepts in the course most accessible. And based on years of experience, I’ve found that Type-I and Type-II Error are the kinds of concepts that students typically can understand – but they often get tripped up with the terminology. If we can make some element of statistics easier for students to understand, maybe we should! In the next edition of my statistics book (which could take years to realize!), I’m going to introduce formally some new terms for these concepts – let’s see if it catches on!