To Err is to do Statistics: Types of Statistical Errors

Raise your hand if you have ever made a big mistake? Sometimes mistakes can hurt, for example, as anyone who has misinterpreted signs of affection on a date would know. Other times, we mess up by passing on golden opportunities, saying no to something we will regret later.

When we perform a statistical test, we should not expect an answer with 100% certainty. When we compare two variables from two different groups, we are testing whether we are sampling from two separate populations.

Types of Statistical Errors

Depending on chance and the size of our samples, we could miss a real difference or find a difference when the two samples come from identical populations. Here is a table that will help you define the different types of statistical errors.

The False Positive error is also called the alpha-error or Type I error. Sometimes, the null hypothesis is rejected, even though it should not. The opposite is called a Type II error or a false negative result. Let’s look at a visual to see what it would look like when comparing two different distributions.

Proofing Against Error

When conducting experiments, it goes without saying that we would prefer to get a true positive or true negative. To improve the power of your statistics, make sure that you have enough participants or measures within your dataset. The more values that you sample from your population, the more representative the sample.

Another way to ensure you are not making errors is to make sure that your data fits your assumptions. Are your measurements continuous and distributed across a normal distribution? Have you made sure that your experimental measurements and scales are reliable?

You can also rerun your experiment or have someone in another laboratory or location attempt to replicate from your methods. Science and social science are built on consensus, where different researchers replicate each other’s results. After all, you can make an error when you are sampling from a population once. But it is unlikely to make the same error ten other times in a row.   

Why You Should Care About Error

Many different medical tests determine if someone has a disease or not; just think about coronavirus testing. The rate of false positives and the rate of false negatives, tell us about the test’s reliability.

Sensitivity describes how well our test catches all positive cases. Sensitivity is calculated by dividing the number of true-positive results by the total number of positives (which include false positives).

Specificity describes how well our test classifies negative cases as negatives. Specificity is calculated by dividing the number of true-negative results by the total number of negatives (which include false negatives). We want a test with high levels of sensitivity and specificity for a proper diagnosis, a high positive-predictive value.