T-Test

The first statistical test we learn about sounds like a pleasant, warming drink. Unfortunately, the T-test bares very little semblance to actual teas. Instead, it gives us a very easy way to see if there are differences between the means of two different groups. Advertising agencies and small business owners are also familiar with the T-test.

To optimize marketing strategies, many data analysts might run an A/B test.  Everyone who sees an advertisement for the business or product with either see Version A or Version B. Then, the number of clicks on each advertisement can be recorded over the course of a month.

The T-Test then helps identify whether one advertisement outperformed another. A null hypothesis in this situation would ask whether these differences occurred by random chance.

Let’s outline the assumptions and requirements for running the two different types of T-tests.

Independent samples

• Assumption of independence

You need two independent categorical groups. This might be Advertisement A vs Advertisement B, Male vs Female or Kids vs Adults.

2. Assumption of normality

The dependent variable should approximate a normal distribution. Luckily, we don’t need to know anything about the population or its mean/standard deviation. Here, a dependent variable could be clicks per day on an advertisement.

3. Assumption of homogeneity of variance

The variances of the dependent variable should be approximately equal.

When you conduct this test, you’ll look up an alpha-level, the acceptable Type I error for this test (often 0.05). You can also do a directional T-test where your null hypothesis is simply, the mean of Group B is not lower/higher than the mean of Group A.

If your significance value is greater than your alpha value, that means that these results are likely to have occurred through chance.

Repeated measures or paired samples

In this t-test, there are two dependent measures of our dependent variable, taken from each sample. We could look at the effects of a certain intervention, taking measurements, a before and after condition for each of our samples.

If we are looking at the impact of a study aid, we could compare test scores before using the study aid and test scores afterwards. Since we take measurements from every single student, we reduce the levels of bias attributed to interpersonal variation.

The rest of the test is similar enough – you decide if you want to do a single-tailed/directional test or not. Then you pick you alpha level and if your results are greater than this alpha, your results are not more likely than random chance.

Takeaway

If you’re dealing with continuous variables and two comparison groups, the T-test is what you’ll want to use. Make sure that you check the data doesn’t violate any assumption of the T-test. Most of all, remember that you’re always testing a null hypothesis. Even if you can’t reject the null hypothesis, it doesn’t mean that a population-level difference does not exist.