Imagine you read a far-fetched news headline about a recently published study. The headline reads: “Study Shows Running in Circles Prevents Cancer”. You’d probably scratch your head in disbelief.
Then you read on to discover the sample size was five people. And of those five people who ran in circles, one person didn’t get cancer. So was this coincidence or does the study indicate that running in circles prevents cancer? Probably the former.
The study for such a conclusion doesn’t have statistical significance — though the study was indeed performed, its conclusions don’t exactly mean anything because the sample size was so incredibly small.
This Kissmetrics guide will take you through everything you need to know about statistical significance and how to calculate it.
What is Statistical Significance and How Is It Calculated?
As we mentioned above, the fake study about spinning in circles isn’t statistically significant. This means that the conclusion reached isn’t valid because there’s simply not enough evidence to support that what happened was not random chance or due to luck.
A statistically significant result occurs when you reach a certain degree of confidence in the results after rigorous testing; we call that degree of confidence the confidence level, which demonstrates how certain we are that our data was not skewed by random chance or luck.
More specifically, the confidence level is the likelihood that an interval will hold values for the parameter we’re testing.
Statistical significance helps quantify whether the result is likely due to chance or some factor of interest. When a new finding is significant, it means you can feel confident that it’s real — not that you just got lucky (or unlucky) in choosing that specific sample.
Calculating statistical significance accurately by hand can be a pretty complicated task that requires a solid understanding of calculus and statistics. When you calculate by hand, however, it’ll help you more fully to understand the concept.
Follow these steps for calculating statistical significance:
- Create a null hypothesis.
The very first step in calculating statistical significance is to determine your null hypothesis. This should state that there’s no significant difference between the sets of data you are using. Keep in mind that you don’t need to believe the null hypothesis.
- Create an alternative hypothesis.
Next, you’ll need to create an alternative hypothesis. Typically, this is the opposite of your null hypothesis since it will state that there is, in fact, a statistically significant relationship between your data sets.
- Determine the significance level.
The next step involves determining the significance level, or rather, the alpha. This refers to the likelihood of rejecting your null hypothesis even when it is true. A common alpha is five percent (i.e., .05).
- Decide on the type of test you will use.
Once you’ve created both hypotheses and have determined the significance level, you’ll need to determine if you’ll use a one-tailed test or a two-tailed test. Whereas the vital area of distribution is one-sided in a one-tailed test, it is two-sided in a two-tailed test. In simpler terms, one-tailed tests analyze the relationship between two variables in one direction, while two-tailed tests analyze the relationship between two variables in two directions. If the sample you happen to be using lands within the one-sided critical area, the alternative hypothesis is considered to be true.
- Perform a power analysis to find out your sample size.
Next, you will need to do a quick power analysis to determine your sample size. This involves the sample size, effect size, statistical power, as well as significance level. For this important step, it might be best to use a calculator. You’re going to see how big a sample size you’ll need in order to learn the effect of a given test within a degree of confidence. In other words, it will let you know what sample size is best to determine statistical significance. If your sample size ends up being a bit too small, it won’t give you the accurate result you’re looking for.
- Calculate the standard deviation.
Now it’s time to calculate the standard deviation. To do this, you will need to use the following formula:
standard deviation = √((∑|x−μ|^ 2) / (N-1))
Where:
∑ = the sum of the data
x = individual data
μ = the data’s mean for each group
N = the total sample
- Use the standard error formula.
Next, you’ll need to use the standard error formula. For our purposes, let’s just say that you have two standard deviations for your two groups. The standard error formula would look as follows:
standard error = √((s1/N1) + (s2/N2))
Where:
s1 = the standard deviation of the first group
N1 = group one’s sample size
s2 = the standard deviation of the second group
N2 = group two’s sample size
- Determine the t-score.
For your next step, you will need to locate the t-score. Here is the equation:
t = ((µ1–µ2) / (sd))
Where:
t = the t-score
µ1 = group one’s average
µ2 = group two’s average
sd = standard error
- Find the degrees of freedom.
Once you’ve determined the t-score, it’s time to find the degrees of freedom:
degrees of freedom = (s1 + s2) – 2
Where:
s1 = samples of group 1
s2 = samples of group 2
- Lastly, use a t-table.
Once you’ve completed all the steps listed above, you’ll calculate the statistical significance using a t-table. Begin by looking at the left side of your degrees of freedom and find your variance. Then, go upward to see the p-values. Compare the p-value to the significance level (aka, the alpha).
Keep in mind that a p-value less than 5 percent is considered statistically significant — this is where you’ll often see the famous (p=.05) or (p=.01) in whatever study you’re looking at.
Don’t worry, you can easily determine the statistical significance of experiments — without any math (or headaches) — using an analytics tool like Kissmetrics.
Where Does Statistical Significance Factor Into Data Analysis, and How Can It Help My Company?
Statistical significance plays a huge factor in data analysis because brands use it to understand how strongly the results of an experiment, poll, or survey they’ve conducted should influence the decisions they make.
To give you a quick example, if a marketing manager runs a pricing study to understand how best to price a new product or service, she will calculate the statistical significance so that she knows whether the findings should affect the final price.
Statistical significance can help you and your company immensely — and it’s pretty easy to see why. Think about it: would you rather make a critical business decision based on a study with a small sample size, or would you prefer to make that same crucial decision based on accurate statistics and testing?
At the end of the day, data is not valid if it’s not statistically significant.
Real Life Example: Trying to Determine the Correlation Between a New Ad Campaign and Sales
Now that you understand exactly what statistical significance is and how to calculate it, let’s take a look at a real-life example:
Let’s say you want to increase sales by attracting more customers to your growing business — so, you decide to run a new ad campaign. In doing so, you take into account how many advertisements should be made in print, and many should be made digitally.
You rely on your past marketing campaigns using ads to forecast how many you will need of each. If you determine that your p-value is about 5%, you will end up with a result that is not statistically significant. This means that there is a greater than 5% chance that the relationship between the two ads was left up to chance.
Therefore, this result would indicate that it’s not reasonable to use the previous ad campaign as a guide to drive in new sales.
The Takeaway
Statistical significance is an extremely useful tool for marketers to validate their insights and provides credibility to their research. Although calculating statistical significance by hand is cumbersome, there are some incredible analytic tools like Kissmetrics that can help.
With the click of a button, Kissmetrics can help managers and researchers alike have confidence in their business decisions by providing them with the tools they need to run accurate tests and gain valuable insight into product and marketing campaigns.
Statistical significance is a powerful tool — are you using it?
Check out Kissmetrics today to add the power of p-values to your arsenal to increase your product’s ROI.
Sources:
