Analytics and data science
A Refresher on Statistical Significance
by Amy Gallo
February 16, 2016
When you run an experiment or analyze data, you want to know if your findings are significant. But business relevance (i.e., practical significance) isnt always the same thing as confidence that a result isnt due purely to chance (i.e., statistical significance). This is an important distinction;unfortunately,statistical significance is often misunderstood and misused in organizations today. And yet because more and more companies are relying on data to make critical business decisions,its an essential concept for managers to understand.
To better understand what statistical significance really means, I talked with Tom Redman, author ofData Driven: Profiting from Your Most Important Business Asset.He also advises organizations on their data and data quality programs.
What is statistical significance?
Statistical significance helps quantify whether a result is likely due to chance or to some factor of interest, says Redman. When a finding is significant, it simply means you can feel confident thats it real, not that you just got lucky (or unlucky) in choosing the sample.
When you run an experiment, conduct a survey, take a poll, or analyze aset of data, youre taking a sample of some population of interest, not looking at every single data point that you possibly can. Consider the example of a marketing campaign. Youve come up with a new concept and you want to see if it works better than your current one. You cant show it to every single target customer, of course, so you choose a sample group.
When you run the results, you find thatthose who saw the new campaign spent $10.17 on average,more than the $8.41 those who sawthe old one spent. This $1.76 might seem like a big and perhaps important difference. But in reality you may have been unlucky, drawing a sample of people who do not represent the larger population; in fact, maybe there was no difference between the two campaigns and their influence on consumers purchasing behaviors. This is called a sampling error, something you must contend with in any test that does not include the entire population of interest.
Redman notes that there are two main contributors to sampling error: the size of the sample and the variation in the underlying population. Sample size may be intuitive enough. Think about flipping a coin five times versusflipping it 500 times. The more times you flip, the less likely youll end up with a great majority of heads. The same is true ofstatistical significance: with bigger sample sizes, youre less likely to get results that reflect randomness. All else being equal, youll feel more comfortable in the accuracy of the campaigns $1.76 difference if you showed the new one to 1,000 people rather than just 25. Of course, showing the campaign to more people costs more, so you have to balance the need for a larger sample size with your budget.
Variation is a little trickier to understand, but Redman insists that developing a sense for itis critical for all managers who use data. Consider the images below. Each expresses a different possible distribution of customer purchases under Campaign A. In the chart on the left (with less variation), most people spend roughly the same amount of dollars. Some people spend a few dollars more or less, but if you pick a customer at random, chances are pretty good that theyll be pretty close to the average. Soits less likely that youll select a sample that looks vastly different from the total population, which means you can be relativelyconfident in your results.
Compare that to the chart on the right (with more variation). Here, people vary more widely in how much they spend. The average is still the same, but quite a few people spend more or less. If you pick a customer at random, chances are higher that they are pretty far from the average. So if you select a sample from a more varied population, you cant be as confident in your results.
To summarize, the important thing to understand is that the greater the variation in the underlying population, the larger the sampling error.
Redman advises that you should plot your data and make pictures like these when you analyze the data. The graphs will help you get a feel for variation, the sampling error, and, in turn, the statistical significance.
No matter what youre studying, the process for evaluating significance is the same. You start by stating a null hypothesis, often a straw man that youre trying to disprove. In the above experiment about the marketing campaign, the null hypothesis might be On average, customers dont prefer our new campaign tothe old one. Before you begin, you should also state an alternative hypothesis, such as On average, customers prefer the new one, and a target significance level. The significance level is an expression of how rare your results are, under the assumption that the null hypothesis is true. It is usually expressed as a p-value, and the lower the p-value, the less likely the results are due purely to chance.
Setting a target and interpreting p-values can be dauntingly complex. Redman says it depends a lot on what you are analyzing. If youre searching for the Higgs boson, you probably want an extremely low p-value, maybe 0.00001, he says. But if youre testing for whether your new marketing concept is better or the new drill bits your engineer designed work faster than your existing bits, then youre probably willing to take a higher value, maybe even as high as 0.25.
Note that in many business experiments, managers skip these two initial steps and dont worry about significance until after the results are in. However, its good scientific practice to do these two things ahead of time.
Then you collect your data, plot the results, and calculate statistics, including the p-value, which incorporates variation and the sample size. If you get a p-value lower than your target, then you reject the null hypothesis in favor of the alternative. Again, this means the probability is small that your results were due solely to chance.
How is it calculated?
As a manager, chances are you wont ever calculate statistical significance yourself. Most good statistical packages will report the significance along with the results, says Redman. There is also a formula in Microsoft Excel and a number of other online tools that will calculate it for you.
Still, its helpful to know the process described above in order to understand and interpret the results. As Redman advises, Managers should not trust a model they dont understand.
How do companies use it?
Companies use statistical significance to understand how strongly the results of an experiment, survey, or poll theyve conducted should influence the decisions they make. For example, if a manager runs a pricing study to understand how best to price a new product, he will calculate the statistical significance with the help of an analyst, most likely so that he knows whether the findings should affect the final price.
Remember that the new marketing campaign above produced a $1.76 boost (more than 20%) in average sales? Its surely of practical significance. If the p-value comes in at 0.03 the result is also statistically significant, and you should adopt the new campaign.If the p-value comes in at 0.2 the result is not statistically significant, but since the boost is so large youll likely still proceed, though perhaps with a bit more caution.
But what if the difference wereonly a few cents? If the p-value comes in at 0.2, youll stick with your current campaign or explore other options. But even if it had a significance level of 0.03, the result is likely real, thoughquite small. In this case, your decision probably will be based on other factors, such as the cost of implementing the new campaign.
Closely related to the idea of a significance level is the notion of a confidence interval. Lets take the example of a political poll. Say there are two candidates: A and B. The pollsters conduct an experiment with 1,000 likely voters. 49% of the sample say theyll vote for A, and 51% say theyll vote for B. The pollsters also report a margin of error of +/- 3%.
Technically, says Redman, 49% +/-3% is a 95% confidence interval for the true proportion of A voters in the population. Unfortunately, he says, most people interpret this as theres a 95% chance that As true percentage lies between 46% and 52%, but that isnt correct. Instead, it says that if the pollsters were to do the result many times, 95% of intervals constructed this way would contain the true proportion.
If your head is spinning at that last sentence, youre not alone. As Redman says, this interpretation is maddeningly subtle, too subtle for most managers and even many researchers with advanced degrees. He says the more practical interpretation of this would be Dont get too excited that B has a lock on the election or B appears to have a lead, but its not a statistically significant one. Of course, the practical interpretation would be very different if 70% of the likely voters said theyd vote for B and the margin of error was 3%.
The reason managers bother with statistical significance is they want to know what findingssay about what they should do in the real world. But confidence intervals and hypothesis tests were designed to support science, where the idea is to learn something that will stand the test of time, says Redman. Even if a finding isnt statistically significant, it may have utility to you and your company. On the other hand, when youre working with large data sets, its possible to obtain results that are statistically significant but practically meaningless, like that a group of customers is 0.000001% more likely to click on Campaign A over Campaign B. So rather than obsessing about whether your findings are precisely right, think about the implication of eachfinding forthe decision youre hoping to make. What would you do differently if the finding weredifferent?
What mistakes do people make when working with statistical significance?
Statistical significance is a slippery concept and is often misunderstood,warns Redman.I dont run into very many situations where managers need to understand it deeply, but they need to know how to not misuse it.
Of course, data scientists dont have a monopoly on the word significant, and often in businesses its used to mean whether a finding is strategically important. Its good practice to uselanguagethats as clear as possiblewhen talking about data findings. If you want to discuss whether the finding has implications for your strategy or decisions, its fine to use the word significant, but if you want to know whether something is statistically significant (and you should want to know that), be precise in your language. Next time you look at results of a survey or experiment, ask about the statistical significance if the analyst hasnt reported it.
Remember that statistical significance tests help you account for potential sampling errors, but Redman says what is often more worrisome is the non-sampling error:Non-sampling error involves things where the experimental and/or measurement protocols didnt happen according to plan, such as people lying on the survey, data getting lost, or mistakes being made in the analysis. This is where Redman sees more troubling results. There is so much that can happen from the time you plan the survey or experiment to the time you get the results. Im more worried about whetherthe raw data is trustworthy than how many people they talked to, he says. Clean data and careful analysis are more important than statistical significance.
Always keep in mind the practical application of the finding. And dont get too hung up on setting a strict confidence interval. Redman says theres a bias in scientific literature that a result wasnt publishable unless it hit a p = 0.05 (or less). But for many decisions like whichmarketing approach to use youllneed a much lower confidence interval. In business, Redman says, theres often more important criteria than statistical significance. The important question is, Does the result stand up in the market, if only for a brief period of time?
As Redman says, the results only give you so much information: Im all for using statistics, but always wed it with good judgment.
Amy Gallois a contributing editor at Harvard Business Review, co-host of the Women at Work podcast, and the author of the HBR Guide to Dealing withConflict. She writes and speaks about workplace dynamics. Watch her TEDx talk on conflictand follow her on Twitter.
Read more on Analytics and data science or related topics Analytics and data science, Data management, and Experimentation