Pass or fail? Binomialrelated hypothesis testing and confidence intervals using independent samples - Improving the User Experience through Practical Data Analytics - page 102

FIGURE 4.8
Output comparing search engines N(1) and B(2) for task 1; SPSS.
So, using the p -value of 0.020 (we are doing a two-sided hypothesis test, as indicated
by the H0 and H1 being = vs. ≠), and since this is less than 0.05, we reject H0 and con-
clude that for task 1, the (true) pass rate for Novix is not the same as that for Behemoth.
Now, this is the type of result you might have predicted from the beginning of this
chapter. After all, Behemoth had 9 passes out of 10, while Novix had only 3 passes
out of 10. But once you've done these calculations, you hold a much stronger hand.
Scientia Potentia Est. The p -value tells you that if the true pass rates were the same
for Behemoth and Novix, for this task, the difference we have observed (9 out of
10 vs. 3 out of 10) will occur only 2% of the time, that is, on average, 2 out of 100
times. Since this is below the traditional accepted signiicance level of 0.05, we con-
clude that the difference in the pass rates is due to the difference between the search
engines and not due to mere chance. Simply put, your current engine does better than
the newfangled engine on this particular task.