Pass or fail? Binomialrelated hypothesis testing and confidence intervals using independent samples - Improving the User Experience through Practical Data Analytics - page 111

Ultimately, we arrive at Figure 4.10 and examine the p -value.
As can be seen in Figure 4.10 (see arrow), the p -value is 0.009559, or just
about 0.01. This is, of course, less than our traditional benchmark of 0.05, and
we reject H0 and conclude that for task 1 (in Table 4.9 ), the pass/fail rates are
not the same for the three search engines. While the chi-square test does not
directly test in a formal way how the pass/fail rates differ (if they do), we can
feel very comfortable in this example if we conclude that the N pass rate differs
from the other two search engine pass rates, while it is extremely likely that the

Table 4.9 Theoretical Expected Frequencies with Three Search Engines

Task 1
Search Eng:

1 3 7

Pass 6.667 6.667 6.667

Fail 3.333 3.333 3.333

FIGURE 4.9
Excel input, three search engines.