Pass or fail? Binomialrelated hypothesis testing and confidence intervals using independent samples - Improving the User Experience through Practical Data Analytics - page 108

Staring at the chart, you're ready to answer Blitz's questions. You insert your lat-
est conidence interval chart into your presentation and send it to Hans. As expected,
he pops his head into your ofice within 15 minutes.
“Thanks, buddy. But what's with the enormous ranges?” That's as precise as you can get?
“It's the nature of binomial data with a sample size so small. Those ranges shrink
only as your sample sizes increase.
“English, please.”
“Even though the intervals are broad, they hardly ever overlap on those crucial
tasks 1, 2, 3, and 7. This further conirms that our old engine really did do better on
those tasks.”
“Got it. Make sure your phone is on between 10 and 11 am tomorrow. That's
when I'm going to show Joey this stuff. I may need you to cover me in the crossire.”
4.6 SUMMARY
In this chapter we studied hypothesis testing for “nominal/categorical” data, more
speciically, “pass/fail” data, for independent samples, and the determination of con-
idence intervals for true proportions. These situations arise primarily when we are
assessing the rate (proportion) of successful completions of various tasks, although,
as exhibited in this chapter, the key issue was comparing successful completion rates
of a task for different search engines. We assume that the underlying probability
process is binomial and discuss the assumptions implied by this.
When we perform hypothesis tests to inquire whether successful-completion
rates are the same for a task with different search engines, the underlying test
we perform is the chi-square test of independence, sometimes referred to as the
chi-square contingency-table test. When we are comparing speciically only two
search engines (or two “whatevers”), each with the binary result of pass/fail, we
recognize that in this one situation, it is superior to replace the aforementioned
chi-square test by Fisher's exact test. The hypothesis tests are illustrated using
Excel and SPSS. Conidence intervals are not constructed by the software, but
details are provided to enable the reader to easily do the construction “by hand”
(which expression includes the use of a calculator!). There are online calculators
that compute the adjusted Wald conidence interval. One example is www.measu
ringusability.com/wald.htm ).
4.7 ADDENDUM 1: HOW TO RUN THE CHI-SQUARE TEST FOR
DIFFERENT SAMPLE SIZES
When we covered how to do the chi-square test using Excel, we gave you a simple
rule for computing the table of theoretical expected frequencies (called “expected
range” by Excel) of the pass/fail values. As we noted, since we have independent