Pass or fail? Binomialrelated hypothesis testing and confidence intervals using independent samples - Improving the User Experience through Practical Data Analytics - page 106

p-bar ADJ =.43

n ADJ =13.84

and we can note that p-bar is increased by about 7% and n is increased by about 38%,
a modest increase in p-bar, but a relatively sizable increase in n.
However, if n = 100 and p = 0.4, then

p-bar ADJ =.404

n ADJ =103.84

and we can note that p-bar is increased by about 1% and n is increased by about
3.8%, both somewhat modest increases that are unlikely to be material (which is why
we do not bother with the Wald adjustment when n is large). 2
Then, we substitute the adjusted proportion and the adjusted sample size into the
standard Eqn. (4.1) , repeated for convenience:
p-bar±Z*SQRT([p-bar] * [1−p-bar]/n)
OK, let's return to our data from our two search engines. If we consider task 2
using search engine B, in which the sample size = 10, and there were 7 successfully
completed tasks out of the 10, then the p-bar = 7/10 = 0.7. To calculate the 95% coni-
dence interval for the true proportion of people who would complete task 2 using the
Behemoth search engine (from the presumed large population of people who would
or could try task 2 using the Behemoth search engine), we irst calculate the adjusted
proportion and the adjusted sample size:

(
10*.7+1.96 2 /2

) (

10+1.96 2

)

p-bar ADJ = /

=8 . 96/13 . 84= . 64

n ADJ = (10+1.962) =13.84

Using these adjusted values, and assuming a 95% conidence interval is desired,
we simply plug into the Eqn. (4.1) . This gives us

.64±1.96*SQRT([.64] * [1−.64]/13.84)

.64±.25

or

or

.39−−−−−−−−.89

We can be 95% conident that the interval 0.39-0.89 contains the true proportion, “p,”
of people who will successfully complete task 2, using the Behemoth search engine.
This is not as narrow a conidence interval as would be preferred, but it's what we get,
given we have the small (but common in usability testing) sample size of (only) 10.
You calculate the other binomial conidence intervals for all tasks on both search
engines, resulting in Table 4.5 .

2 It might be noted that if p-bar < 0.5, the adjusted p-bar will exceed p-bar, while if p-bar > 0.5, the
adjusted p-bar will be less than p-bar. Since we work with both p-bar and (1 - p-bar), multiplied, this
distinction is not material. The adjusted p-bar will exactly equal the actual p-bar when p-bar = 0.5.