Comparing two designs (or anything else!) using paired sample T-tests - Improving the User Experience through Practical Data Analytics - page 85

Within moments of sending the report out to the team, Andy Moodboard charges
into your cube, clutching the report: “Are you kidding me? The 'Long Scroller' was
almost 1.5 minutes faster? That's amazing!”
“Yes,” you calmly reply. “As much as we're wed to that 'Wizard,' the task times
show the folks really get through the form faster with just a long scrolling vertical page.”
“Wow,” Andy breathes, “I never would have predicted that. I mean, we always
thought the wizard would keep everybody focused, so they could inish it faster.”
“Well, the data show otherwise. And that low p -value proves there's not much
doubt that we would get the same result if we could magically test with our entire
client base.”
“Yeah, very cool,” Andy says. “Listen, I'm gonna schedule a lunch meeting with
Hans tomorrow so you can present the results. He's gonna love this stuff! You need
any more prep time to inish up your report?”
“No, I'm ine with lunch tomorrow. Can you avoid ordering from that lame faux
Italian place?”
“You can name the place from here on out!”
3.6 ADDENDUM: A MINI-DISCUSSION WHY THE
INDEPENDENT AND PAIRED TESTS NEED
TO BE DIFFERENT
You might wonder why there needs to be a different procedure performed by the
software depending on whether we have two independent samples or we have two
samples that are paired. After all, the hypotheses being tested are basically the same:
H0:μ1=μ2
H1:μ1≠μ2
The main reason for the two approaches is a bit subtle and is due to the way variabil-
ity is properly measured for each test. The hypothesis tests we have looked at all have
one thing in common; they consider the difference in sample means and they put that
difference in ratio to a measure of the variability in the data.
Here's why. The logic of why we consider the difference in means in ratio to the
variability is that when there is a small amount of variability, the two sample means
are relatively close to their respective “true values,” and, therefore, a modest differ-
ence in sample means tends to indicate that the true means do, indeed, differ.
However, when there is a lot of variability in the data, the sample means may
be quite far away from their respective true values, and it sensibly requires a larger
difference in the sample means to be convinced that a real difference exists. By con-
sidering the ratio, this concept is accounted for.
So, granting the above, it is crucial to consider the proper measure of variability
to use as the denominator. In other words, we need different variability measures to
act as the proper denominator against which to compare sample means, depending
whether we have independent data or paired data. Why? Read on, dear reader!