A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 67

We analyzed the effect of order of search engines used on the different dependent
variables: number of terms in a query, number of queries, click-through percentage,
and time spent on the search engine. P@10 was not influenced by the order of use
of search engines. In other words, any given search engine received the same P@10
regardless of whether it was used first, last, or anywhere in between.
We found two statistically significant order effects. An ANOVA revealed that the
clickthrough percentage was affected by order

( ( , )= . , < . )

.There
was a dip in clickthrough rates at the middle of the experiment session. In other
words, subjects navigated to a smaller proportion of matches than at the beginning
or end of the session, as shown in Table 4.8 .

F 4 92 2 89 p 0 05

Source Sum of squares df Mean sum of squares F Pr > F

0.00 ∗

Subject 5.98 23 0.26 3.07

0.03 ∗ Engine 0.98 4 0.24 2.89

Residual 7.80 92 0.08
∗ The difference is statistically significant at the p < 0.05 level

Table 4.8: ANOVA for order effect on clickthrough

Fig. 4.8 Order effects. ( a ) Order effect on clickthrough and ( b ) order effect on time spent

. Sub-
jects tended to spend more time on earlier search engines and spent less time on
subsequent search engines (Table 4.9 ).
Figure 4.8 b shows the amount of time spent on the search engines decreased from
the first engine to fifth engine. The small increase in time from fourth engine to fifth
engine was not significant.

There was an order effect on the time spent ( F ( 4 , 92 )= 11 . 36 , P < 0 . 01 )