A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 59

H 1 : The search process is affected by the size of the search target, i.e. user be-
havior is different when searching for blocks of code than when searching for sub-
systems.
H 1 : The search process is affected by the search engine used, i.e. user behavior
is different when searching with Google, Koders, Google Code Search, Krugle and
SourceForge.

4.3.6.2 Interaction Effects

With the three independent variables, we had four possible interactions. We ex-
pected two of them to be significant, so we describe them here.
The first one is the interaction effect between intention of search and search en-
gine. We expected that user behavior with a particular search intention would be
affected by the search engine used. Some search engines would be better for refer-
ence examples than others, e.g. Google versus SourceForge.
H I 1 : The search behavior when searching with a particular search intent (i.e.
reuse or reference example) is dependent on the search engine used.
The second interaction effect was between the size of search target and the search
engine. We predicted that user behavior when searching for a particular target size
would be affected by the search engine used. Some search engines would be better
at locating components than others.
H S 1 : The search behavior when searching for a particular size of code (blocks or
subsystems) is dependent on the search engine used.

4.4 Results

We transcribed the audio, video, and screen recordings of the experiment. Using the
recordings and the transcripts, we coded for our independent variables. The signifi-
cant effects are summarized in Table 4.1 .

4.4.1 Query Length

Overall, the average number of terms per query was found to be 4. The distribution
of the number of terms in a query is given in Fig. 4.1 .
We conducted an analysis of variance (ANOVA) on the number of terms in the
query and found four statistically significant effects. The query length was affected
by the intention of the search

. Searches for refer-
ence examples involved 4.3 terms on average, while searches for components had
only 3.8 terms on average. Query length was also affected by the search engine
(

( F ( 1 , 265 )= 9 . 29 , p < 0 . 01 )

F ( 4 , 265 )= 4 . 06 , p < 0 . 01 ) . Searches using Google tended to have the most terms

(4.7) (Table 4.2 ).