A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 58

experiment procedure, we asked them to conduct a web search for a binary search
tree. Each participant was trained individually.
During the experiment, participants were randomly assigned a search scenario,
which was a combination of intention of search and size of search target. They had
to conduct the search using five search engines. They could choose the engines in
any order. They were free to change their query as many times they wished. Once
they found a page with satisfactory results, they had to rate the relevance of the first
ten matches (P@10). Finally, in the debriefing stage, the participants completed a
questionnaire on their preferences and background in code search.
4.3.5 Subjects
Twenty-four subjects participated in this study. They were recruited on the criteria
that they should have some prior programming experience in either a professional or
an academic setting. The average age of subjects was 27.3 and on average they had
4.3 years of programming experience. All of them had searched for source code on
the Web previously, worked with multiple programming languages and had worked
with a team. Fifty percent of the subjects searched for source code “frequently”,
33 % searched for it “occasionally,” and only 16 % of the subjects searched for
source code “rarely.” Sixty-seven percent of the subjects declared their primary job
responsibility as “Programming.” Twenty-three subjects had experience with Java,
22 subjects had experience with HTML, 21 subjects had experience with C and 20
subjects had experience with C++. Seventy-one percent of the subjects had worked
on small sized teams and 29 % of the subjects had worked on medium sized teams.
4.3.6 Hypotheses
We planned to examine the effect of intention, size of search target, and search
engine on the search process. We identified the following hypotheses to be tested in
our study.
4.3.6.1 Main Effects
Since there were three independent variables, there were three main effects in our
analysis.
H 1 : The search process is affected by the intention of search, i.e. user behavior
is different when searching for reference examples than when searching for code to
reuse as-is.