A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 56

researchers to look for general trends and tendencies in user behavior, by having
multiple subjects perform the same task.
In a previous study, we found that searches for code on the Web tend to vary
along two continuous dimensions [ 17 ]. The first dimension is the intention behind
the search. The endpoints on the continuum are reference example and as-is reuse.
Points along the continuum are distinguished by the amount of additional code that
needs to be written. Reference examples are reminders of syntax or how-to do some-
thing. Once found, the developers usually write the code needed from scratch. Dur-
ing searches motivated by as-is reuse, the developer is looking for components to
use without modification. The second dimension is the size of the source code de-
sired. We draw attention to three points along the continuum: block (few lines of
code); subsystem (class, package, or library); and system (a program that can be
used on its own).
Bajracharya and Lopes [ 2 ] analyzed the log of a code search engine to identify
common search topics. Brandt et al. [ 3 ] looked at how developers search for ex-
amples of how to use an API, using both logs and a laboratory experiment. They
concluded that the Web is used for learning and reminding. To our knowledge, there
are no observational field studies of code search on the Web. This paper reports on a
laboratory experiment where we studied how developers searched for source code.
4.3 Method
In this section, we describe the design of the laboratory experiment. Many of the
design decisions, such as the independent variables, search scenarios, and search
engines, were motivated by a previous study [ 17 ].
4.3.1 Independent Variables
There were three independent variables in this study and our analysis: size of search
target, intention of search and the search engine. Size of search target and intention
of search were between-subjects factors.
Size of search target refers to the amount of source code searched by the devel-
opers on the Web. This variable had two levels: block or a subsystem. Block refers
to few lines of code, for e.g. a function. Subsystem is a piece of functionality that is
not a stand-alone application and is used as a component.
Intention of search refers to the motivation for searching source code on the Web.
This variable had two levels: reuse as-is or reference example. Reusable code is
source code that can be simply dropped in a program without any modification. A
reference example is a piece of code that gives a direction to do something, for e.g.
how to use a particular GUI widget.
Search engine was a within-subjects independent variable with five levels:
Koders, Google Code Search, SourceForge, Google and Krugle.