A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 53

Chapter 4
A Controlled Experiment on the Process
Used by Developers During Internet-Scale
Code Search
Susan Elliott Sim, Megha Agarwala, and Medha Umarji
Abstract It has become common practice for developers to search the Web for
source code. In this paper, we report on our analysis of a laboratory experiment with
24 subjects. They were given a programming scenario and asked to find source code
using five different search engines. The scenarios varied in terms of size of search
target (block or subsystem) and usage intention (as-is reuse or reference example).
Every subject used five search engines (Google, Koders, Krugle, and Google Code
Search, and SourceForge). We looked at how these factors influenced three phases
of the search process: query formulation, query revision, and judging relevance.
One consistent trend was searching for reference examples required more effort,
as measured by average number of terms per query, average number of queries,
clickthrough rate, and time spent. This additional effort paid off in a higher rate of
precision for the first ten results.
4.1 Introduction
Searching for information on the Web has become a daily occurrence. This prac-
tice has also extended into the world of software development. Programmers often
search for source code examples to remind themselves of syntax and for compo-
nents to reuse on projects. The search process used by software developers during
S.E. Sim ( )
Many Roads Studios, Toronto, ON, Canada
e-mail: [e-mail]
M. Agarwala
Columbia University, New York, NY, USA
e-mail: [e-mail]
M. Umarji
Symantec Corporation, Mountain View, CA, USA
e-mail: [e-mail]