A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 74

In our study, subjects spent on average 6 min, but they were required to judge the
relevance of the first ten results. When conducting a search on the web, it is more
common to take the first good match and stop. The distribution of all these variables
followed an exponential curve, similar to the ones in our study.
4.6.3 Comparison of Searches for Reference Examples
and Components to Be Used As-Is
The independent variable of Intention had a consistent effect on every dependent
variable. Searches for reference examples required more effort than searches for
components to be reused as-is, and yield a higher proportion of results. The breadth
of this effect suggests that there is an important difference between these two kinds
of searches.
The separation between reference examples and as-is reuse searches was previ-
ously identified in an online survey [ 17 ]. A similar separation can be seen in pro-
totypes search tools. As mentioned in Sect. 4.2 , some tools were designed to search
for components, while others are intended to retrieve snippets. Components tend to
be reused as-is, or with a minimum of modification, adapting, or wrapping. Snip-
pets tend to be modified most extensively, or only consulted as an example, so that
a developer can make use of that knowledge.
We speculate that snippets for reference and components for reuse tap into differ-
ent ways of thinking about programming. Components are black boxes of technol-
ogy, assumed to work, with internals that are not necessarily understood. Snippets
are pieces of knowledge that needs to be understood well enough in order to be
reused. They are kind of how-to knowledge that is embedded in the context of an
example.
4.6.4 Implications for Design
A consistent trend in our study is developers expended more effort to search for
reference examples than for code to reuse, and were rewarded with more results that
were perceived to be relevant. These findings suggest that creating tools to help find
components to reuse as-is will be a challenge. Developers seem to be less willing
to expend effort, and there are additional constraints on the suitability of the search
results, for example, architectural match and compatibility of the software license.
However, we found it encouraging that subjects were willing to experiment with
query terms and features in the search engines.
Subjects using Google employed a higher average number of terms per query,
a higher average clickthrough rate, and more time overall. Again, this additional
effort was rewarded by a more matches that were perceived relevant. The suc-
cess of Google was likely due to the number and variety of pages that it indexes