A Controlled Experiment on the Process Used by Developers During Internet-Scale Code Search - Finding Source Code on the Web for Remix and Reuse - page 57

4.3.2 Dependent Variables
In our analysis, we used five dependent variables: number of terms in a query, num-
ber of queries in a session, clickthrough percentage, P@10 and time spent on a
search engine. These dependent variables helped us to study the search process of
developers.
A query term is defined as any string of characters separated by some delimiter
such as a space, a colon, or a period, depending on the syntax of the search specifi-
cation. We counted Boolean operators or search filters in a query as one term. For
example, the query “apache access log parser, lang:python” has five terms.
We counted as a query any search specification that was entered and results were
returned (including a null set).
Clickthrough percentage is the proportion of returned matches that a user clicked
on to view in detail.
P@10 is the relevance of the first ten matches as judged by the subject.
A session is considered to start when a user begins using a search engine and
to end when the relevance judgment is provided. The duration of a session is the
amount of time spent in seconds.
4.3.3 Scenarios
Multiple scenarios were constructed for every treatment. Two scenarios were gener-
ated for searches for blocks and three were constructed for searches for subsystems.
A complete list of scenarios has been described elsewhere [ 14 ]. An example is pre-
sented below:
You are working in the Python programming language, and need to have multi-
threading functionality in your program. You have never used threads before, and
would like to know how to create threads, to switch between threads, and so on.
Look for examples by which you can learn. Any thread implementations of Python
programs are relevant. Remember you will not be using the code directly, you would
like to learn how to use it.
4.3.4 Procedure
The experiment consisted of three stages: training, experiment task, and debriefing.
In the training stage, the participants were given a warm up task to make them
familiar with the experimental setup and the think aloud procedure. To train the
participants on think aloud, we asked them to identify how many windows were
in their parents' house. Generally, to answer this question they had to mentally
walk through the house and count the windows. To train the participants on the