Determining What to Study - Evaluation Methods in Biomedical Informatics - page 51

There is always a range of potential players in any evaluation (see Chapter
2), and there is no formula directing whom to consult or in what order. The
investigators will apply their common sense and learn to follow their
instincts; it is often useful to establish a steering group to advise on whose
concerns are most relevant.
Through discussions with various stakeholder groups, the hard decisions
regarding the scope of the study are made. A significant challenge for inves-
tigators is the risk of being swamped in detail created by the multiplicity of
questions that can be asked in any study. To manage through the process,
it is useful to reflect on the major issues identified after each round of dis-
cussions with stakeholders and then identify the questions that address
these issues. Where possible, it is helpful to keep questions at the same level
of granularity. Some investigators find tools for issue tracking, such as white
boards, Post-It notes, or mind-mapping software, useful. What is important
at this stage is to keep a sense of perspective, distinguishing the issues as
they arise and organizing them into some kind of hierarchy; for example,
with low-, medium-, and high-level issues. Inter-dependencies should be
noted and care should be taken to avoid intermingling global issues with
more focused issues. For example, when evaluating an electronic lab note-
book system for researchers, it is important to distinguish focused, low-level
issues such as the time taken for users to enter data from global issues such
as the impact of the resource on research productivity.
To help investigators in their study planning, the full range of what can
potentially be studied will be listed, as well as what can be derived from
this and a catalogue of nine study types relevant to evaluation across all of
biomedical informatics. To both ensure that the most important questions
do get “on the table” and to help eliminate the less-important ones, it can
be useful to start with such a comprehensive list.
The Full Range of What Can Be Formally Studied
In relation to biomedical settings, there are five major aspects of an infor-
mation resource that can be studied:
1. Need for the resource: Investigators study the status quo absent the
resource, including the nature of problems the resource is intended to
address and how frequently these problems arise.
2. Design and development process: Investigators study the skills of the
resource development team and the development methodologies em-
ployed by the team to understand if the resulting resource is likely to
function as intended.
3. Resource static structure: Here, the focus of the evaluation includes spec-
ifications, flow charts, program code, and other representations of the
resource that can be inspected without actually running it.