Hardware Reference
In-Depth Information
whereas for an organization with immature processes, the capacity to deliver high-
quality products is unreliable and cannot be predicted. There is no reason to believe
that this assumption is not valid for OSS. Concretely, we expect that a higher level of
process maturity will lead to better products and more sustainable communities, and
that successful OSS communities often owe a good portion of their success to the
introduction of sound software processes.
Indeed, many OSS communities have been able to consistently produce software of
adequate quality, making regular releases over the years. There is evidence that this
consistency does not stem from some mysterious property of OSS development that
makes it work against all odds, or from the sheer talent of individual developers, but
that it could be the result of good software development practices being applied and
enforced by OSS communities in a disciplined fashion [7]. For this reason, the EU
project QualOSS—which is generally concerned with the overall quality of OSS
products, as well as with the sustainability of the communities around them—decided
to add a process evaluation framework to its quality model, which is aimed at deter-
mining the ability of an OSS community to consistently deliver adequate products
over time.
In this paper, we describe the first version of this process evaluation framework,
and discuss our preliminary experience with applying it to a small number of OSS
projects. In order to provide some background to the reader, Section 2 briefly de-
scribes the overall quality model defined by the QualOSS project. After a short
discussion of related work in Section 3, Section 4 presents the QualOSS process
evaluation in detail. Our initial experience with the process evaluation is discussed in
Section 5. We close with some general conclusions and a brief discussion of future
work in Section 6.
2 The QualOSS Quality Model
The process evaluation framework we describe in this paper is one component of the
comprehensive quality model developed for the
Quality of Open Source Software
(QualOSS) project. Since the process evaluation framework was designed from the
ground up to contribute to the overall QualOSS model, we start by describing it
briefly.
The QualOSS quality model (or, simply, “QualOSS model” for short) is intended
to support the quality evaluation of OSS projects, with a focus on evolvability and
robustness. One central, underlying assumption while defining the model has been
that the quality of a software product is not only related to the product itself (code,
documentation, etc.), but also to the way the product is developed and distributed. For
this reason, and since the development of OSS products is the responsibility of an
open community, the QualOSS model takes both product- and community-related
issues into account on an equal basis, and as comprehensively as possible.
The QualOSS model is composed of three types of interrelated elements: quality
characteristics, metrics, and indicators. Quality characteristics correspond to the con-
crete attributes of a product or community that we consider relevant for evaluation (see
below for an explanation of how these characteristics were chosen). Metrics corre-
spond to concrete aspects we can measure on a product or on its associated community
assets that we expect to be correlated with our targeted quality characteristics. Finally,
