Strategies for Testing LINT - Cryptography in C and C++ - page 299

In addition to these tools, the freely obtainable Splint has been adapted
to run under Linux. Splint distinguishes four modes ( weak , standard , check ,
strict ), each connected with certain presets, and which carry out tests of varying
degrees of rigorousness. In addition to the typical lint functions, Splint offers
possibilities to test programs for the presence of particular specifications, which
are inserted as specially formatted comments in the source code. In this way
boundary conditions for the implementation of functions and their invocation
can be formulated and their conformity to specifications checked, and there are
additional possible semantic controls.
For programs not equipped with supplementary specifications the mode
set with the option -weak is recommended as standard. However, according
to the manual a special reward will be presented to the first person who can
produce a “real program” that produces no errors with Splint in -strict mode.
As a precondition for using these two tools in a reasonable manner it proved to
be useful in testing the FLINT/C functions to test precisely which options are
used and to create corresponding profile files so as to configure the tools for
individual use.
After extensive revisions of the FLINT/C code, at the end of the test phase
neither of the two products produced any warnings that on close examination
could be considered serious. With this goes the hope that we have come a long
way in fulfilling the conditions set above for the quality of the FLINT/C functions.
13.2 Run-Time Tests
The goal of run-time tests should be to prove that a building block of a piece of
software fulfills its specifications. To give the tests sufficient expressive power
so as to justify the expense of time and money that goes into their development
and execution, we must make of them the same demands that we do of scientific
experiments: They must be completely documented , and their results must
be reproducible and able to be checked by outsiders. It is useful to distinguish
between testing individual modules and integrated system tests, though here the
boundaries are fluid (see [Dene], Section 16.1).
To achieve this goal in testing modules the test cases must be so constructed
that functions can be exhaustively tested to the extent possible, or in other words,
that as large a coverage as possible be achieved of the functions being tested.
To establish test coverage various metrics can be employed. For example, for C0
coverage what is measured is the portion of instructions of a function or module
that are actually run through, or, concretely, which instructions are not run
through. There are more powerful measurements than C0 coverage, which take
note of the portion of branches that are taken ( C1 coverage ), or even the portion of
paths of a function that are run through. The last of these is a considerably more
complex measure then the first two.