Translating Biomarker Discovery into Companion Diagnostics through Validation and Regulatory Consideration - Genomic Biomarkers for Pharmaceutical Development - page 173

scores and a positive clinical outcome can be drastically modified if the cut-point is increased
by a certain factor. The negative predictive value could become suboptimal to the positive pre-
dictive value, thus altering the intended implementation and interpretation of the assay.
Patient scores that are more uniformly distributed, or present no level of multimodality,
can be more challenging to identify a cut-point in, even with change-point regression models
or unimodality tests [23] . It is important to note that most supervised classification methods
can appear to construct a dividing plane between any two populations with minimal or no
errors on a training set of data. However, if this dividing plane or cut-point does not repeat
on an independent set of patients, it is not a viable decision rule or threshold. Assays with
indeterminate cut-points, or gray areas , complicate the interpretation of results by physicians
and careful consideration should be given to the value of the assay when discrimination is
not apparent. Various methods have been developed to account for such instances, where the
values close to the cut-point, or gray zone values, are regarded as less confident prediction
scores and, as a result, are either not reported or down-weighted in the final assay test result
[24,25] . The approach proposed by Coste et al., uses the training set to establish post-test
probability scores that are used to calculate likelihood ratio positive and negative thresholds,
or upper and lower acceptance values that flank the cut-point at predefined predictive values
in setting a gray zone [25] . When prediction scores fall in this gray zone , the scores are regarded
as low confidence scores and not reported. The obvious concern with such a method is the
absence of test results for scores that fall in this region. These non-reported results can be
combined with assay failure frequencies since, in both cases, no test result is provided to
the patient, thus questioning the predictive value of the assay.
The studies included in this section should demonstrate data from the training and test
sets and the potential predictive value demonstrated in these two data sets. In addition, the
distribution of the signature positive and negative subjects, and where this cut-point lies,
should be apparent.
7.4.3.7 Controls and Calibrators
The use of assay controls allows for the assessment of fluctuations in the assay and the
establishment of run pass / fail criteria. While the identification and use of controls and cali-
brators may seem trivial in the research setting, it becomes increasingly meaningful as the
assay is developed and validated. In a research setting, a single sample 'normal' or a pooled
'normal' created from a number of donors can often be used as a calibrator. However, the
practical applications of such a sample become much more difficult under expanded test-
ing in a clinical setting. It becomes increasingly difficult to produce a consistent calibrator
sample in this fashion given the constraints of the testing environment. For this reason it is
important to identify and implement appropriate controls early in assay development, and
limit the need for calibrators if possible. Controls and calibrators can also be generated from
a synthetic target, and this should be considered as a viable option as the production of large-
scale, highly consistent synthetic targets is logistically more feasible than implementing sub-
ject sample controls. Upon identification and implementation of a control and calibration
scheme, data included in validation from this section should demonstrate the reproducibility
of the controls and calibrators, including the setting of pass and fail criteria based on this per-
formance. In addition, fluctuations in calibrator performance and their effect on assay results
should be clearly demonstrated.