Translating Biomarker Discovery into Companion Diagnostics through Validation and Regulatory Consideration - Genomic Biomarkers for Pharmaceutical Development - page 164

This is a very aggressive clinical trial design in which one must have strong confidence in
the scientific link between diagnostic status and drug response. This design has a number of
significant drawbacks. Perhaps the most critical one is the lack of understanding of the response
rates in the diagnostic negative patient population. Excluding patients from treatment with-
out understanding the potential clinical response rate is a high-risk strategy and one not easily
accepted by the FDA. As a result, this strategy is predominantly utilized only when the diagnos-
tic is predicting some significant toxic response, and by preventing these subjects from receiving
the therapy the benefit-risk profile of the therapeutic is increased. Should this be the case, the
therapeutic would certainly be limited in its label to include only test-positive patients. Another
drawback of this approach is that the test would have to be nearly fully validated to be utilized
in this manner. This limits the use of this design to later stages of development only, as data to
properly validate the test as well as to indicate the toxic response occurring only in the negative
group would be required from early phase trials. This would require a significant number of
subjects to demonstrate statistical significance; most likely a very large Phase II trial is needed to
show that the therapeutic of interest does not benefit patients in the diagnostic negative group
and is statistically different from that of the diagnostic positive group. In terms of benefit, this
trial design is likely to enroll quickly if the prevalence of the diagnostic positive subjects in the
population is high. Usually, this is not the preferred design of either the FDA or the investigator.
Due to the risk and aggressive nature of this design, it is employed only in those cases where
there is a direct link between the diagnostic and drug mechanism of action (MoA) or where
there are significant safety concerns in the diagnostic negative patient population.
7.4.2.2 Establishing a Training and Test Set of Samples
The process of training and testing as two independent steps in the development of a
biomarker classifier is very important, though sometimes incorrectly applied. A training set,
by definition, is the data set that is used to optimize the features (e.g., transcripts, proteins,
cellular subtypes), account for the variability within the target population, and ultimately
establish a cut-point to discriminate conditions or groups. A test set, also called a validation
set or hold-out set, is an independent data set with similar characteristics to the training set,
used to evaluate the performance of the classifier, where performance can be defined by met-
rics such as sensitivity, specificity, predictive value positive / negative, and / or area under the
curve [12,13] . There are methods to utilize the training set and perform cross validation pro-
cedures to mimic the structure of a test set. These approaches can range from n -fold random
or stratified splits to leave-one-out cross validation, though it can be argued that true perfor-
mance evaluation is best represented using a completely independent data set for testing,
and as such, the discussion provided here will focus on the training / test set design.
For appropriate use of a test set, there should be no structure from the training set or
training process introduced. The test set should serve as a real world application of the clas-
sifier performance, such that all of the model parameters that have been optimized in the
training process, should be held constant and not modified once model testing has com-
menced [14] . Though this concept appears straightforward at first glance, there are various
points of either disparity or contamination between the two data sets that can be over-
looked, thus introducing bias leading to an overfit result into the biomarker classifier devel-
opment process. The next few sections discuss some of the more prevalent inconsistencies
or potential pitfalls in classifier development for a biomarker application.