Biology Reference
In-Depth Information
7.4.2.3 Patient Population Consistency
One of the most important principles in identifying and developing a biomarker classi-
fier is characterization and subsequent selection of the target patient population. Consistency
in the population should be maintained in both the training and test sets by matching on
known relevant patient baseline variables. Multiple human genetics studies have dem-
onstrated the inherent differences between race, ethnicity, and
/
or gender and the ways the
modes within these patient variables can differ with respect to biological processes such as
gene
/
protein expression patterns or clinical outcomes. Further, patient characteristics or life-
style variables such as comorbidities, medications, smoking status, and alcohol consump-
tion status can all potentially drive observed biological alterations. Disease heterogeneity is
another key factor that should be controlled for, particularly in indications with known sub-
types of disease. All of these potential sources of disparity in the patient population should
be accurately assessed in the training set and maintained in the test set to ensure consistency
in the biomarker performance.
7.4.2.4 Iterative Evaluation Cycles
A potential source of bias in the biomarker development process between the training and
test set is the concept of multiplicity, or excessive refinement iteration. This can occur if the
training and test sets are utilized simultaneously in an iterative model optimization progres-
sion. For example, if parameters are optimized on a training set and then applied to the test
set to evaluate classifier performance, and this cycle is repeated multiple times until an opti-
mal result is obtained, there is a concern for overfitting, or non-generalizability of the classifier
[15,16]
. Although, for this example, the test set was not utilized in the parameter optimization
or training process, the iterative fitting of the training set with multiple evaluations using the
test set can be viewed as a biased approach. Depending on the number of cycles conducted
using this approach, as well as the number of patients in each data set, it is possible to reach
convergence to an acceptable performance, and likely overfit performance accuracy, simply
by the sheer number of iterations. By using this practice, typically the training and test set
classifier performance metrics are evaluated side by side within each iteration and the ulti-
mate model parameters are selected on the basis of either the top ranking test set prediction
result or, more likely, one of the top ranking test set prediction results and prediction concord-
ance with the training set performance. Both of these selection scenarios can lead to an overfit
result and the fundamental difference between these approaches, as compared to an iterative
fitting process on the training set
only
(with minimal test set predictions conducted), should be
understood.
7.4.2.5 Clinical Material Considerations
Regardless of the clinical design employed, the acquisition of sufficient clinical material
to analytically validate the diagnostic assay is of utmost importance. Frequently, throughout
the history of an assay's migration to companion diagnostic status, a number of changes will
occur to the assay to enable a more commercially viable test. In the course of these changes,
equivalence in call to previous versions of the test must be established. This is known as
bridging. Bridging is the testing strategy employed whereby a sample is tested with two
different versions of the test to demonstrate equivalence in diagnostic call. Optimally, the
Search WWH ::
Custom Search