External Evaluation of Population Pharmacokinetic Models of Vancomycin in Large Cohorts of Intensive Care Unit Patients

DISCUSSION

Population PK modeling has become increasingly prevalent since the innovative breakthrough by Sheiner et al. (26) and Beal and Sheiner (27). Such a technique can be used to estimate individual PK parameters through the implementation of empirical Bayes estimation with sparsely sampled data, e.g., the trough concentration only. With the individual parameters obtained, population PK models can in turn be used to predict the plasma concentrations of drugs for individuals following a dosage regimen under consideration. Further, predicted concentrations can be made use of to calculate PK exposure, e.g., the AUC for optimizing the vancomycin dose in clinical practice (10). Over the past few decades, population PK models of vancomycin have been extensively reported (11–17), and among these, some have described the kinetics of vancomycin in ICU patients (11, 21–25). So far, however, these published models have not been sufficiently evaluated with data from external data sets. Deng et al. did evaluate published vancomycin models but did so with data for a population of mainly general patients and with no discrimination of the patient population in which the published models were created (28). One may assume that a population pharmacokinetic model should be able to predict concentration-time data properly for external subjects that belong to the same population in which the model was created. However, our study suggests that this is not necessarily true. The results warn us that it is crucial to evaluate a model in the target populations before applying it in clinical practice. To the best of our knowledge, our study is the first to externally evaluate the published vancomycin models specifically for ICU patients.

The predictive performance of the models evaluated in our study varied considerably and seemed to be very random, as the models of Mangin et al. (24) and Medellín-Garibay et al. (22) tended to overpredict our data, while other models tended to underpredict our data. Among all models evaluated in this study, only the model of Roberts et al. (21) met the predefined criterion for model validity for both VUmc patients and OLVG patients. The reason may be that the characteristics of the patients in our study, which were partly represented in the models by covariate effects, did not fully overlap those of the patients in the published studies, as can be seen in Table 1. Examples are the acute physiology and chronic health evaluation II (APACHE II) and simplified acute physiology II (SAPS II) scores, which were higher in our patient populations than in the populations of patients from the previously published studies, indicating that the patients in our study were more severely ill. When performing an external validation, the effects of covariates in the published models were extrapolated out of the range of values for the original patient population. This may have resulted in a reduction in the accuracy of the model prediction. In addition, a high PE may also be caused by unidentified covariates, which were then, of course, not taken into account in the published studies but which play a role in vancomycin pharmacokinetics in ICU patients. Apart from differences in patient characteristics and covariate effects, poor predictive performance may also be caused by differences in the manner of vancomycin administration in our study and the previously published studies, i.e., applying a model developed with data collected under continuous infusion to a set of data for samples collected during intermittent infusion and vice versa, yet the model of Roberts et al. (21), developed with data collected during continuous infusion, did perform sufficiently with data from the VUmc data set, in which only data for samples collected during intermittent dosing were present.

The result obtained with the SDR model was intended to show how far off vancomycin concentrations could be when treating every patient as if he or she has the same clearance and volume of distribution. The SDR model showed poor PE values, obviously since the SDR did not tailor the dose to the characteristics of ICU patients, like a large V and an often poor or augmented CL. Nevertheless, the SDR model did not perform the worst. Still, our findings provide support for population PK model-based dosing in order to achieve the target concentration and, as such, provide a means that may bring us a step closer to precision medicine; i.e., the use of a model could be better than the use of no model, as long as an external validation is performed.

Some remarks need to be made regarding the design of this study. First, the data that we used in this study were retrospectively collected from clinical settings, in which it was usually hard to precisely record the data. Although data inspection was carried out preceding the model validation, there might remain random errors in the data. However, as the model to be selected will be used in everyday clinical practice, the same imprecision may also be present under such circumstances and, as such, has been automatically accounted for. Second, the models were evaluated using the two data sets separately. The main reason to do so is that the model to be selected was intended to be implemented in the EMRs of both ICUs, where vancomycin is dosed in different regimes (continuous infusion versus intermittent infusion). Additionally, given the imbalance in the quantity of data in the two data sets (the OLVG data set was noticeably larger than the VUmc data set), evaluation of the data sets separately avoided the possibility that the data set with a larger quantity dominated the results. Third, we identified eight models in total from the literature, but two of them were omitted from our study. The model of Del Mar Fernandez De Gatta Garcia et al. was not developed by the use of a nonlinear mixed-effects modeling approach and, thus, did not meet the inclusion criteria (18). The other model that was omitted was from the study of Escobar et al., which incorporated a high-volume hemodialysis filtration rate as the key covariate on clearance and intercompartmental clearance (20). This covariate was not present in our data set. Fourth, in order to evaluate the predictive performance of the published models for the population rather than for particular individuals, population predictions (PRED) instead of individual predictions (IPRED) were used to calculate the PE for each patient. As such, the PE obtained from the models is built up out of the predictive ability of the models and the unexplained variabilities (random effects). Lastly, the reason that we chose 20% as the cutoff value for the prediction error was based on our clinical practice, as with such a prediction error, the risk of unrightfully adjusting the dose based on a predicted concentration above or below the target window while the actual concentration is on target is minimal. For example, for continuous infusion, the vancomycin target range is between 15 mg/liter and 25 mg/liter. Given that the actual concentration is 20 mg/liter, the predicted concentration will be between 16 mg/liter and 24 mg/liter with a PE of 20%. In neither case would a dose adjustment be indicated, which could have been the case with a PE of, e.g., 30%. Additionally, the precision of the predictive performance was not considered in the model selection procedure. Our main concern was that precision does not guarantee the accuracy of a model, which is crucial for validity. As seen in the results (Table 3; Fig. 1 and 3), the models of both Revilla et al. (11) and Llopis-Salvia and Jiménez-Torres (23) showed acceptable precision but had poor accuracy with a clear downward bias. As such, these models were precisely inaccurate in predicting our data and, therefore, less suitable for being selected for use in our clinical settings. Nevertheless, the choice of the criteria used to establish model validity is subjective and can surely be debated.

In conclusion, our study has shown that the model of Roberts et al. (21) had the best predictive performance for ICU patients in both hospitals evaluated in the present study. The fact that not all evaluated models passed the external validation procedure serves as an alert that extensive validation should precede the adoption of any model used for clinical care for ICU patients.