Due to the escalating cost of drug development and the steady decline of marketed product approval, emphasis in the drug discovery industry has long focused on improving the rate of drug attrition. This ratio is estimated at one drug making it to market for every 10,000 compounds that fail along the way.
The “fail early, fail often” paradigm often applied to drug discovery focuses on making improvements to the preclinical stage of development. In vitro and in vivo models are used to predict how compounds will behave in humans in terms of efficacy, pharmacokinetics, and safety. The findings from these studies are typically used to seek approval from the FDA to conduct human clinical studies.
Given that animal models are currently the gold standard for predicting human responses, over the past five years extra effort has been given to the development of more predictive models. This is especially important considering that retrospective industry experience estimates that rodent toxicology studies predict only 43% of human toxicities; non-rodent studies predict 63%; and the combination of both predict about 71% of human toxicities.4 There is clearly room for improvement. (Cui et al. 2010)
The article is actually about knockout rats, and I will get to that in a moment, but for now, I want to analyze the above.
Number one. Only 1 out of 10,000 chemicals makes it to the marketplace. This in part is why drugs are so expensive and take so long to develop.
Number two. If animal models are the gold standard for predicting human response and only 1 out of 10,000 drugs make it to the marketplace then clearly the gold standard leaves much to be desired. This fact alone might force some, those without a vested interest in the process, to question the entire paradigm of using animals to predict human response.
Number three. Cui et al then reference Olson et al. (the 4 in the above) as proving that the prediction rate for animal models in toxicity studies is around 71%. We address this blatant misrepresentation of the data in Animal Models in Light of Evolution. We list the many flaws of the study and how it is wrongly used. Below are from our list:
2. The study says at the outset that it is aimed at measuring the predictive reliability of animal models. Later the authors concede that their methods are not, as a matter of fact, up to this task. This makes us wonder how many of those who cite the study have actually read it in its entirety.
3. The authors of the study invented new statistical terminology to describe the results. The crucial term here, unqualified at the beginning of the article, is “true positive concordance rate” which sounds similar to “true positive predictive value” (which is what should have been measured, but was not). A Google search on “true positive concordance rate” yielded twelve results (counting repeats), all of which referred to the Olson Study (see Figure 13.3). At least seven of the twelve Google hits qualified the term “true positive concordance rate” with the term “sensitivity”—a well-known statistical concept. In effect, these two terms are synonyms. Presumably, the authors of the study must have known that “sensitivity” does not measure “true positive predictive value,” for later in the middle of the article they qualify the term “true positive concordance” with the term “sensitivity.” In addition to “sensitivity” you would need information on “specificity” and so on, to nail down the crucial concept of “true positive predictive value,” and this the authors did not do! If all the Olson Study measured was sensitivity, its conclusions are largely irrelevant to the great prediction debate. Given the weight placed on the Olson study by friends of predictive modeling, we are left wondering how many of those citing the study got beyond the first page, and of those who did, how many understood elementary statistics.
4. Any animal giving the same response as a human was counted as a positive result. So if six species were tested and one of the six mimicked humans that was counted as a positive. The Olson Study was concerned primarily not with prediction, but with retrospective simulation of antecedently known human results.
(This is why I am not surprised when some people, who are classified as scientists, do not understand simple statistics like sensitivity and positive predictive value. Of course, that does not stop them from criticizing the things they do not understand or from denying that there are laws of physics.)
Cui et al admit the dismal failure rate of using animals to predict human response but then try to justify the practice by quoting from a study they either did not read, or did not understand, or did understand and simply misrepresented.
Cui et al then go on to explain that knockout rats are the answer to all these problems.
As I discussed in my blogs titled Spinal Cord Regeneration in Mice and Genetically Modified Animals, very small differences in genetic composition can result in dramatic differences in responses to drugs and disease. Knocking out or adding genes in rats will not overcome these problems. (For more on why small differences in genomes outweigh the gross similarities among species, see Animal Models in Light of Evolution.)
Every decade or so, animal experimenters are forced to conclude that the animal models they have been using are simply not predictive for human response. They then announce a new and improved animal model e.g. the xenograft mouse, the knockout mouse, and so forth and reassure society that their tax dollars should continue to be given to the animal experimenters so they can cure childhood cancer, heart disease, AIDS and so forth.
The fantasy that a new animal model will somehow predict human response when all before have failed, and when these failures can be explained by our current understanding of evolution and complex systems, reveals either a ludicrous detachment from antecedent events or a complete and total lack of compassion.
The price for the prostitution of reason in the service of romantic fantasies is being paid everyday in the form of dead patients.
Cui, Xiaoxia, Kristen Bettinger, Phil Simmons, and Edward Weinstein. 2010. Closer to Human. Drug Discovery & Development 13 (6):21-23.
Olson, H., G. Betton, D. Robinson, K. Thomas, A. Monro, G. Kolaja, P. Lilly, J. Sanders, G. Sipes, W. Bracken, M. Dorato, K. Van Deun, P. Smith, B. Berger, and A. Heller. 2000. Concordance of the toxicity of pharmaceuticals in humans and in animals. Regul Toxicol Pharmacol 32 (1):56-67.