Skip to main content

Liars and Statistics. Part II

In Liars and Statistics Part I, I reviewed comments from scientists linking animal models to the failure rate in drug development. Here, I begin my examination of Professor Robin Lovell-Badge’s blog titled: Nine out of ten Statistics are taken out of Context. (I remind the reader that all comments in quotations that are in bold are my emphasis not the original author’s.)

Lovell-Badge states: “Those opposed to animal research often point out that most drugs that pass the legally required toxicology tests in animals go on to fail in human clinical trials. They then go on to suggest that this shows that animal research does not work, or that it is proof that animals are not accurate models for humans.”

That is not exactly what I have said. I have stated that such failures mean that animal models are not of predictive value for human response to drugs. That animal models are not predictive for human response to disease is illustrated by disease research as well as drug development research because many drug targets are chosen based on animal models of disease. But that it is not ALL that I have said. As I pointed out many times, in vitro is not that great either in terms of drug efficacy or toxicity. Pharma needs better tools! Especially for predicting human response in terms of efficacy and safety.

For these reasons, the above by Lovell-Badge is misleading. But the above is also misleading as I have never said that because animal models fail in terms of drug development that, “animal research does not work.” In fact, I break down animal-based research and other uses of animals in science in general, into nine categories and acknowledge that animal use “works” in seven out of the nine. Lovell-Badge is here committing the fallacy of distribution, taking my comments about one part of animal use and making it appear that it is representative of my view of all parts. This is a common fallacy among vivisection activists

To be more specific regarding animal models and drug development, drugs that pass efficacy and toxicity in animal tests fail in clinical trials because of efficacy and safety issues. This is what animal models are primarily used for and it is the main reason drugs fail late in human trials. So failing on these measures is cause for concern for animal modelers as opposed to a drug failing because of acid-base problems, which are usually assessed in vitro. This is not controversial. Pradeep Fernandes, Co-founder, President; Shireen Vali, Co-founder, Chief Scientific Officer; Cellworks Group Inc. writes in 2012:

Drug development programs today have a 5% to 10% probability of success. Almost half of the failures are due to drug safety issues found very late in the clinical development process. The lack of improvement in outcomes, despite advances in technology and the near doubling of pharmaceutical R&D expenditures, highlights the need for novel approaches to drug development.

Currently, the identification of efficacy and safety risks for a lead compound primarily uses cell line and in vivo studies. Unfortunately, these experimental systems are black boxes that offer limited visibility into selected phenotypes and biomarkers and very little insight into the effects of a compound on important physiological pathways. Due to this lack of transparency into pathway effects, it is difficult to generate insights into system-level changes in the physiological network. This is often a reason for potential oversight of toxicity issues and incorrect assessment of efficacy.

Khanna states in 2012:

Similarly, analysis of Phase III submissions during 2007–2010 by CMR, indicated approximately 50% overall failure rate for drugs for the primary or major new indications . . . The formulation changes or close extensions of previously approved drugs were excluded from this analysis. The lack of efficacy (66%) was again the overriding reason for investigational drug failure. The safety concerns and lack of risk/benefit ratio emerged as the second reason and contributed to failure of 21% candidates. [1]

Only 13% of drug failures in Phase III from 2007-2010 were for reasons other than the ones animal studies are supposed to address. THIS is why people say animal models are not predictive! Of course there is also data in the form of direct animal model-to-human comparisons that irritate people trying to spread misinformation. (See previous blogs and FAQs About the Use of Animals in Science: A handbook for the scientifically perplexed.)

Catherine Shaffer, Contributing Editor of Drug Discovery & Development, stated in 2012: “Drug development is an extremely costly endeavor. Estimates of the total expense of advancing a new drug from the chemistry stage to the market are as high as $2 billion. Much of that cost is attributable to drug failures late in development, after huge investments have been made. Drugs are equally likely to fail at that stage for safety reasons, as for a lack of efficacy, which is often well-established by the time large trials are launched.” [2]

Lovell-Badge continues, stating: “However, this is misleading without an understanding of the relevant context and the reasons for the animal safety tests. Ironically, the figures cited by many animal rights activists are actually drawn from industry and are intended to explain the expense of developing safe and useful medicines.” I actually read the drug development literature on a regular basis and that is not the focus of the articles that cite failure percentages. I am no fan of Pharma, see Bad Pharm by Ben Goldacre for reasons why, but they are not trying to justify anything, they are trying to fix a terminal problem. Lovell-Badge is here committing the fallacy known as poisoning the well.

The pharmaceutical industry is not doing as well as they were in the 80s in terms of successfully bringing drugs to market and hence in terms of making money. The articles that I read and quote from, for the most part are addressing that issue and examining what can be done differently. While it is true that Pharma values the bottom line, in order to have a bottom line they have to put out drugs that work or that at least don’t kill a lot of people. Heidi Ledford quotes Anders Ekblom, head of science and integration at AstraZeneca: “The billion-dollar question is ‘how early can I know that the approach I’m taking will definitely turn into a drug that delivers exactly what I would like to see?’ A lot of the cost in today’s drug development is the cost of failures. We are all trying to focus our energy on how we can get different technologies to better predict outcomes.”[3]

Likewise, Cressey states in 2011:

With drug pipelines running dry and a slew of blockbuster medicines about to lose patent protection, the voices arguing that the traditional drug-development process is too expensive and inefficient to survive are getting louder.

Employing thousands of in-house scientists to develop drug candidates from scratch has turned into a billion-dollar gamble that simply isn't delivering enough profitable products to market. Bernard Munos, founder of the InnoThink pharmaceutical policy research group in Indianapolis, Indiana, is not alone in believing that the next three years "will probably see an implosion of the old model" of drug discovery . . . Patrick Vallance, senior vice-president for medicines development and discovery at London-based drug-makers GlaxoSmithKline (GSK), also believes that IP will be the most contentious part of Bountra's model. "I'm completely on board with the idea you don't really know if you're on track until you've done an experiment in the clinic, and that you should publish that early," he says. [4]

I will address the experiment in the clinic in part III.

The following is from Nature Biotechnology April 2011 p300, which is quoting the New York Times, March, 7, 2011: “ ‘This is panic time, this is truly panic time for the industry.’ Tufts’ Kenneth Kaitin, director of the Center for the Study of Drug Development, on pharma’s realization that revenues are too feeble to sustain their R&D as they witness their pipelines drying out.”

Pharma is not trying to justify cost, they are trying to bring new products to market less expensively, with higher efficacy, and better safety. That is how they make money and a quick perusal of the drug development literature would have illustrated this. The claim that Pharma is just trying to justify cost is nonsense, utter nonsense! Read the literature or attend a drug development conference and this fact will be undeniable. Unless of course you have an ulterior motive for making the claim.

Lovell-Badge continues:

To help break down these statistics, it is useful to look at the success rates at each stage. In the diagram below the red percentages show the proportion of drugs that move from one stage to another – so 64% of New Molecular Entities (NMEs – essentially new drugs) will pass the animal tests (preclinical studies) and be moved into Phase 1 clinical trials in humans. Looked another way, animal experiments remove 36% of the potential drugs from moving onto the next stage. This is almost certainly a good thing as it avoids humans being given drugs which are likely to be toxic to them.

According to Nature, the percentages are 64%, 48%, 25%, 67%, and 83% in terms of success rates from preclinical through Phases I, II, and III and then to Registration for all drugs in years 2005-2009. I could not find the 2007-2011 chart Lovell-Badge presented, but what Lovell-Badge presents is in line with the 2005-2009 years so I will accept his figures (moreover, there is little difference between the two). However, his statement regarding animal models removing drugs that would have been toxic to humans is a fallacy called begging the question (from the Latin petitio principii, “assuming the initial point”). Begging the question goes something like this: first you assume your conclusion in some form or fashion, then you state your conclusion based on your assumption, then your assumption has been proven. It resembles circular reasoning. For example:

P1 Murder is morally wrong.

P2 All abortions are murders. (Suppressed)

Therefore, abortion is morally wrong.

Note that the question here is: “Is abortion morally wrong.” Another way to phrase the question would be to say: “Is abortion murder?” as murder is obviously morally wrong. So the speaker has not logically equated murder with abortion, he has assumed it, hence the fallacy.

Lovell-Badge’s statement could be phrased as follows:

P1 Animal tests are predictive for toxicity

P2 Animal tests kept 36% of new drugs from further development because of toxicity

Therefore animal tests prevented drugs from going to humans that would have been toxic to them.

As per P2 in the abortion example, if P1 is correct and animals really are predictive for humans, then the above is logically sound. But the question actually is: “Are animal tests predictive for human toxicity?” and therefore the above is fallacious. We actually do have data from animal and human studies, that I have discussed and referenced many times, that shows that toxicity testing in animals has positive and negative predictive values below 0.5. For teratogenicity and carcinogenicity, the PPV is way below 0.5. A Reuters article on MSNBC discuses a computer-based method for predicting drug toxicity and quotes Francis Collins who supports this position. The chip would test for activation of genes and proteins in various human tissues:

"If things are going to fail, you want them to fail early," Dr. Francis Collins, the director of the National Institutes of Health (NIH), told Reuters on Friday. "Now you'll be able to find out much quicker if something isn't going to work."

Collins said a drug's toxicity is one of the most common reasons why promising compounds fail. But animal tests -- the usual method of checking a drug before trying it on humans -- can be misleading. He said about half of drugs that work in animals may turn out to be toxic for people. And some drugs may in fact work in people even if they fail in animals, meaning potentially important medicines could be rejected.[5]

Collins is not alone. Alan Oliff, former executive director for cancer research at Merck Research Laboratories in West Point, Pennsylvania stated in 1997: “The fundamental problem in drug discovery for cancer is that the [animal] model systems are not predictive at all.” [6] Chapman stated in 2011:

A major factor complicating risk analysis in FIH [first in human] trials is the difficulty of making accurate predictions from preclinical laboratory research on human tissues and animal studies of the likely effect of the investigational agent on humans. According to Rebecca Dresser, risk analysis based on preclinical research can fall short in three ways. It may fail to predict human risks, leading to adverse effects in human trials – one example being the TGN1412 trial. It may predict clinical benefits that then fail to materialize for human subjects. And it may predict nonexistent risks in humans with the result that a potentially useful agent is discarded [7].

Extrapolating from laboratory and animal studies is a complex process under all circumstances, but even more so in proposed FIH trials which usually lack data from comparator studies in humans to help guide the analysis. Although an effort is usually made to choose species based on their similarities to the human biological response under study, there may not be appropriate animal models that accurately replicate the human disease. Moreover, there are significant differences between human and animal physiology. Given the limitations of animal models of many diseases and differences between human and animal physiology, toxicological studies in animals may be poor at predicting toxicity in humans [8]. For similar reasons, the ability to show proof-of-principle in preclinical research, whether in the in vitro or the animal studies, does not provide a therapeutic warrant for humans [9]. . . . [10]

David F. Horrobin wrote in 2003, in Nature Reviews Drug Discovery:

Does the use of animal models of disease take us any closer to understanding human disease? With rare exceptions, the answer to this question is likely to be negative. . . . All the other animal models — including those of inflammation, vascular disease, nervous system diseases and so on — represent nothing more than an extraordinary, and in most cases irrational, leap of faith. We have a human disease, and we have an animal model which in some vague and almost certainly superficial way reflects the human disease. We operate on the unjustified assumption that the two are congruent, and then we spend vast amounts of money trying to investigate the animal model, often without bothering to test our assumptions by constantly referring back to the original disease in humans.

These unexplored assumptions are the fundamental flaws in any animal model scenario. The animal rights campaigners are justified in pointing out that there is little rationale for using animal models which frequently simply draw attention and funds away from the careful investigation of the human condition. The Castalian establishment is wrong in not drawing attention to the unjustified assumption of congruence in most cases of animal experimentation on disease models . . . What can be done to reduce the risk of isolated self-consistency? First, there must be a recognition that in the last analysis the human disease itself must be studied in human subjects. It is at least arguable that if we devoted as much effort to the human disease as we do to unvalidated models, then we might be much further forward in understanding. If we are to have any confidence our models are valid, then we must know at least as much about the diseases we investigate as the models we use. [11]

Horrobin was not an animal right campaigner. Neither are the other scientists quoted in this blog. Lovell-Badge’s claim that animal models have routinely prevented toxic drugs from coming to human trials is without foundation. It is yet another example of the vivisection activist making claims and hoping that society will just take his word for it instead of asking him to provide proof of the claim. Such actions are anti-science, anti-reason, and anti-human.

I will continue my analysis in part III.


1.         Khanna, I (2012) Drug discovery in pharmaceutical industry: productivity challenges and trends. Drug Discovery Today 17:1088-1102. 10.1016/j.drudis.2012.05.007.

2.         Shaffer, C (2012) Safety Through Sequencing. Drug Discovery & Development January 30

3.         Ledford, H (2012) Success through cooperation. Nature, August 24

4.         Cressey, D (2011) Traditional drug-discovery model ripe for reform. Nature 471:17-18. 471017a [pii] 10.1038/471017a.

5.         Reuters (2011) U.S. to develop chip that tests if a drug is toxic. Reuters, October 6 - .To5AMnPaixF

6.         Gura, T (1997) Cancer Models: Systems for identifying new drugs are often faulty. Science 278:1041-1042.

7.         Dresser, R (2009) First-in-human trial participants: not a vulnerable population, but vulnerable nonetheless. The Journal of law, medicine & ethics : a journal of the American Society of Law, Medicine & Ethics 37:38-50. 10.1111/j.1748-720X.2009.00349.x. 2692671.

8.         Lavery, JV (2011) How can institutional review boards best interpret preclinical data? PLoS medicine 8:e1001011. 10.1371/journal.pmed.1001011. 3050913.

9.         Anderson, J, J Kimmelman (2010) Extending Clinical Equipoise to Phase I Trials Involving Patients: Unresolved Problems. Kennedy Inst Ethics J 20:79-81.

10.       Chapman, AR (2011) Addressing the Ethical Challenges of First-in-Human Trials. J Clinic Res Bioeth 2:113. 10.4172/2155-9627.1000113.

11.       Horrobin, DF (2003) Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat Rev Drug Discov 2:151-154. 10.1038/nrd1012 nrd1012 [pii].


Popular Video