At The Amazing Meeting 8, an auction was held for a guest rogue spot on the podcast The Skeptics’ Guide to the Universe (SGU). I won and am the guest on the current podcast, which can be downloaded from iTunes where it is dated as 2-5-11 and from the above link where it is labelled episode #290 on 1-31-11.
I have recommended the SGU podcast many times because it is an excellent, entertaining, and easy way to learn more about science and critical thinking. This is not to say that SGU agrees with everything I say. I am sure they don’t. But SGU is a very good resource and I recommend that everyone listen regularly.
I thoroughly enjoyed my participation in the podcast! The SGU rogues (Jay, Bob, and Evan) were very friendly and helpful and I found the questions from co-host Dr Steve Novella to be very fair and well thought out. He allowed me to explain my position and asked appropriate follow-up questions. We had a limited amount of time so my answers were by necessity brief and incomplete.
But that is what blogs and books are for! I want to here go into a little more detail in answer to some of the questions. None of the below should be misconstrued as implying that Dr Novella asked misleading questions or that he was anything but gracious in allowing me to have my say. Dr Novella conducted a very good and fair interview! But in any time-limited discussion some things get left unsaid or unanswered and that is the reason for the following. (I hope the reader will listen to the podcast before reading further.)
People who actually understand the basics of our position (not everyone that criticizes our position understands the basics, but Dr Novella does) eventually get around to asking whether we are throwing the baby out with the bathwater. Dr Novella asked if we were creating a false dichotomy similar to asking whether a modality like surgery either works or it doesn’t. This is a legitamate question and stems from the fact that we are indeed saying that the animal model per se is not predictive for human response to drugs and disease. We base this on the empirical evidence and place the evidence in the context of evolutionary biology and complex systems. I refer to the context that we find in evolutionary biology and complex systems as the theory that allows us to interpret the empirical evidence. (Some philosophers of science will take issue with this specific use of the word theory however, since others use it in this fashion, so will I.)
A theory like the Theory of Evolution or the Theory of Relativity is not a law of science. There are laws in science, for example Boyle’s law, the laws of thermodynamics and so forth (despite opinions to the contrary; see the first comment). These laws cannot be violated in the universe as we know it. A theory on the other hand allows us to put things in perspective and informs, or even allows predictions, about the future and or explains observations. Knowledge of evolutionary biology and complex systems allows us to explain the empirical evidence (that animal models per se consistently fail to predict drugs and disease response in humans) as well as allows us to make predictions about the future (that animal models will continue to do this).
So, the question that usually arises is something like this: “How do you know that a very specific animal model, say of disease Q, will not give predictive results?” The questioner is not asking about the animal model per se but about one very specific example. There are several parts to the answer to this question.
1. We don’t know with 100% certainty! Theories are not laws, but they do give some idea of what to expect and what we expect is that the odds against an animal model predicting human response to drugs and or disease are very great. So, if a funding agency (e.g. NIH) is trying to decide whether to fund research using animal model of disease Q based on the probability that it will yield results that predict human response, our theory says they should not. Funding something with such a small probability of success is a waste of money.
If a vast majority of animal models are not predictive for drugs and disease response and someone comes along and claims that his model is, then the burden of proof is on him to show data consistent with that claim. Data! Not anecdotes and not retrospective correlations. Further, as we explain in both Animal Models in Light of Evolutionand FAQs About the Use of Animals in Science: A handbook for the scientifically perplexed before a technique, practice, or modality can qualify as predictive, it must have an established track record. It must have a history. So one or two or three correct correlations or outcomes with humans does not make an animal model predictive. (More on this in a minute.)
2. Moreover, even if the data cannot be obtained using any other research method, the harm, vis-à-vis: a) the fact that the information from the animal studies might be misleading (which they often are); b) the continued funding of a research method known to have major problems (thus at least partially condoning and justifying it); and c) the fact that the resources could go elsewhere, must be weighed against the low probability of predicting human response. (I realize that some would include here the harm to the animals.) The animal model will produce data, that is not in question. The question is, how relevant will the data be for humans. (This is of course assuming the research makes the claim for being predictive and hence is not basic research, which makes no such claim.)
3. But lets assume that animal model of disease Q has been shown to reproduce human signs of the disease and at least some of the pathology. Shouldn’t we expect it to also reproduce human response to treatment as well as the rest of the pathology? Because animals are evolved complex systems the answer is that we should not have that expectation. Here’s why.
Complexity is in some ways opposite to the concept of reductionism. Ernst Mayr defines reductionism as: “The belief that the higher levels of integration of a complex system can be fully explained through a knowledge of the smallest components.” [(Mayr 2002) p290] Mayr also pointed out that complexity has been around since Alex Novikoff described it in 1947.
Alex Novikoff (1947), however, spelled out in considerable detail why an explanation of living organisms has to be holistic. "What are wholes on one level become parts on a higher one . . . both parts and holes are material entities, and integration results from the interaction of parts as a consequence of their properties." Holism, since it rejects reduction, "does not regard living organisms as machines made of a multitude of discrete parts (physico-chemical units), removable like pistons of an engine and capable of description without regard to the system from which they are removed." Owing to the interaction of the parts, a description of the isolated parts fails to convey the properties of the system as a whole. It is the organization of these parts that controls the entire system. [(Mayr 1998) p18]
On a similar note, the biologist AG Cairns-Smith pointed out:
Subsystems are highly interlocked…[P]rotein are needed to make catalysts, yet catalysts are needed to make proteins. Nucleic acids are needed to make proteins, yet proteins are needed to make nucleic acids. Proteins and lipids are needed to make membranes, yet membranes are needed to provide protection for all the chemical processes going on in a cell. The whole is presupposed by all the parts. The inter-locking is tight and critical. At the centre everything depends on everything. [(Cairns-Smith 1986) p39]
Again, the animal may in fact respond as do humans to all aspects of a disease and treatment but given that we are dealing with two different complex systems with all that that implies, the odds are decidedly against it. Complex systems are composed of modules and just because an animal possess one module that reproduces certain aspects of human response does not mean that all the other relevant modules will do the same or that the modules will interact in the fashion they do in humans. There are a lot of variable here. Components of complex animal systems, like genes, gene networks, pathways, and organs, are not like pistons. They must be examined in the context of the whole. Hence the field of systems biology and personalized medicine and their emphasis on gene regulation, gene networks, modifier genes, and so forth.
4. Almost every animal modeler that I have spoken to about using animals as predictive models has stated (privately) that claiming predictive ability from animal models is nonsense. EXCEPT, for the research that he is conducting. Everybody else is really using animals merely in basic research, but his animal model of whatever is the real deal and he has shown correlation with human data that does imply predictability and so on. There are a couple of problems with this. First, I should point out that the claimed predictive ability has not been published in the peer-reviewed literature much less been reproduced. We are, in fact, being asked to take the scientist’s word for this. In science, that is a nonstarter.
Prove your claim or back off or be considered a crank. Second, the person making the claim has a vested interest in the claim being true. This does not ipso facto mean the claim is false but should raise some red flags. Third, when almost everyone says that 99% of the time the use of animals as predictive models does not work, that means that essentially everyone else agrees that researcher X’s animal model is not predictive, regardless of researcher X’s opinion. So, if essentially everyone agrees that researcher X’s model is not predictive, we should at least consider that when deciding whether to believe researcher X. Granted, it is possible that no one understands researcher X’s research as well as researcher X, but there is overlap in research, especially research in the same general field—like mammalian biology—so that there are other scientists in a good position to judge things like this.
But the problem gets worse, as review articles will eventually turn up in the scientific literature pointing out both the positive aspects of the animal model that researcher X uses as well as the bad aspects. As it turns out there are important causal disanalogies between the model and the human. Invariably, when I have taken the time to actually search the literature on some specific animal model, I have found that the rosy picture painted by the animal modeler was far from the truth. I really do not think all these guys are lying; I think they just love what they do, have egos, and, like all humans, their judgment is clouded by these things. (There are exceptions of course.)
So, in the final analysis, maybe animal model of Q does in fact predict human response to drugs and disease (and not is merely reverse engineered to reproduce specific aspects) but the burden of proof for this is on the claimant and such proof does not include the researcher’s promise that such a thing is true. The claim needs to be quantified in terms of positive predictive value and so forth and be in line with what is the accepted definition of prediction in science.
5. In light of the myriad times animal models have failed to be predictive in other fields and areas, what does the researcher propose as being different in his models that makes it predictive? The same principles of evolution and complex systems still apply. Is the researcher really proposing that in this instance all the modules of a complex system are interacting exactly like they interact in humans? Such a thing is possible but it is extremely improbable (note allusion to The Hitchhiker's Guide to the Galaxy).
I hope the reader now appreciates why the surgery example differs from the animal model question. 1. Empirical evidence has been collected on various surgeries and some of them are no longer performed. For example, trephination, unless broadly defined so as to include as a means of relieving a subdural hematoma, is not a valid procedure hence is no longer performed. On the other hand, it is rare that empirical evidence is formally collected on animal models. But when it has been, the models have been shown to fail as predictive practices. 2. There is no theory that predicts that surgery, as an entity will fail. (After the germ theory of disease, if a surgeon performed surgery without sterile technique, we could say that a theory existed that predicted the surgery would fail secondary to the patient dying from infection. While not every surgery has been tested this way, we nonetheless accept this principle.) There is a theory, in the form of current knowledge from complex systems and evolution and which we develop in Animal Models in Light of Evolution, which predicts that animal models will fail to predict human response to drugs and disease. While this theory does not have the weight of the 2nd law of thermodynamics, it does have a long history of successful predictions and thus should be seriously considered before granting money to researchers that claim their animal model will predict human response to drugs and or disease.
The following from Massimo Pigliucci is relevant:
Perhaps the most appropriate question for us in this chapter, which can be applied to all the pseudoscience we have briefly examined, is the one that Nature reporter Lucy Odling-Smee asked in the piece on the closing of the PEAR lab: "how permissive should science be of research that doesn't fit a standard theoretical framework, if the methods used are scientific?" In other words, how many times do we have to show that alleged instances of telepathy, clairvoyance, telekinesis, ufos, ghosts, psychic abilities, astrological forces, and the rest of the shebang can better be explained by perfectly normal means? There is, of course, no simple answer to this question, as the appropriate amount of resources and time to devote to the investigation of fringe science depends on how important the claims would be if they turned out to be true, how many times they have been disproved before, and, frankly, how limited the resources of scientists and universities are in practice. We hear repeated calls from ufologists, parapsychologists, psychics, and astrologers to keep an open mind, and there are certainly examples of skeptics who close the door to further investigation a bit too quickly, as in the case of the Campeche UFOs recounted above. But, as astronomer Carl Sagan once aptly put it, you do not want to keep your mind so open that your brain is likely to fall out. (Pigliucci 2010) p82-3
There is no theoretical framework to believe that animal models will allow prediction of human response to drugs and disease. As I have said, what seems like a thousand times, this does not mean animal models cannot be used in basic science or in other ways; it only means that the probability of animals predicting drug and disease response in humans is orders of magnitude lower than what is acceptable in science. A person claiming that a specific animal model or animal models in general can predict human response has a very large burden of proof.
Just as an interview does not allow an in-depth exploration of a topic, so too a blog is, by necessity, going to leave much out. Shanks and I discuss all of the above in detail in Animal Models in Light of Evolutionand, in a manner more accessible to the nonscientist, in FAQs About the Use of Animals in Science: A handbook for the scientifically perplexed.
My exchange with Dr Novella was one of colleagues discussing a controversial topic. No ad hominems, no straw man arguments, just two professionals honestly attempting to see where they agreed and where they disagreed then letting the listeners decide whether they want to purse the subject further. I hope the listeners of SGU will pursue the subject as I think that, considering the resources being consumed and the fact that harm has come from the claim that animal models are predictive, some critical examination is justified.
Thanks again to SGU. You guys are great! Keep up the good work.
Cairns-Smith, A G. 1986. Seven Clues to the Origin of Life: A Scientific Detective Story: Cambridge University Press.
Mayr, Ernst. 1998. This Is Biology: The Science of the Living World: Belknap Press.
———. 2002. What evolution Is: Basic Books.
Pigliucci, Massimo. 2010. Nonsense on Stilts: How to Tell Science from Bunk. Chicago: University of Chicago Press.