Some vivisection activists pretend that the notion of predictive modalities is unheard of in medical science. I addressed this in Predicting Responses in Complex Systems. Part II. In parts IV and V, I want to give some examples of the fallacious reasoning and misrepresentations I have already described, along with others, including the supposed ignorance regarding predictive modalities by some in the medical profession. I will be quoting from Dr Harriett Hall’s essay: Learning from Animals: Evolutionary Medicine with a Twist. I mentioned this essay in my September 14 blog, if it sounds familiar. Quotes from other blogs will be referenced.
In medical science as well as other fields of science, it is well known and uncontroversial that some methods for predicting outcomes work well and others do not. The usefulness of these methods depends on what the specific situation requires. Shanks and I stated in Animal Models in Light of Evolution:
The comparison with coin tossing is appropriate in the sense that a gambler with a taste for games of chance, might play a game with such odds (50% chance of success), and may well prefer this to a game with only a 10% chance of success (other things being equal). Gambling of this kind is not a luxury for those whose business is human health, safety and well-being—animal investigators themselves make this very point. Where the costs of errors are high (in terms of death or serious injury, not to mention the enormous social costs associated with the misclassification of substances as harmful), a modality’s success rate of 90% may fail to be adequate. . . . If I predict the right outcome 51% of the time in Las Vegas, in the long run I would be rich. p257-8
Suffice it to say, getting the right answer 51% of the time is not adequate in medical science, or in most fields of science. The question then arises: “How can we measure (quantify) how a modality performs?” Technically, the way to quantify how well a modality performs is by employing positive predictive value (PPV) and negative predictive value (NPV) or some variation on that theme. Again, this is well known in science in general. The recent conviction of Italian geologists illustrates this concept. The full story has been covered in multiple places (including some that I will discuss) but the short version is that the geologists did not instruct residents to leave the area when minor earthquake activity was detected. A major earthquake followed the minor quakes and people died. Steven Novella, MD, explains this in his essay Guilty Verdict for Italian Earthquake Scientists, states:
The relevant science seems pretty clear. Low level seismic activity rarely is followed by a large earthquake, and most large earthquakes are not preceded by such minor tremors. Therefore the predictive value of the preceding tremors was very low. It was therefore entirely accurate for the scientists to say that the risk of a large quake was “unlikely.”
In this case, the modality being used to judge the likelihood of a major earthquake was a series of minor earthquakes. As I have stated, PPV and NPV can be used to assess, not just medical tests, but any modality or practice that is used to evaluate the probability of an event or outcome where the reality of that event is eventually determined (the earthquake either happened or it did not). Novella acknowledges such modalities have predictive value as opposed to being mere hypothesis generators when he uses the phrase “predictive value.”
If one makes a statement of probability, and the low level probability outcome occurs, that does not make one wrong. This concept comes up in medicine all the time. Physicians come up with a “differential diagnosis” – a list of possible diagnoses in order of likelihood. A physician may act on the diagnosis that is 95% likely to be true, but for 1 patient in 20 the 95% diagnosis will be wrong. That does not mean the differential diagnosis was wrong. Sometimes, even, a patient turns out to have a rare disease. Saying that the diagnosis is unlikely prior to the diagnosis being made would not be wrong. (Emphasis added.)
So again, we see probability being used as a guide to acting on information. If there is a probability of 0.95 (95%) that the patient has disease X, then pursuing that diagnosis in the form of relevant tests would be the best way to help the patient. As opposed to ordering tests that would determine whether the last disease on the differential diagnosis list was the problem.
Later, in the comments section, Novella adds: “If you correctly characterize an outcome as unlikely, eventually the unlikely will happen anyway. That doesn’t mean you were wrong.” If animal models have on average a PPV of 0.3 for toxicity, that means 30% of the time there will be an animal model that exhibits the same toxicity seen in some humans. Once again, this is immaterial as the modality or practice only has a predictive value of 30%. Cherry picking the ones that the practice got right does not make the practice predictive. In the earthquake example, this would be similar to using this one time when minor quake activity was followed by a major quake in order to claim that minor quake activity proves a major quake is coming. Such cherry picking would be an example of the fallacy of insufficient statistics. Yet, this is exactly what vivisection activists do when pointing out that an animal reacted the same as humans in terms of an adverse drug event.
Also addressing the topic of the Italian geologists, David Gorski, MD, states:
. . . the science of earthquake prediction is much more uncertain than most medical diagnoses. . . . In a report in Nature from a couple of years ago, after it was decided that the scientists would have to stand trial, a retrospective analysis of seismic activity in Italy was cited that found that a medium-sized shock in a swarm forecasts a major event within several days only 2% of the time.
This is an excellent example of predictive value. A modality used to predict earthquakes that is correct only 2% of the time is not even helpful, much less a predictive modality. Nevertheless, two times out of one hundred there will be a major earthquake after a series of minor quakes. Just as with animal-based toxicity testing, one cannot cherry pick those two instances and claim that they prove the modality works.
Gorski quotes Alan I. Leisher of the AAAS from a letter to the President of Italy: “Years of research, much of it conducted by distinguished seismologists in your own country, have demonstrated that there is no accepted scientific method for earthquake prediction that can be reliably used to warn citizens of an impending disaster.” Note that Leisher does not say: “Gosh darn, even though using minor quakes to predict major quakes is only correct 2% of the time, we should continue to use it as we have no other options.” Contrast this with Calli Arcale, who commented on the Hall essay, Learning from Animals: Evolutionary Medicine with a Twist:
Regarding the validity of animal models, they may not be perfect, but they are among the best models we have. As the old adage goes, all models are wrong; some are useful. That animal models are imperfect is no justification for abandoning them. You don’t quit science just because it’s hard. The key is to understand (and constantly seek to improve the understanding) the strengths and weaknesses of a specific model in a specific application.
The fallacy Calli Arcale is alluding to here is the perfect solution fallacy, meaning that a modality or model does not have to be perfect in order to be useful. I have never denied this. However, in suggesting that I am guilty of the perfect solution fallacy, Calli Arcale is committing the straw man fallacy. If the modality is being used as a predictive modality as opposed to being used as a heuristic, as the presence of minor earthquakes was being used, then it does have to have a high enough PPV and NPV to qualify as predictive. Using minor earthquakes to predict major earthquakes does not have a high enough PPV to be useful, much less qualify as a predictive modality. The same is true of animal models in the above example of toxicity testing. (Other examples could be used as animal models have been examined for numerous predictive purposes in drug and disease research.) If Calli Arcale is saying animal models are useful heuristic models, I have also so stated many times. But the discussion on Hall’s blog was about using animals as predictive models, not as heuristics. Whenever a person addresses an issue that is not part of his opponent’s position, he is committing the fallacy known as ignoratio elenchi (irrelevant thesis). Ignoratio elenchi is frequently employed by the vivisection activist.
Calli Arcale continues:
You don’t quit science just because it’s hard. The key is to understand (and constantly seek to improve the understanding) the strengths and weaknesses of a specific model in a specific application.
This is true but irrelevant as the topic under discussion is not whether models are useful but rather do they get the right answer often enough to qualify as a predictive modality. Insinuating that I am saying that the use of models should be abandoned because such use is "hard," is an ad hominem as well as a straw man.
I will continue to address comments from Dr Hall’s essay as well as what Dr Gorski stated about the Italian geologist decision in part V.