The Nature of Baseball Analysis

| by Hardball Times

Tommy Bennett has a thought-provoking piece today at Baseball Prospectus on the nature of baseball analysis. It's a fascinating article that has me thinking.

There exists, I think, a certain set of analysts in baseball who adhere to a sort of logical positivism. That belief is demonstrated by the drive to completely separate outcomes from processes, despite the fact that such cleavage is not actually possible. For example, to eliminate the effect of luck is one of the guiding goals of data-driven performance analysts. Here are two theorems that reflect that goal:

* Pitchers have only limited control over outcomes on balls in play
* Evidence of clutch performance as a repeatable skill does not manifest itself in any measurable statistics

There are others, too, that are more minor. But these two theorems in particular—commonly accepted among sabermetricians and fellow travelers—are widely rejected by analysts and fans more generally. In each of the two cases, well-reasoned, data-backed arguments from very smart people go all the way to the water’s edge. But the data stops short of allowing analysts to defend conclusive statements—dogmas, if you will—like “pitchers have no influence over the rate of hits on balls in play” and “there is no such thing as clutch hitting.” Such certainty would perhaps be desirable—it’d certainly make analysis easier—but it is simply not supported by the data available.

Popular Video

This young teenage singer was shocked when Keith Urban invited her on stage at his concert. A few moments later, he made her wildest dreams come true.

I'm not sure he has the second theorem stated quite correctly. "The Book says: For all practical purposes, a player can be expected to hit equally well in the the clutch as he would be expected to do in an ordinary situation." Tango, Lichtman, and Dolphin state that "the fact that one of three players performs at least .006 [points of wOBA] better or worse in the clutch doesn't mean that we can tell which players have this skill, even when looking at several seasons worth of data" because one would need "7,600 appearances in clutch situations (this would require over 200 seasons in the majors)" to accurately quantify just half of a player's clutch skill. We don't claim that clutch skill doesn't exist or that it doesn't have an impact on the game. It's that we don't believe we can find a way to measure it with any accuracy.

Perhaps that's picking a nit with Tommy's argument. I'm not sure yet. I'm still digesting what Tommy said. I'd wait to write this in response until I had fully digested his thoughts if I didn't suspect I'll be thinking about the nature of baseball knowledge until the day I lay down my pencil.

A true sabermetrician recognizes the limitations of his knowledge and his theories. Yes, there are tendencies toward logical positivism among baseball analysts. I, however, see much more of Karl Popper (hence the title) among the best saberists.

P1 » TT » EE » P2.

Popular Video

This young teenage singer was shocked when Keith Urban invited her on stage at his concert. A few moments later, he made her wildest dreams come true:

Here 'P' stands for 'problem'; 'TT' stands for 'tentative theory'; and 'EE' stands for '(attempted) error-elimination', especially by way of critical discussion. My tetradic schema is an attempt to show that the result of criticism, or of error-elimination, applied to a tentative theory, is as a rule the emergence of a new problem; or, indeed, of several new problems. Problems, after they have been solved and their solutions properly examined, tend to beget problem-children: new problems, often of greater depth and ever greater fertility than the old ones. 'This can be seen especially in the physical sciences; and I suggest that we can best gauge the progress made in any science by the distance in depth and expectedness between P1 and P2: the best tentative theories (and all theories are tentative are those which give rise to the deepest and most unexpected problems.

Problem-children. There is something I like about that term.

Having been trained as a physicist, I am accustomed to responding to baseball questions by proposing and then testing a model. To the extent that the model is helpful in understanding reality, it is accepted and used. However, models are limited and are not themselves reality. They lead to new and deeper questions and better, but still finite and tentative, models.

Implicit in this framework is the idea that the model must be continually tested and probed for weakness and limitations. From Popper, this is the idea of falsification. is easier to find out that a theory is false than to find out that it is true (as I have explained in detail elsewhere). We have even good reasons to think that most of our theories-even our best theories are, strictly speaking, false; for they oversimplify or idealise the facts. Yet a false conjecture may be nearer or less near to the truth. Thus we arrive at the idea of nearness to the truth, or of a better or less good approximation to the truth; that is, at the idea of 'verisimilitude'.

I'm not an expert in philosophy in general or Karl Popper's critical rationalism in particular, but Tommy Bennett's article struck a chord with me this morning. It seemed he had a very good point, and yet, he wasn't quite getting at the way that sabermetric thought is really advanced. Sabermetric thinking may well be pushed out to the masses in logical positivist form, but is that how it is created? Nevertheless, this is an important topic.

Matt Swartz and I recently had a discussion via Twitter about the nature of batted balls. It's something I hope to write about more fully in future. Matt asserted that because we don't measure a persistent skill for pitchers in allowing or not allowing line drives, that pitchers do not have this skill. (Forgive me, Matt, if I've bastardized your position in distilling it here.) I asserted that the physics of the ball-bat collision tells us that pitchers ought to have similar control over the spectrum of batted ball launch angles, including that portion of the spectrum we label "line drives."

Matt was arguing, as best I can tell, from the point of view of pragmatism. I was arguing, I think, for a critical rationalist approach. To quote again from Popper:

Nobody denies that pragmatic usefulness and such matters as predictive power are important. But should there exist something like the correspondence of a theory to the facts, then this would obviously be more important than mere self-consistency, and certainly also much more important than coherence with any earlier knowledge' (or 'belief'); for if a theory corresponds to the facts but does not cohere with some earlier knowledge, then this earlier knowledge should be discarded.

Similarly, if there exists something like the correspondence of theory to the facts, then it is clear that a theory which corresponds to the facts will be as a rule very useful; more useful, qua theory, than a theory which does not correspond to the facts. (On the other hand, it may be very useful for a criminal before a court of justice to cling to a theory which does not correspond to the facts; but as it is not this kind of usefulness which the pragmatists have in mind, their views raise a question which is very awkward for them: I mean the question, 'Useful for whom?'.)

Although I am an opponent of pragmatism as a philosophy of science, I gladly admit that pragmatism has emphasised something very important: the question whether a theory has some application, whether it has, for example, predictive power. Praxis, as I have put it somewhere, is invaluable for the theoretician as a spur and at the same time as a bridle: it is a spur because it suggests new problems to us, and it is a bridle because it may bring us down to earth and to reality if we get lost in over-abstract theoretical flights of our imagination. All this is to be admitted. And yet, it is clear that the pragmatist position will be superseded by a realist position if we can meaningfully say that a statement, or a theory, may or may not correspond to the facts.

Thus the correspondence theory does not deny the importance of the coherence and pragmatist theories, though it does imply that they are not good enough. On the other hand, the coherence and pragmatist theories assert the impossibility or meaninglessness of the correspondence theory.

So without ever mentioning the word 'truth' or asking, 'What does truth mean?' we can see that the central problem of this whole discussion is not the verbal problem of defining 'truth' but the following substantial problem: can there be such a thing as a statement or a theory which corresponds to the facts, or which does not correspond to the facts?

The question of approach, of philosophy, of the nature of knowledge, is an important one. It shapes how we arrive at our theories, how we use them, and how we communicate our findings. My hat's off to Tommy for broaching the subject.

Read more great baseball stuff at The Hardball Times.