It was the Good Cop’s turn today to take a pot shot at the solid empirical work of Michael Simkovic and Frank McIntyre in their important new paper, The Economic Value of a New Law Degree (powerpoints here).
Of course, by Good Cop I mean Brian Tamanaha of Washington University who earned that little sobriquet from the Cato Institute, the right wing “think” tank, when he spoke there alongside a former “law and literature” (yeah, I don’t know what that is either) professor from University of Colorado who earned from Cato the title of “Bad Cop” in legal education reform, presumably because of his oddly testy demeanor and willingness to use slander and smears as part of his repertoire.
Well, today, however, the Good Cop apparently took the gloves off until he was hit on the knuckles by Professor Seto of Loyola, who is acknowledged by the authors for his comments in advance of its posting. Professor Seto noted on TaxProf that Tamanaha had peppered his account with a wide range of personal attacks aimed at questioning the motives of the authors of the JD value study. As Professor Seto pointed out if there are questions about the data or methodology then one has to come up with genuine errors or go out and prove the data wrong. Professor Seto wrote:
“Prof. Tamanaha needs to tone things down: “puffed up exaggeration, a brazen bluff”, “sloppy, ad hoc”, “chest-pounding confidence”. He explicitly attacks Simkovic’s and McIntyre’s motivations: “Rather than conduct an open-minded inquiry into the economic value of a law degree, it appears that S&B were hell bent on proving that a law degree pays off handsomely for nearly all law grads.” I would have expected ad hominem attacks from Above the Law. They seem out of place in scholarly debate.
“Simkovic and McIntyre’s analysis is the most sophisticated and careful we have seen to date, by orders of magnitude. For the first time, they apply the methods of labor economics to a problem previously addressed largely through amateurish, back-of-the-envelope approximations. Whether they have correctly applied those methods, time will tell. But calling them names and questioning their motivations is unprofessional.
“The standard scholarly response to an empirical study with which one does not agree is a more sophisticated empirical study that takes into account one’s critiques. I look forward to reading Prof. Tamanaha’s study.”
Now it seems Tamanaha admits he went too far and in deference to what he called Professor Seto’s professionalism he seems to pull back but then just repeats his claim that the study was indeed “sloppy”:
Ted, Out of respect for your professionalism, I removed from the Balkinization post “brazen bluff” and “sloppy.” The other terms were entirely appropriate and descriptively accurate and well supported. (And what they did was sloppy, by the way, but I can see that it is an insensitive way to put it.)
Of course it is not Ted to whom Tamanaha owes an apology and retraction, it is the authors of the study. Their work is anything but sloppy. It was peer reviewed prior to its acceptance at the American Law and Economics Conference at Vanderbilt earlier this year. It benefited from the comments prior to its posting on SSRN of several senior leading scholars in several fields in addition to Seto, including Bernie Black and Kate Litvak at Northwestern, both respected empirical law and economics figures; Eric Rasmusen at the Kelley School of Business at Indiana University, a MIT trained economist; Ronald Ehrenberg, a senior labor economist at Cornell’s School of Industrial and Labor Relations; and Louis Kaplow, a Harvard-trained lawyer and economist who has long held an endowed chair at Harvard. Paul Oyer at Stanford and Peter Arcidiacono at Duke, both well known labor economists, also reviewed the work.
If the work of the authors is sloppy then presumably Tamanaha thinks the reviewing done by these individuals was also sloppy.
Now, it is clear that Tamanaha is genuinely threatened by the working paper’s conclusions and it is understandable that he should be. He, like his Colorado colleague, have completely re-engineered their careers to jump on the law school attack bandwagon. If the JD Value study is right then they have made a colossal error.
I am certain that the authors will reply in greater detail to the random comments made by Tamanaha but let me note a few things.
First, Tamanaha admits that he has only a “crude” understanding of the empirical methodology that the authors employ. That should have been a warning to him. Instead of jumping to conclusions he should have consulted someone on his faculty who does have the technical background to help him understand the argument. As I have learned over the last six or seven years in my work on market microstructure with my co-author, an economist, there is a level of subtlety to empirical analysis that can easily escape the uninitiated. Instead, Tamanaha irresponsibly jumped into the fray with a range of personal attacks questioning the motives of the authors.
This has now led Eric Rasmusen of Indiana to comment wryly: “Anyone who is dismissive of Simkovic and McIntyre should read their working paper and rethink. If you don’t understand it, you’re not smart enough to dismiss it.”
I am also reminded of my reaction when I read and commented on Tamanaha’s book Failing Law Schools. I noted:
“Tamanaha relies on very generalized data sets that do not provide persuasive evidence of misbehavior by any individual school, fails to test for counter-factual explanations, and draws conclusions that are only one among several possible explanations for the current situation….[Instead of performing] careful econometric analysis…Tamanaha chose not to do that work but to rely…on broad-brush statistical material and anecdotal information that allows us, in fact, to come to several possible explanations for the situation we face.”
It is one thing to be a conceptual or theoretical scholar who relies on serious qualitative methodology. That is a perfectly acceptable alternative. But it is another to try to dismiss the work of someone with whose methodology you are entirely unfamiliar.
Second, Tamanaha has conceded apparently a key argument I have been putting forward for some months and that the Simkovic and McIntyre paper provisionally confirms: there is a cyclical effect to the earnings premium associated with the JD. That was my instinct and yet it was met with derision and worse, from the critics camp. Now comes Tamanaha who demands that the authors extend their substantial data base to include what? Yet another cycle! He wants the authors to push back the start date of their research to 1992 so that it encompasses the LBO/real estate crash of the early 1990s. Yet, when I pointed to that very event on Faculty Lounge six months ago it was dismissed as an attempt to demonstrate the legal market was cyclical.
And now it is at the centerpiece of Tamanaha’s response to the new research. How can that be? Tamanaha’s whole case has been based on the idea that we were now in a new normal. That it is different this time. That, as I put it in another post, some kind of neutron bomb has hit the legal profession to cause the collapse in JD jobs of late that will now be permanent. In other words, that there is no historic long lasting cyclical pattern at work but something secular and structural.
Of course, Tamanaha behaves as if the Simkovic and McIntyre paper does not deal with this issue. Yet he ignores the evidence they present of the upturn in the earnings premium recently. And despite his suggestion that somehow the research does not show the impact of the recent downturn he has to accept the fact that for all JD holders who graduated from 1996 to 2008, and who worked through the last five years of that downturn, their earnings premium over non-JD holders has, once again, consistent with past behavior, turned north.
Equally important is the fact that the earnings premium they document accrues over a lifetime. Pushing back the data base to rope in a few years more of JDs starting out their careers is likely not significant (statistically – and I hope Tamanaha understands what that means.) I made a similar point in an earlier post about the initial argument by some critics that the inability of the authors to look at the prospects of post 2008 JD graduates. It likely is not material because the amount earned in the early years of a career by JDs is not where the premium gets a large boost. That takes time. The authors point out that it “peaks late” in life. (See Slide 7.)
This makes intuitive sense because, of course, only a small number of students leave school, join a major NY or Silicon Valley law firm and then go on to become full equity partners in that firm. A substantial number of lawyers still enter practice by setting up their own shop or joining a small practice. It takes time to learn the ropes and even more importantly build up a reputation in their community and in the local courthouse.
In any case, the research DOES capture the impact of the very steep and deep downturn that hit the profession after the dotcom crash, the downturn that followed the Asian financial crisis (when it was “pens down” on every deal I was working on for two months), AS WELL AS the impact of 2008. The latter point is not well understood, certainly not by Tamanaha. Since Simkovic and McIntyre track lifetime earnings of all JD holders who entered the profession from 1996 going forward they capture the impact of both of those events. The point is that the premium over BA’s held, if volatile, because of the advantages of a JD to those individuals. Those individuals were working,or not, through 2011 and yet the premium was positive at the median and mean to the tune of hundreds of thousands of dollars.
And there is an even more fundamental objection to the idea of going back to 1992: why stop there? Does Tamanaha not recall what led to the LBO/real estate crash in the post-92 period? It was the boom of the Reagan era in the late 1980s. So why don’t we push the research back to 1982. A simple glance at the up and down nature of the post-Carter era American economy demonstrates the cyclical nature of the situation. It is inherent in the nature of capitalism and of course lawyers are not immune from this volatility.
There is more to what Tamanaha has to say, in particular, his amateurish attempt to wade into the complex area of “ability sorting.” This is another of the areas that the critics have jumped on to question the research results. Again, this is an entirely separate discipline with which it is clear Tamanaha not only has very little prior exposure but about which he seems not to have attempted prior to commenting to gain some understanding. It is clear that the authors have spent a good deal of time considering its effects and attempting to control for them. They control for the impact of non-JD college graduates for the possible effect of the JDs having talents and backgrounds that would lead to higher incomes aside from the degree.
Even an admirer of Tamanaha’s anecdotal work noted the problem in a comment:
Brian,I have been one of your supporters.
However, your points here are wrongs. I read carefully the slides published by Simkovic and your claim about the 10% premium doesn’t hold. The authors have studied 3 groups: JDs (a), bachelor degree holders who look like JDs based on different variables (b), and the broader bachelor degree holder population (c). The authors have compared group (a) to group (b) in estimating the premium. You are saying that they have compared (a) to (c). So your conclusions are pointless, because that’s not what the study did….I understand that your points about the problem with law school are touching the heart of many students who are having a rough time. However it’s important that you don’t mislead people on this important issue.
I am beginning to wonder if Tamanaha even understands what it means in empirical analysis to “control” for something.
The professionally valid scientific response when someone suspects that results might not be valid or that the wrong methodology has been applied is to take the same data and try to replicate those results. I challenge Tamanaha to do that, until then he should hold his tongue.