‘Economic Value’ Paper a Mistrial at Best, Part 2 of 2

[This is the second part of my response to the Simkovic and McIntyre article, “The Economic Value of a Law Degree” (“Economic Value”) and the authors’ response on The Am Law Daily. The first part of this response can be found here.]

Criticism 4:

“And [Leichter] largely ignores our extensive discussion of ability sorting in Section II.I. of The Economic Value of a Law Degree, while claiming that we did not consider these issues.”

False: I never discussed ability sorting in my article, so I ignored the “extensive discussion” in Section II.I. because it was irrelevant to my argument. Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

It’s a pity none of the few—”though by no means all”—of the misrepresentations Simkovic and McIntyre charged me with were meritorious. Oh well.

Criticism 5:

“Studies by labor economists have found that increased earnings from education generally extend across multiple occupations.”

False: Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

Also, applying the results of “studies by labor economists” about increased earnings for people with various types of education to legal education is an ecological fallacy. The authors must demonstrate that legal education alone is versatile.

Criticism 6:

“Any ability biases remaining after our controls may be offset by an equally important source of bias that Harper, Leichter, and many other critics have ignored: less educated, lower income households systematically over-report earnings and more educated, higher income ones under-report. Less-educated survey respondents tend to forget periods of unemployment, while more- educated households tend to forget end-of-year bonuses and SIPP caps maximum reported earnings to preserve confidentiality. This has been documented in SIPP surveys and it biases our results downward, making them too low, since the comparison group of bachelor’s recipients is systematically lower income than the law graduates.”

False: I ignore the bias because it’s not relevant to my argument: Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

(Implied) Criticism 7:

“What does all this say about law school reform? On the one hand, untested reforms should not be rushed through primarily based on a sense of desperation or crisis, or a belief that changes couldn’t possibly make things any worse. On the other hand, the high returns to law school do not suggest that legal education can’t be improved—some reforms may be beneficial, and should be considered on their merits. Our preference, as always, would be to test proffered theories empirically as best as possible and we look forward to future work that does.”

False: “Economic Value” says nothing about law school reform whatsoever. If anything, it gives us a good reason to eliminate the subsidies to legal education because those subsidies make law school more expensive. Without unlimited, nondischargeable government loans law schools would have to reduce their tuition costs to remain solvent, which would increase the “premium” Simkovic and McIntyre believe they’ve discovered.

Bonus! Criticism 8:

“Leicther’s [sic] description of our take on BLS projections is lifted from context, since we note that even BLS economists are skeptical of these sorts of projections.”

The “context” is in footnote 10 where Simkovic and McIntyre write:

“BLS and other labor economists have cautioned against using occupational employment projections to guide educational investment.”

False: I frequently write about BLS employment projections, so these statements raised an eyebrow because the BLS has featured them prominently in its Occupational Outlook Handbook (OOH) for many years. The OOH pretty clearly targets a non-academic audience on the desirability of various careers, and nowhere does it caution readers to not rely on the 10-year projections, e.g. in the FAQs page. That’s not to say it encourages blind reliance, but it would take a substantial showing to persuade me that the BLS puts in so much effort in a project it doesn’t really believe in.

Here’s a taste of what the 1996-97 edition said about lawyers:

“Even though jobs for lawyers are expected to increase rapidly, competition for job openings should continue to be keen because of the large numbers graduating from law school each year. During the 1970s, the annual number of law school graduates more than doubled, outpacing the rapid growth of jobs. Growth in the yearly number of law school graduates tapered off during the 1980s, but again increased in the early 1990s. The high number of graduates will strain the economy’s capacity to absorb them. Although graduates with superior academic records from well-regarded law schools will continue to enjoy good opportunities, most graduates will encounter competition for jobs. As in the past, some graduates may have to accept positions in areas outside their field of interest or for which they feel they are overqualified. They may have to enter jobs for which legal training is an asset but not normally a requirement. For example, banks, insurance firms, real estate companies, government agencies, and other organizations seek law graduates to fill many administrative, managerial, and business positions.” [Emphasis added]

It’s curious that the BLS would publicly characterize the job outlook for lawyers with such lucid direness yet be “skeptical” of such projections. Looking more closely at footnote 10 explains why: Simkovic and McIntyre use a see-cite to just two articles to infer a general consensus within labor economics that the employment projections are unreliable. Neither article supports that inference:

• The first quote, from an article by Horrigan, merely says that it’s difficult to predict how employers will respond to labor shortages in specific occupations. In both his abstract and conclusion he writes, “These projections form the basis for providing career advice to individuals entering the job market, changing careers, or making further educational and training choices.” This statement, appearing prominently twice, directly contradicts the inference Simkovic and McIntyre are claiming to draw from the quotation. Talk about taking things out of context!
• This leaves the article by Neumark, Johnson, and Cuellar Mejia, which asks whether the retiring baby-boomer cohort will lead to a skills shortage. The quoted passage only says that the authors didn’t find higher underemployment of highly educated workers in their particular analysis of BLS projections and other data. In no way can their findings be imputed to all of labor economics. In fact, Neumark et al. address “conflicting evidence” produced by Harrington and Sum, who found significant evidence of “mal-employment” of highly educated people working in low-skill occupations and earning a scant premium as a result.

At best Simkovic and McIntyre can say that three labor economists in one paper disagreed with aspects of the BLS projections and that there’s disagreement among labor economists on the topic of mal-employment of highly educated workers. We still don’t know whether there is a consensus on the reliability of BLS projections or upon which side that consensus falls. Additionally, even if there is such a consensus, it isn’t necessarily correct. The labor economists Simkovic and McIntyre cite in their paper and others (including David Card from the previous post) tend to prefer human capital theories of higher education over signaling theories, and they rarely point out that elasticity of labor demand is a significant independent variable in occupational wages. In short, it’s reasonable to believe that they are either biased or that their human capital theory is wrong. Such things happen in academics.

Finally, again the authors demonstrate that they don’t really believe their own theory. They should be saying that the projections don’t matter because law degrees increase human capital for all occupations, not because they’re unreliable.

Wait, I take it back! People shouldn’t rely on the BLS’ lawyer employment projections after all…

(Source: OOH, BLS Monthly Labor Review)

…Because they regularly overestimate the number of lawyer jobs that will be created. Oh, but if law school graduates can’t get professional, demand inelastic jobs, then I guess we can’t say applying to law school is rational.

Conclusion

This isn’t meant to be an exhaustive response to “Economic Value” and its authors’ subsequent comments; it could go on and on and on forever, but I’d really like to stop here. Although through its errors their paper has indirectly taught me much about signaling theory and the factors that influence an occupation’s wages, I hope Simkovic and McIntyre leave legal education to more diligent researchers. It would be a tragedy if someone applied to law school based on “Economic Value,” and shame on anyone who uses the paper to induce anyone to do so. But one reason I’ve extended this topic much longer than it deserved is that I have a degree in the social sciences, and “Economic Value” is an excellent example of how not to conduct social science research. To sum up, here’s a list of the authors’ research offenses both in the paper and subsequently:

(1)  Not informing readers of alternative theories discussed in their own citations (“sheepskin effect” – In other papers this one might be minor, but it makes a difference in this case because the audience is not labor economists and readers are unlikely to read the article’s citations as I have.)

(2)  Not falsifying the alternative theories discussed in their own citations before conducting their calculations (“sheepskin effect” – Scientific method, shmientific method.)

(3)  Failing to acknowledge how limitations in their theory impair their methodology’s applicability to the real world (omitting elasticity of labor demand from their calculations by insisting occupations are not “pretreatment covariates”)

(4)  Avoiding discussion of potential intermediate causes (how can human capital theory possibly explain why those 25-year-old law grads in figure 4 go from earning $20,000 to $80,000 in a few years? – This might be attributable to simple oversight and it’s come up late, but given some of the errors here (esp. #5) on balance I think not.)

(5)  Not discussing the demographic content of the data they found for readers (who were the 1,382 law grads in the SIPP data? how many were in each age bracket (25-34, 35-44, 45-54, 55-64)? what jobs did the 25-34-year-olds have? what’s the frequency distribution of non-lawyer jobs? – This one is really egregious, even if it doesn’t have a material effect on the study’s outcome. Omitting a discussion of the data is a hallmark of bad social science research.)

(6)  Falsely charging critics with misrepresentations (at this point, this one’s minor)

(7)  Misrepresenting their critics’ claims (e.g. “distributional data” in Criticism 3)

(8)  Evading critics’ meritorious claims (non sequitur, path dependency, credential inflation—and these are just mine)

(9)  Misusing see-cites to misrepresent a consensus in a discipline readers may know little about (reliability of BLS employment projections)

(10)  Claiming their findings are applicable to real-world policy issues when they are not (legal education reform – Other papers might be able to get away with this, but not “Economic Value.”)

(11)  Omitting independent causal variables from their calculations, but conveniently using them to prove other points (elasticity of labor demand isn’t a “pretreatment covariate,” but it is evidence that lawyers aren’t going to suffer reduced earnings or be structurally unemployed in the future due to off-shoring/automation)

Any remaining advocates of the “law degree premium” or the bottleneck/backlog argument are invited to tell us what demand-inelastic, professional jobs law school graduates will eventually obtain. When will they realize their “premia”? Will it pay off their debts? Will they be able to have families, go on vacations, and save for retirement?

Law school optimists need to answer these questions. Unemployed law grads can’t wait.