‘Economic Value’ Paper a Mistrial at Best, Part 2 of 2

[This is the second part of my response to the Simkovic and McIntyre article, "The Economic Value of a Law Degree" ("Economic Value") and the authors' response on The Am Law Daily. The first part of this response can be found here.]

Criticism 4:

“And [Leichter] largely ignores our extensive discussion of ability sorting in Section II.I. of The Economic Value of a Law Degree, while claiming that we did not consider these issues.”

False: I never discussed ability sorting in my article, so I ignored the “extensive discussion” in Section II.I. because it was irrelevant to my argument. Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

It’s a pity none of the few—”though by no means all”—of the misrepresentations Simkovic and McIntyre charged me with were meritorious. Oh well.

Criticism 5:

“Studies by labor economists have found that increased earnings from education generally extend across multiple occupations.”

False: Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

Also, applying the results of “studies by labor economists” about increased earnings for people with various types of education to legal education is an ecological fallacy. The authors must demonstrate that legal education alone is versatile.

Criticism 6:

“Any ability biases remaining after our controls may be offset by an equally important source of bias that Harper, Leichter, and many other critics have ignored: less educated, lower income households systematically over-report earnings and more educated, higher income ones under-report. Less-educated survey respondents tend to forget periods of unemployment, while more- educated households tend to forget end-of-year bonuses and SIPP caps maximum reported earnings to preserve confidentiality. This has been documented in SIPP surveys and it biases our results downward, making them too low, since the comparison group of bachelor’s recipients is systematically lower income than the law graduates.”

False: I ignore the bias because it’s not relevant to my argument: Simkovic and McIntyre never demonstrated that legal education causes higher earnings.

(Implied) Criticism 7:

“What does all this say about law school reform? On the one hand, untested reforms should not be rushed through primarily based on a sense of desperation or crisis, or a belief that changes couldn’t possibly make things any worse. On the other hand, the high returns to law school do not suggest that legal education can’t be improved—some reforms may be beneficial, and should be considered on their merits. Our preference, as always, would be to test proffered theories empirically as best as possible and we look forward to future work that does.”

False: “Economic Value” says nothing about law school reform whatsoever. If anything, it gives us a good reason to eliminate the subsidies to legal education because those subsidies make law school more expensive. Without unlimited, nondischargeable government loans law schools would have to reduce their tuition costs to remain solvent, which would increase the “premium” Simkovic and McIntyre believe they’ve discovered.

Bonus! Criticism 8:

“Leicther’s [sic] description of our take on BLS projections is lifted from context, since we note that even BLS economists are skeptical of these sorts of projections.”

The “context” is in footnote 10 where Simkovic and McIntyre write:

“BLS and other labor economists have cautioned against using occupational employment projections to guide educational investment.”

False: I frequently write about BLS employment projections, so these statements raised an eyebrow because the BLS has featured them prominently in its Occupational Outlook Handbook (OOH) for many years. The OOH pretty clearly targets a non-academic audience on the desirability of various careers, and nowhere does it caution readers to not rely on the 10-year projections, e.g. in the FAQs page. Here’s a taste of what the 1996-97 edition said about lawyers:

“Even though jobs for lawyers are expected to increase rapidly, competition for job openings should continue to be keen because of the large numbers graduating from law school each year. During the 1970s, the annual number of law school graduates more than doubled, outpacing the rapid growth of jobs. Growth in the yearly number of law school graduates tapered off during the 1980s, but again increased in the early 1990s. The high number of graduates will strain the economy’s capacity to absorb them. Although graduates with superior academic records from well-regarded law schools will continue to enjoy good opportunities, most graduates will encounter competition for jobs. As in the past, some graduates may have to accept positions in areas outside their field of interest or for which they feel they are overqualified. They may have to enter jobs for which legal training is an asset but not normally a requirement. For example, banks, insurance firms, real estate companies, government agencies, and other organizations seek law graduates to fill many administrative, managerial, and business positions.” [Emphasis added]

It’s curious that the BLS would publicly characterize the job outlook for lawyers with such lucid direness yet be skeptical of such projections. Looking more closely at footnote 10 explains why: Simkovic and McIntyre use a see-cite to two articles to infer a general consensus within labor economics that the employment projections are unreliable. In neither article is that inference reasonable:

  • The article by Horrigan, for example, merely says that it’s difficult to predict how employers will respond to labor shortages. This is not exactly a sacred holy war against the OOH’s use of employment projections for career-guidance purposes.
  • This leaves the article by Neumark, Johnson, and Cuellar Mejia, which asks whether the retiring baby-boomer cohort will lead to a skills shortage. Again not a holy war against the employment projections that can sustain a see-cite inference. Neumark et al. also admit in footnote 4 that they are using a human capital theory in their paper, not a signaling theory. The authors do not discuss the sheepskin effect or elasticity of labor demand in their paper at all. Moreover, they do not consider dispersal of earnings.

At best Simkovic and McIntyre can say that the authors of these two papers alone don’t think people should rely on the employment projections when choosing a career—even that’s a stretch—but there are still a few other problems:

  • We don’t know if these authors speak for labor economics as a discipline.
  • If there is such a consensus, we don’t know if other economists and disciplines disagree with good cause.
  • The labor economists Simkovic and McIntyre cite in their paper and elsewhere (including David Card from the previous post) tend to prefer human capital theories of higher education over signaling theories, and they rarely point out that elasticity of labor demand is a significant independent variable in occupational wages. In short, it’s reasonable to believe that they are either biased or that their human capital theory is wrong. Such things happen in academics.

Finally, again the authors demonstrate that they don’t really believe their own theory. They should be saying that the projections don’t matter because law degrees increase human capital for all occupations, not because they’re unreliable.

Wait, I take it back! People shouldn’t rely on the BLS’ lawyer employment projections after all…

BLS OOH Lawyer Employment Projections

(Source: OOH, BLS Monthly Labor Review)

…Because they regularly overestimate the number of lawyer jobs that will be created. Oh, but if law school graduates can’t get professional, demand inelastic jobs, then I guess we can’t say applying to law school is rational.

Conclusion

This isn’t meant to be an exhaustive response to “Economic Value” and its authors’ subsequent comments; it could go on and on and on forever, but I’d really like to stop here. Although through its errors their paper has indirectly taught me much about signaling theory and the factors that influence an occupation’s wages, I hope Simkovic and McIntyre leave legal education to more diligent researchers. It would be a tragedy if someone applied to law school based on “Economic Value,” and shame on anyone who uses the paper to induce anyone to do so. But one reason I’ve extended this topic much longer than it deserved is that I have a degree in the social sciences, and “Economic Value” is an excellent example of how not to conduct social science research. To sum up, here’s a list of the authors’ research offenses both in the paper and subsequently:

(1)  Not informing readers of alternative theories discussed in their own citations (“sheepskin effect” – In other papers this one might be minor, but it makes a difference in this case because the audience is not labor economists and readers are unlikely to read the article’s citations as I have.)

(2)  Not falsifying the alternative theories discussed in their own citations before conducting their calculations (“sheepskin effect” – Scientific method, shmientific method.)

(3)  Failing to acknowledge how limitations in their theory impair their methodology’s applicability to the real world (omitting elasticity of labor demand from their calculations by insisting occupations are not “pretreatment covariates”)

(4)  Avoiding discussion of potential intermediate causes (how can human capital theory possibly explain why those 25-year-old law grads in figure 4 go from earning $20,000 to $80,000 in a few years? – This might be attributable to simple oversight and it’s come up late, but given some of the errors here (esp. #5) on balance I think not.)

(5)  Not discussing the demographic content of the data they found for readers (who were the 1,382 law grads in the SIPP data? how many were in each age bracket (25-34, 35-44, 45-54, 55-64)? what jobs did the 25-34-year-olds have? what’s the frequency distribution of non-lawyer jobs? – This one is really egregious, even if it doesn’t have a material effect on the study’s outcome. Omitting a discussion of the data is a hallmark of bad social science research.)

(6)  Falsely charging critics with misrepresentations (at this point, this one’s minor)

(7)  Misrepresenting their critics’ claims (e.g. “distributional data” in Criticism 3)

(8)  Evading critics’ meritorious claims (non sequitur, path dependency, credential inflation—and these are just mine)

(9)  Misusing see-cites to misrepresent a consensus in a discipline readers may know little about (reliability of BLS employment projections)

(10)  Claiming their findings are applicable to real-world policy issues when they are not (legal education reform – Other papers might be able to get away with this, but not “Economic Value.”)

(11)  Omitting independent causal variables from their calculations, but conveniently using them to prove other points (elasticity of labor demand isn’t a “pretreatment covariate,” but it is evidence that lawyers aren’t going to suffer reduced earnings or be structurally unemployed in the future due to off-shoring/automation)

Any remaining advocates of the “law degree premium” or the bottleneck/backlog argument are invited to tell us what demand-inelastic, professional jobs law school graduates will eventually obtain. When will they realize their “premia”? Will it pay off their debts? Will they be able to have families, go on vacations, and save for retirement?

Law school optimists need to answer these questions. Unemployed law grads can’t wait.

About these ads

One Response

  1. Nicely done! I strongly suspect Simkovic is going to write a whingeing letter to wordpress to admonish you for your inappropriate tone, publish the letter in a comment-free post on Leiter’s site, and then disappear. You know, the same way he did for Steven Harper’s criticisms.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 120 other followers

%d bloggers like this: