Professor’s Research Shows Many Criminal Justice Reforms Don’t Work

Most interventions and reforms in criminal justice have little enduring impact, argues University of Virginia School of Law professor Megan T. Stevenson in a new article that reviews 50 years of data.

Or, as one blogger put it in discussing her paper, “Nothing works. Now with proof!”

In “Cause, Effect, and the Structure of the Social World,” which was published in December in the Boston University Law Review, Stevenson acknowledges that the “nothing works” trope goes back at least as far as the 1970s, when criminologist Robert Martinson said as much with regard to prisoner rehabilitation. (He later walked back that claim, she said.)

But Stevenson delves into more than five decades of randomized control trials to argue that it’s almost impossible to engineer lasting change in the criminal justice and rehabilitation system. She examined interventions both liberal and conservative, ranging from job training programs to “swift, certain, and fair” sanctioning.

Stevenson’s analysis extends beyond criminal justice, challenging the prevalent “engineer’s view” of the social world — a perspective that has long suggested social processes are both structured and manipulable.

Here she explains the implications, including how to think about social change.

Your article seems to challenge the effectiveness of nearly all criminal justice reforms, which goes to the heart of what you do. What led you to look at this, and what do you hope readers take away?

First of all, I want to clarify something. My claim about the ineffectiveness of criminal justice reforms pertains only to certain types of reforms: those that get evaluated by randomized controlled trials. Randomized controlled trials hold all else constant while you tweak on one aspect alone. They are, by definition, systems-conserving reforms in that they require manipulating one aspect of a person’s experience while leaving the rest of the system intact. Reforms of this type tend to be of relatively limited scope. So, what I hope readers take away from this study is that this type of limited scope, systems-conserving intervention has little impact.

You acknowledged that your arguments stem from personal experience that has transformed the way you see the world. Was there any specific moment or realization during your academic career that triggered this shift?

I was at a conference about seven years ago and saw a presentation about the results of a large-scale randomized controlled trial on job training programs. There were five different job training programs across five different sites, all tailored towards people reentering society after prison or other difficult-to-employ groups. And these programs basically had no lasting effect on either their employment or their likelihood of committing another crime. To me, this was really discouraging, because this is the type of supportive intervention that I would hope would make a difference in people’s lives. Anyway, this is probably the first time I started thinking seriously about how difficult it was to shift people’s trajectories with limited-scope interventions. Since then, it’s been a consistent pattern I’ve noticed in the literature — when they are evaluated by gold standard research designs, most interventions are shown to have little effect or the results don’t replicate in other settings.

Does this paper upend some of your own past research and suggestions for reform? How do you feel about that?

Honestly, a lot of my past research has been about interventions that failed to bring about meaningful change. For instance, people were really optimistic about the use of risk assessments (statistical crime prediction tools) for use at bail or sentencing. There were some policy simulations that predicted that this would be a real panacea — by making smarter decisions, we would be able to dramatically lower crime without increasing incarceration or vice versa. My studies showed that, in real life, this didn’t happen. The adoption of risk assessment tools does not generally bring meaningful benefits in terms of lower incarceration rates or crime.

The article questions the “engineer’s view” of the social world. How has this view influenced reform efforts, and how has it hung on so long despite scholars and evidence challenging its validity?

I use the phrase “engineer’s view” to describe the belief that the social world is amenable to manipulation and control, and that we can achieve social change with minimal risk by using randomized control trials to identify a set of best practices that can then be widely adopted. My argument is that, at least when it comes to the type of limited-scope interventions that get evaluated by such trials, the engineer’s view is a myth. But it’s an extraordinarily compelling myth! People want to have control. They want science to provide the answers about how to engineer a better future. This is at the core of the “evidence-based” reform movement, which has been extraordinarily influential.

Your write about “stabilizing forces” that limit the impact of reforms. What are some examples of these forces and how they operate in the context of criminal justice interventions?

Well, take job training programs, for instance. You can provide someone recently released from prison with help building a resume. You can teach them a few skills or provide them some temporary subsidized employment. But the major structural factors that made it difficult for them to find or keep a job previously are still there. They still have a conviction on their record. They may have low levels of education, or substance abuse or mental health problems. They may lack the kind of networks that are necessary to find or maintain employment. And so forth.

What can policymakers do to create meaningful and lasting outcomes? What should they consider when designing and implementing interventions?

I see three paths. First, there is directly alleviating harm. If people are hungry, feed them. If they need shelter, give them a place to stay. Second, there is systemic reform. Instead of job-training programs, you could try to address the various structural problems that make it hard for people recently released from prison to find employment. This type of systemic reform falls outside of the scope of my argument because it’s not a limited scope, systems-conserving intervention — it would be too complex and large-scale to evaluate with a randomized controlled trial. For that same reason, however, you’d be proceeding with uncertainty. Systemic reform is not the type of thing you can pilot test before adopting at scale. Finally, there is incremental change. Fifty years of evidence shows us that limited-scope interventions are generally not going to have large or lasting effects. But they may have small effects — too small to be detected by a randomized controlled trial. So it’s possible that we are taking small steps in the right direction. But there’s no guarantee — it’s also possible that we are taking small steps in the wrong direction.

Should we find this depressing?

I personally don’t find it depressing. In a way, I find it a testament to human agency. If it was so easy to change people’s trajectories with limited-scope interventions, then you’d have to wonder why they didn’t already make the changes themselves. I tend to think people are doing the best they can with the opportunities and constraints life has provided for them. The remaining constraints just happen to be sticky. This doesn’t negate free will or human agency, it just forces humility in terms of our ability to engineer change for other people.