Robert Shea on Evidence Based Policy's Impact and Potential

Bob Hahn: Good morning, my name is Bob Hahn and I’m a Senior Fellow at the Technology Policy Institute. It’s my pleasure this morning to be interviewing my good friend and colleague Robert Shea. Robert played an important role as a commissioner on the Evidence Based Policy Commission, where he had the misfortune of serving with me. Notwithstanding that, he worked on this commission very hard, and we’re going to talk a little bit about the commission today. Prior to that, he chaired the National Academy of Public Administration, which puts out a lot of great publications on public administration, of all things. He did some distinguished service at OMB, the Office of Management and Budget, and he worked for several years, probably more than he cares to admit, on the Hill. He’s currently a principal at Grant Thornton. And Robert, can you just spend two seconds telling us a little bit about what you do at Grant Thornton? And then we’ll jump into the exciting subject of evidence based policy.

Robert Shea: Sure. Thank you, doctor. It’s good to be with you today and delighted that we have a chance to chew on evidence based policy making. I lead what’s called the Strategy Service Line at Grant Thornton Public Sector, and I help federal, state, and local government entities measure and improve their performance.

Hahn: Great. So that may tie us in, I’m guessing, to evidence based policy, but for those people who live outside the beltway or have another life, perhaps we could start by answering the question, at least one of us, what is this thing called evidence-based policy?

Shea: Yeah, we could probably do a whole show on just that question, but the way I think about it is increasing the extent to which laws, regulations, policies are made based on evidence of what works. Generally, those kinds of things are intended to achieve an important public policy objective, and you want them to be most effectively tailored to what we know works and to do that you need a bigger and bigger body of evidence about whether or not the practices we’ve employed in the past have made a difference.

Hahn: So that’s a great, great answer. Kind of an abstract academic answer. Can you give me a concrete example where this stuff might be used or how it might help?

Shea: Sure. The gold standard methodology generally employed to study an intervention is a randomized controlled trial, where you would isolate what you’re doing from other factors. You would apply the intervention randomly to one group and measure their performance against another random group and see whether or not what you were doing worked. The most common example of this is in pharmaceuticals, where you, after a series of tests, only after a series of tests have proven safe and effective, will that drug be made available to the public. But those methodologies are increasingly applied in social services and beyond. Whether it’s the extent to which our policies are improving the health of underprivileged, increasing home ownership, reducing homelessness, any number of examples we can use to show how evidence-based policy making has impacted policymaking and moved the dial on performance.

Hahn: So how does that differ from how things work in Washington or other places around the world today? I mean, don’t politicians and civil servants use evidence? What’s the big deal here?

Shea: Yeah, you must’ve gotten this a lot too when you told people you worked on the commission on evidence-based policymaking, oh my god, we need a commission for that? That’s not what we do every day? No, employing evidence to make policy is an unnatural act. Today, politics, emotion, anecdote are more likely to drive policy than evidence. This whole movement, the evidence movement is all about trying to more and more get policymakers to look at data and evidence, and use it in their policy making. Today, if you looked at the response to the coronavirus, or the issues around equity and policing, the simple correlation or emotion are likely to get a rise out of the population and drive policymakers to want to do something. Don’t just stand there, pass a law. So this movement is intended to change what we know is not happening enough today, and that is get lawmakers, political leaders, policymakers, to look at the evidence before they make important policy decisions.

Hahn: So you mentioned COVID-19. So policymakers are under the gun here, because we got a virus that’s basically shut down the country or our legislators and politicians have decided to put things on hold. What should we be doing to use evidence here? And is this a problem where, we’re in such a grave situation that we don’t want to wait for the evidence to come in? How should I think about that?

Shea: Yes, you should think about it in all those ways. I mean, there’s an enormous healthcare-industrial complex that’s been studying both the risks of pandemic and what to do about them. There’s going to be a long tail from this crisis in which we study what worked and what didn’t. And hopefully we’ll apply those lessons in the future. But how good were our models to tell us the extent to which we would be impacted once we saw the rise in COVID-19 cases pop up abroad, but then eventually on our shores, once that happened, what did we know about what interventions were most likely to bend the curve? We’ve talked a lot about bending the curve and we’ve talked about innumerable tactics that would help us do that. One of the reasons I’m talking to you from my basement today. Then you learn about masks, are masks effective for spreading the coronavirus? Do they increase social distancing or do they make people bring their guard down? CDC officials warned that the risk of wearing masks was people would feel they didn’t need to social distance since they had that barrier. But a recent study suggests that wearing a mask actually makes people walk away from you. So it actually increases social distancing. And then of course you’ve got treatment and vaccines. Those are things in which science is applying rigorous methodologies to ensure their safety and efficacy, but the media doesn’t have similar restraints. It can report on evidence of various treatments and the potential for a vaccine in ways that muddy the waters. And so what the real need here is for trusted sources of genuine, reliable data and evidence on which decisions can be based. And again, trust in those forces has declined because the myriad ways these things are being reported don’t necessarily help us distinguish fact from fiction.

Hahn: All right, well, I like both fact and fiction [Laughter]

Shea: I do too, just not when I’m trying to figure out whether or not I’m going to get a communicable disease.

Hahn: So let me press you on this virus stuff a little bit, cause I’d like to understand it better. So are you saying that, broadly speaking, evidence-based policy can help us deal with the virus in real time and can also help provide lessons for the future? Or is it just providing lessons for the future?

Shea: Yeah, I’m humbled that you suggest that listening to me can help you understand better because I remind you, you’ve taught me more about this than I could teach you in the rest of my life. But we talked about wearing a mask or social distancing. Our policymakers govern one of the most diverse populations on the planet and our culture is one of touching and embracing. So moving people to wear a mask, to stay in their homes, when they go out to social distance, those are things that are uncommon, against our culture. So getting people to understand the import of that, to abandon their routines, their cultural norms and do these things. Those are things that what you studied so long, behavioral economics, can be brought to bear, to driving the right behavior. So we can study in real time what practices are working, getting people to deploy these practices, and then you can study their impact by increase in COVID-19 infections and death.

Hahn: Got it. So you’re saying there’s kind of a potential win-win here. We might be able to learn some good stuff in the short term to address the pandemic. And, hopefully, if somebody writes this down and uses it, we might be able to do something over time as well. Is that more or less what you’re saying?

Shea: That’s what I’m saying. Do you agree? I mean, that’s the real potential here. 

Hahn: I mean, just going back to your drug example where you’ve got maybe the placebo and the drug and whatever, and seeing if there’s a difference in reducing illness or reducing the spread of the pandemic, I think that stuff has to be done in real time, but I also think through publications and just writing this stuff down, we can learn and maybe address pandemics more intelligently in the future.

Shea: Some of the heroes here are the individuals who stepped up to participate in vaccine trials. Those people are being injected with things that may or may not work to prevent the virus in order to save millions of people long into the future.

Hahn: So let me ask you another, perhaps dumb, question. Suppose we have the best evidence policy based blip-blop that you can imagine, you have all the resources you need for it, and you produce all this great evidence. How do you get the people who are in charge to actually use this stuff?

Shea: Yeah, that’s a tough nut to crack. I remember being in the OMB when an agency proposed spending money to retrofit exhaust pipes on school buses, the theory being kids were sitting around, sucking up bus fumes to their detriment. It sounded like a great proposal, certainly intuitive. And as we were considering the proposal, some smart aleck felt the need to bring up the fact that there was zero evidence that there was any health impact to students, kids, being exposed to bus exhaust. In the end, though, the optics of that proposal, were just too attractive for the policy makers. And we nonetheless proposed spending millions of dollars to retrofit school buses across the United States. And that’s an example of where evidence was brought to bear on the debate, policymakers considered it and made a different decision. In my view, that’s not a terrible outcome in the drive to inject evidence into policymaking. They heard the evidence, considered it, and made a policymaking decision. Nothing is going to force policy makers to make decisions. We’re not going to automate democracy through evidence, but our job is to increase the availability of it, to help people understand it, that is to translate findings in such a way that they’re accessible to a broad audience. And the policymakers are exposed to this evidence in such a way that they have to hear it while they’re considering evidence. The climate we’re in today, where we are consuming so much information in such an emotional time makes it really hard to sift strong evidence from weak evidence. And I think one of the things that our community, you can call the evidence movement a community, is to figure out how we can distill those things, how we can put a good housekeeping seal of approval on what constitutes rigorous evidence, on which policy makers can reliably make decisions.

Hahn: So you mentioned trust in the beginning a few minutes ago, who do you trust? And then you mentioned one of my favorite things, which people may or may not know, which is the good housekeeping seal of approval. What should we be doing in this regard? Do we need to have something out there that gives the good evidence based policy seal of approval, or are we fine the way it is, having multiple respected voices and universities and think tanks around the country, or is there a problem there in trust, and is it related to evidence based policy or is it much bigger?

Shea: This is a problem. Trust in government has been on the decline for decades and it’s now at its lowest point in our history. I remind people, our nation’s government was born out of mistrust of government. So it’s in our nature, to some extent, but you do want to spend some time strengthening trust in important institutions. You and I spent a lot of time thinking about the statistical infrastructure across government. And so I think the statistical agencies, those that spend their livelihoods producing data about our country, I think they deserve our trust. And so the extent to which we can burnish their reputation as trusted agents of data, we should do that. I say this, knowing that the Bureau of Labor Statistics just made a major goof in reporting unemployment statistics. There are also increasingly clearing houses, evidence clearing houses, where you can go to find studies that show whether or not a specific intervention or activity was proven effective when studied rigorously. I think those are tools that can apply an objective standard to a set of facts and help the reader understand what use can be made of that evidence. Rarely do we get evidence that says, if you do this, it will produce, reliably, this exact result. There are a lot of nuances to this thing, but the more and more we can get evidence that things tend to produce better outcomes, the better off we’ll be to the extent those things influenced decision making.

Hahn: That was a mouthful. So what I hear you saying is, or what I may be saying, if I don’t put words in your mouth, is there may be institutions out there that already do this. One that comes to in a different context is the Congressional Budget Office or CBO. They’ll try to scrub the budget numbers to give you their best projections of what they think is going to happen to the economy over the next five years, based on some economic model and the Bureau of Labor Statistics will put out statistics about wages or whatever they do. I guess what I’m asking is, do you think there’s a need for, dare I say it, another government agency or another government group, you’ve had a lot of experience in government, or are we doing quite fine the way things are? A number of groups and I see a bunch of websites in the private sector that address all bunch of different kinds of evidence. Sometimes they’re in universities. Sometimes they’re just some person who decides to create a website in his or her basement or whatever. So getting back to this issue of trust, should we just let a thousand flowers bloom, because it’s relatively easy to collect this information or whatever, what should we be doing to get at this good housekeeping seal of approval, or are things fine the way they are, or it’s just a tough question.

Shea: Yes, all of those things are true. I’m loathe to recommend a new government bureaucracy to get at this problem. There are institutions that could fill the role of refereeing evidence, if you will. The Congressional Budget Office is certainly one of them. Congressional Research Service is another. The Government Accountability Office is still another. You also have the Chief Statistician in the US Office of Management and Budget, or the Bureau of the Census. Those are some logical entities where you might house this referee. The job would be huge. I recall an assignment when we inventoried the government’s programs aimed at improving STEM education across the United States, science technology, engineering, and math. And we came up with about 117 studies of program effectiveness. We found about 10 of them used a reliable methodology and only three of those showed any positive impact, any statistically significant positive impact. So that took a year or so to get done. So this is not a small task. But I do think some arbiter of what constitutes rigorous evidence is needed because the voices clamoring to tell you that they’ve got the answer, and that their data is the most reliable is going to crowd out those folks who have the most concrete, reliable evidence available.

Hahn: So most commissions that you hear about in Washington, or elsewhere, get together for a year or two, they write a report. And my impression is that report goes into the circular file, which is a nice word for the garbage can. What was it about this commission that the report that they issued, or that we issued, led to a law. And can you tell us a little bit about that process and, two, about the law?

Shea: In less than 30 minutes, and you know, you could answer this as well as I can or better. I think one of the factors that led to its success is its bipartisan roots. As you’ll recall, the commission was created out of a bipartisan relationship between Senator Patty Murray and then-Speaker Paul Ryan. And then we were a genial group of do-gooders who spent a little bit over a year studying every facet of this problem. And as our recommendations were emerging, Congress was following it closely. So they knew what elements of our recommendations were most easily translatable into enactable law. So as we were finishing up, they were drafting what became known as the Foundations for Evidence Based Policy Making Act, which they kindly referred to as a down payment on our recommendations, didn’t include a lot of the tough ones, but it did create the Evidence Officers, Chief Data Officers, Chief Statistical Officials. It requires agencies to produce a learning agenda, what are the big questions that the agency needs answered in order to better fulfill their mission capacity assessments? So soon we’ll know to what extent agencies have existing evidence building capacities, or they need to build them. And then an evaluation plan, how are they going to conduct evaluations in such a way that are focused and help answer those big questions? All of those strengthen the governance of evidence building, create the potential for building a greater body of evidence. Then the challenge will be how do we inject that into policymaking, but we’ve certainly set the stage for a dramatic improvement in this area.

Hahn: That sounds great to me. Where should we go from here? 

Shea: So, I think following agency implementation of the evidence act is going to be really important. Things are likely to distract agencies. Something like a global pandemic might make people look away from doing the things they need to do to build an evidence culture in their organizations. So reminding people that this is important, not just in the regular context, but in a global emergency, like the one we find ourselves in today. Congress has a role; it should be funding these efforts. I don’t think agencies got a lot of new dollars to meet these responsibilities so they can’t do it without additional resources. Congress can give them those resources. One of the most important things Congress can do is ask questions, demonstrate that there’s a demand for the evidence and that they’ll use it when they get it. And then of course, we’re coming on a presidential election, as if there weren’t enough turmoil going on, and you’ve got two opposing views on how to run the government, but both of them have shown support in the past for evidence based policy making. So getting them on the record so that this movement is sustained, whether or not we have a presidential transition in the fall and winter, should be really important.

Hahn: So I’ve always viewed the words evidence-based policy as something that cuts across party lines and is hard to be against. And the question in my mind is how do you mobilize support for this among John Q public? And, also, how do you, as you pointed out, I don’t think there were budget lines in the initial enabling legislation for evidence based policy. So if you had to sort of map out a strategy, for those of us who kind of like this idea, over, say, the next two years, what should we be doing?

Shea: I think I see things, at least for the most part, at the federal level, but there’s growing interest, probably advanced interest, in these areas at the state and local level. So I would, as you consider where to cast your vote, do it based on the extent to which candidates demonstrate an understanding that they ought to be making decisions based on what’s proven to work, not based on gut. And I’m talking about things that are outside the hot button issues that are traditionally debated, but you know how to educate kids, how to reduce crime, how to improve health and welfare, how to reduce traffic fatalities. These are things for which there is a growing body of evidence, and the extent to which people understand that that’s where, how they ought to be making decisions. We ought to see that at the elected and government official level across the board.

Hahn: I’m reminded of this beer commercial, I don’t know if I’m allowed to say this on podcasts or radio or whatever, Miller had a beer commercial, less filling tastes great. Are you saying we should be looking and people who care about policy should be looking for things where they can get more mileage for less gas or better bang for your buck and things like that, and it doesn’t necessarily have to be politicized and those may be some win-wins for us?

Shea: I think that’s right. I’m pollyannaish about this, but today, it’s sad to say, the vast majority of programs in which we invest at the federal level, when evaluated rigorously, are shown not to have the impact that we hope. The more and more we can move investment away from those programs and into ones that have an evidentiary basis, you’re going to get more results, where in the past you got none. So overnight you’re going to increase your return on investment. I think that’s an attractive bipartisan argument. And the assertion that programs don’t work is going to rub some people the wrong way, because they’ll think it’s an anti-government message. There are some who suggest when you want to move investment into what’s proven to work that you’re a big government interventionist. Whether it’s big or small, my point of view is everybody ought to want government to work, and work better and better. So I do think this is an attractive bipartisan argument for the vast majority of Americans.

Hahn: So you stay away from what you want the size of government to be in principle, and you just say, we’re going to provide you with a set of tools and a set of outputs that enable you to make more effective decisions, get more bang for your buck, what you said in terms of return on investment. Is that what you’re arguing?

Shea: That’s right. Again, a little pollyannaish. I know that there are a lot of other factors that go into decision making and a lot of baggage people bring to governing that they had to take on in order to get where they were. But major policy priorities, all the way down to low hanging fruit, evidence can make a big difference in all of those areas.

Hahn: Well, you’ve persuaded me.

Shea: I had an easy target. 

Hahn: Robert, I want to thank you for joining us today. It’s been a pleasure as always, and we’ve been listening to Robert Shea who’s a principal at Grant Thornton and head of the strategy group there and who I had the privilege of serving with on the Evidence Based Policy Commission. Have a great day.

Shea: You too. 

Hahn: Thanks a lot, Robert.