Protecting Privacy and Moving the Evidence Ball Down the Field with Nancy Potok

Robert Hahn:

Hello, and welcome to the Technology Policy Institute’s podcast, Two Think Minimum. I’m your host, Robert Hahn. Today, we’re excited to talk with Nancy Potok as part of our podcast series on evidence-based policy. Dr. Nancy Potok served as the Chief Statistician of the United States until January of this year, 2020. She has over 30 years of leadership experience in the public, nonprofit, and private sectors. Nancy also served as a commissioner on the US Commission on Evidence-Based Policymaking, where I had the privilege of making her acquaintance. She’s also a contributing editor to the Harvard Data Science Review, and you can see her bio online. Welcome, Nancy, and thank you for joining us.

Dr. Nancy Potok:

Thanks for having me, as they say, I’m delighted to be here.

Robert Hahn:

And thank you for staying safe during COVID, and to our audience, we’re doing this remotely. So to get the ball rolling, I was wondering if you might tell us a little bit about evidence-based policy, your interest in this area, and what you mean by the term?

Dr. Nancy Potok:

Sure. I think the way to boil it down to its essence is to say that you want to put information that provides relevant, timely and meaningful insights in the hands of the people who are making decisions that either create policy, or implement policy, so that they have information that’s meaningful in order to have the policies that work the best. Because when you don’t have that information, you may just be going by political influence or by intuition, possibly by experience, but without really having meaningful information that answers some of the big questions, it’s hard to have policies that you know are working.

Robert Hahn:

Okay. So I’m a relatively simple minded economist, and I’m going to try to parse that statement sort of from an economic perspective. It seems to me that what you might be saying is you need two things. You need to have good evidence, and you need to get that evidence to the right people. Is that kind of what you’re saying?

Dr. Nancy Potok:

Yes

Robert Hahn:

Okay. So how do we get the good evidence and what are some examples of that?

Dr. Nancy Potok:

I think there’s some very good examples where this has worked. I wish there were more. One example I think that’s very useful to look at is something that HUD did called the Family Options Study. So HUD is in the business of housing and one of the problems in society that they’re trying to tackle is homelessness, particularly homelessness among families with children. So if we go back a few years to 2017, there were about 150,000 families with children that were homeless in 2017, and HUD wanted to know what are the best interventions that we can do. Should we do voucher programs? Do we put money into more shelters? How do we best help these families tackle this once and for all, not these kind of Band-Aid solutions that sort of bleed money, but maybe never solve the problem. And so they realized they needed access to data from many different sources.

So, there’s something called the National Directory of New Hires, which actually was created and is used to make sure that deadbeat parents are paying child support, so they can garnish their wages. But it’s very valuable because it’s basically employment records for people in the country. There’s HUD’s own housing records. There’s a lot of child welfare data where you can look at outcomes for children, how they’re doing in school, all kinds of things, but those are usually kept at the state and local level. And there are many sources of data that you would want to put together to see a holistic picture of what’s happening with these families, because these are complicated dynamics. So how did that actually, they undertook that and they created very compelling evidence. In fact, it showed that housing subsidies had a very significant and a cost effective impact relative to the other interventions for helping these families get back on their feet and get out of this situation where they didn’t have more permanent shelter. And that resulted in the administration at the time, asking Congress to put $11 billion into housing vouchers to reauthorize this program. So had they not done that study, had they not pulled the information together from multiple sources, they wouldn’t really know what is the most effective. And if you’re going to spend something like $11 billion, I think you want a good sort of cost benefit comparison there to understand, “Is this really going to work or not?”

Robert Hahn:

That’s a fantastic example. Just one or two questions. What is HUD, just for our general audience, and how did that decision-making process work? And do you have any idea what it costs to do a study like that? That sounds like if you’re looking at it after the fact, you’re getting a very good bang for your buck. So is the basic idea that you get some smart folks in a room to design a study like you were talking about at HUD, and then you figure out what it’s going to cost to do this study, you go out in the field and do it, and then you give some of these fine that you share some of these findings with the decision makers, or how does it work?

Dr. Nancy Potok:

You asked many questions! I’ll try to take them one at a time. Yeah. So in this case, let’s start with HUD. It’s the Department of Housing and Urban Development. It’s a federal cabinet level agency and they commissioned this study. So the information got in the hands of the agency, that’s responsible for the program because the agency was the one that sponsored it and then worked with the folks in the U S Office of Management and Budget, which is the agency that’s part of the White House that puts together the president’s complete federal budget request to Congress every year. And so HUD was able to take the information that they went out and got and work that into a budget process. It’s very important, and I think maybe we can come back to that later, but the cost of doing these studies can vary considerably and it’s changing.

It doesn’t really have to be a federal agency that does this sort of once in a lifetime study. What we’re trying to do, what I am advocating, is that we make this a regular way of doing business for all federal agencies on their big-ticket programs that are tackling these major issues across society. But I think also it’s very important not to overlook the role of state and local agencies in this. So, you know, at the local level, and at the state level, is where kind of the rubber meets the road on a lot of these, what I would call, social safety net programs. And so what you want is the States and the federal government working together in sharing data so that they can figure out how best some of these big grant programs can work. But it’s nice if it starts with the people who are decision makers and they say, we want to know this information, and then they find people either within government or academic researchers who can actually do the evaluations for them and do these studies.

Robert Hahn:

Got it. So you mentioned sort of words that warm the cockles of an economist’s heart, or I don’t know if that’s the right metaphor.

Dr. Nancy Potok:

They have hearts?

Robert Hahn:

*Laughter* Touché, but you mentioned the idea of cost benefit analysis. So my understanding is the August institution where you worked as Chief Statistician of the US OMB, the Office of Management and Budget already does, or requires a lot of cost benefit analysis. So how does evidence into this picture, like in your example, with housing vouchers or whatever?

Dr. Nancy Potok:

Well, evidence is quite important because the fact that OMB requires something doesn’t mean that an agency meets the requirement very well. They don’t always do a good job on the cost benefit analysis. And I will tell you, I know a lot of people always think of evidence in terms of social science and some of these safety net programs, but it happens at the office that I was working in also was the office that looks at federal regulation, and cost benefit, of course, plays a huge role in the regulatory scheme of things. And the biggest problem is that agencies really were not doing very good cost benefit analysis in the scientific evidence to support some of the regulations or to support deregulation, just wasn’t there. And it was hard to access and it didn’t have a lot of the characteristics that you’d like to see in a rigorous scientific study.

And I think evidence is scientific evidence, as well as this kind of evidence of what is happening on the ground with people’s lives, from like a sociology and an economic standpoint. So they all really play together. So, I wouldn’t want to leave that out. I can get actually a really important component is the regulatory scheme. And I saw so many bad studies and bad designs and bad evaluations that were being put forth. So I think the government really needs to step up its competency in that area.

Robert Hahn:

So, if you were going to give the government a grade in the area of evidence based policy, my understanding is they’re just starting out, but how would you grade them? And how would you think about the key areas where they might improve their performance since you actually work there?

Dr. Nancy Potok:

So I would, I guess I would separate out a little bit. I would give Congress a C- because I think that even when the evidence is there, they often ignore it for overriding political considerations. I mean, yes, you can have evidence, you can present all these things to Congress, but that may not be the way that they’re making the decisions. So it was very hard to get legislation that requires the evidence. So I don’t think Congress actually does a great job in that area, but it’s a political body. So, it’s not surprising. In the executive branch, I would have higher expectations. Because even though the president is an elected official, and the heads of agencies get appointed by the president, and there is a political agenda, for the most part, you would want decisions to be made based on objective, unbiased information. And most of that information exists, but is grossly underutilized. And I’m not faulting the people in the agencies. The structure of the executive branch is very fragmented, and the resources are not being prioritized either by OMB, or by the leadership of the agencies to get this done. I think there’s a hunger for it. I’ve talked to a lot of rank and file people, you know, scientists and people running programs. They want the evidence, but the cards are stacked against them. It’s very difficult to do in the government environment. I think that’s changing, but slowly, so C to C- at the moment with hope for the future.

Robert Hahn:

So I’m glad I’m not taking the course you’re offering this semester. Cause I can see you’re a tough grader. Let me ask you a question. So there are a lot of people in the agencies who support this. Obviously, we got a law out of Congress. No thanks to me. But I mean, Congress passed a law with strong bipartisan support, which appears to support the gathering and production of evidence that might be useful to agencies. What would be appropriate next steps? I know you’ve done a lot of thinking about this for how agencies can sort of take the bull by the horns, or whatever the right metaphor is, and move the ball down the field.

Dr. Nancy Potok:

Yeah. So Bob, I think, you know that The Foundations for Evidence Based Policymaking Act, which was passed in 2018, the president signed it in January of 2019, took 11 of the evidence based policy making commission’s recommendations and put them into law. They were very important foundational pieces from the recommendations. They didn’t take all of them, so there’s still a ways to go, but what it did was it took some of the recommendations to create more competency and capacity in agencies to do these types of evaluations. And it set up some key positions in the agencies, an evaluation officer, a statistical official, a chief data officer, to really start governing the data that an agency has. Because as I mentioned before, they have a lot of data. They’re not utilizing it to kind of leverage it, to optimize the information that they could get out of it.

So this kind of takes that first step of saying, “Hey, you have to get these people in place and set up a governance structure, identify the data that you have, do an inventory so everyone can see what you have and start to work on some of these quality issues, get some good meta-data in place.” Look at your interoperability and start with the high priority datasets because agencies are just awash in legacy data. And there’s no point. I mean, they don’t even know where to start. So the point is, start with the stuff that’s most important. And how do you know what’s most important? The agencies now have to put together evaluation plans and OMB put out guidance that says, “We want you to have a long-term evaluation plan, so we know that you’re answering the questions that are the big questions for your mission, whether it’s based on regulations, you want to put out whether it’s driving billions of program dollars out that you have grant programs or whatever does that you’re doing.

Dr. Nancy Potok:

What are the key things you need to know that you don’t know, but they had an, a learning agenda and we’re going to tie that to your budget and OMB it, make sure that you do those studies, but in order to do the studies, you need the data and it may be in your agency or maybe that someone else has it, but let’s see what’s out there. So that’s a big start is just sort of getting people organized and getting the right people in place and starting to identify what are these big questions and what data do we need. Then there’s a whole series of things that allow privacy and confidentiality to be protected on some of these studies, which was where the evidence commission was coming from, saying that if you’re going to do these evidence building activities, particularly with sensitive data using individuals, it has to be protected like statistical data is protected. And you need to have an environment where it’s restricted, and it’s not just open data that’s out there. So there’s a lot of building blocks to getting to that. What does the infrastructure look like? How do you protect the data? Who gets access? That’s itself a whole new series of activities.

Robert Hahn:

So that’s some great stuff there. And I want to come back at you in two ways. First, let me see if I understand what’s happening with this law and what the agencies are doing. So I think what you’re saying is someone, maybe OMB, The Office of Management and Budget, is asking the agencies to set priorities and make a plan. And the way that they’re going to try to enforce this, they’re saying, if you do a good job implementing this plan, whatever it is, we’ll give you a bigger budget. Is that right?

Dr. Nancy Potok:

Not necessarily a bigger budget, but a budget that’s tailored to… I mean, that’s the whole idea of sort of cost benefit. It’s not just more dollars. It’s targeting the dollars to be more effective. I mean, there’s billions and billions of dollars going out, and we’re already way in debt, as you know, so you want to get bang for the buck if you’re going to put more money out there.

Robert Hahn:

So, this is music to my ears. But as you and I know, the government has gone through several iterations, shall I say, of trying to improve decision making? Is there any reason to think this is going to work better or worse than earlier efforts, in your opinion?

Dr. Nancy Potok:

I am an optimist, but I am a cautious optimist. I think it, it has the ingredients to work better, but there are a couple of elements not there that are critical that I’m advocating for. One thing that I will mention because I’m coach hearing a task force run by the National Academy of Public Administration that is looking at sort of a whole series of what they call the big challenges, big issues, and this is part of that. So we’re writing a paper, it should come out somewhere around September 11th, in that timeframe. And we have a whole series of recommendations of how OMB needs to act differently to make sure that this happens and really put a priority and a focus on information, evidence building and how information policy can be pushed through the agencies. And it includes looking at these inter-governmental relationships because frequently the role of the States is, is completely overlooked and it’s critical.

That’s one thing that has to happen, it’s got to come as a priority from the very top. The second thing you have to have, in addition to the right people, you have to have the infrastructure to do this. So under the auspices of the Data Foundation and with some assistance from the Sloan Foundation, Nick Hart and I, Nick is the president of the Data Foundation. We coauthored a paper that looks at what is the right structure that we need in government to really enable better access, to allow data from different agencies to be linked together, but at the same time, to enhance transparency and privacy protections and protect the confidentiality around data. So we just came out with a paper and recommendations on how to go about doing that. So I think without support from the top and some reprioritizing at OMB, and without this infrastructure going to be very difficult to make fast progress on this.

Robert Hahn:

So, I looked at the paper that you and Nick did, and Nick was extremely helpful on the evidence based policy commission. And you talk about sort of modernizing the nation’s data infrastructure. So to me, those are big words, data infrastructure. What do you mean by that? And if you had to pick one or two things that are most important for a lay person to know about in that area, what would you say that they are? My understanding is our major concern here is how do we preserve privacy while allowing researchers perhaps like myself to take a closer look at the data to inform decision makers?

Dr. Nancy Potok:

Yes, the data infrastructure is really what are the platforms and the processes around accessing data, whether you’re accessing it as a federal agency or accessing it is an outside researcher. And what we have, what the problem is, is every agency is different. We know that is often the case in the federal government, but different in key ways. One is their willingness to prioritize this activity. And the second is we have still a patchwork of laws that govern data sharing. We have differing levels of resources available to promote this activity. Agencies have legacy data that’s in varying conditions, and varying levels of accessibility, and agencies are really different in their competencies in terms of their ability to protect privacy and their ability to even really think about this and be innovative. So what you want is maybe in these areas that are very difficult for an individual agency to tackle on their own is some kind of an ability to put together a more centralized approach to bringing these competencies together and having a go to place for both the agencies and the researchers.

So they’re not replicating these, some of these very difficult things to do over and over again, and multiple agencies with varying degrees of success. Let’s just try to do this once the right way and make it available. So that’s what we were looking at. If you were going to do that, what would that look like? And we were able to not start from scratch because the Evidence Based Policy Commission had made several of its recommendations laying on the foundation of having something that the commission is, you know, about called the National Secure Data Service that would serve this purpose. And at the same time, the National Academy of Sciences had also put a panel together that was looking at how could we do this just for the statistical agencies, to share statistical data. So Nick and I put some of those ideas together and added our own spin to it, to look at what were some of the key objectives? What were the characteristics? How would you go about doing this, and how would you run it? Where should it be?

Robert Hahn:

Super. I think that’s very helpful in terms of laying out what a data infrastructure might look like and what characteristics you might want. I want to talk a little bit about some concrete examples, which I’m sure you’re familiar with, where we’re all in the midst of this COVID-19 pandemic. My question to you is how might evidence based policy be used to address some of the problems associated with this pandemic, or is it really designed for that?

Dr. Nancy Potok:

No, it could be. You know, it depends on how narrow a view you want to take of the pandemic. Had something like a national secure data service been in place before the pandemic, and had there been more of a concerted effort to standardize some of the health data previously, it’s been going on for decades really, and it has not reached where it should be. So, if you could have stepped that up and had a national secure data service, you could have had a lot more information sharing of consistent data between the state, local, and federal government. I know most people think of that in terms of, you could allocate resources. You could look at hospital beds, that type of thing, but I would go even beyond that. Because some of the things that I’ve been looking at, where there’s been some surveys, first, the data foundation did a survey with NROC, and then the Census Bureau picked up a pulse survey, is the economic impacts on people of the pandemic, and how do we fashion and economic recovery from this.

And so we need information about what happened to businesses, small businesses, large businesses. What’s going on with the safety net programs and food insecurity? I mean, really large percentages of people have been responding to these surveys saying at least one person’s unemployed in the household as a result of the pandemic. That if they had to come up with that kind of standard $400 for emergencies, they couldn’t do it. There’s a lot of mental health issues out there. So recovery is also very key here. It’s not just, how do we get ventilators to the right hospital? It’s a much, much bigger issue. So we need data. For example, from the Department of Agriculture on food stamps, the SNAP program, the I’m with Women and Infants and Children Program, to look at what’s going on with food. We need Medicare and Medicaid information. We need other things from NIH, National Institutes of Health, but we also need the Census Bureau’s information, the Bureau of Labor Statistics, and looking at how employment is playing in. So these are very complex dynamics, and you need data from many places to actually build a body of evidence that says, okay, these are the steps that we should be taking at the federal level and maybe at the state level to move us more rapidly towards recovery, instead of just guessing.

Robert Hahn:

So, in your view, do we have a lot of this data out there and it’s just not being connected or put together in the right ways, or is it really matter that we need to collect a lot more, or both?

Dr. Nancy Potok:

No, it’s there. It just needs to be connected. I mean, if you could connect to unemployment insurance records and you could figure out, how is people’s employment, how is that affecting how many people and in what geographic locations are using the food stamps, and are they going on and off? And how does that correlate to the number of cases in the number of deaths and what happened with the re-openings. You have to tie all that together. All the data actually exists somewhere, but there is not a person who is thinking this through and saying, let’s pull all of this together and look at these various dimensions of what’s going on here, so we can see the dynamics and the whole of society. We’re looking at fragments of information and trying to do a policy that has to be much more holistic if we’re going to come out of this sooner rather than later.

Robert Hahn:

So, when I hear you talk about this, I hear you saying there’s a huge potential payoff to being able to link these data sets like for COVID-19, learning about different aspects of the problem, food insecurity and so forth. Do you have any idea or does anyone have any idea what it would cost to make this happen? Are we talking, billions? Are we talking trillions? Are we talking?

Dr. Nancy Potok:

No, no, no, no. Not at all. Doing this study with existing records is very cost beneficial. It may cost billions of investment to recover. I can’t really speak to sort of what the programs should be or where money should be coming from at the federal or the state level for recovery. Okay? But to actually do the studies is not super expensive. Here’s one, the only example that I can give. So very recently, just late last week, there was a bipartisan bill introduced called the Health Statistics Act, which is a fantastic bill. And it has some elements in it that I think are crucial. So, one of the things it does is it speeds up the standardization of data, and it requires the secretary of HHS to put out what the standards are going to be both for the definitions and interoperability of health data. But then, most importantly, and this is where a lot of efforts fall down, it puts money in for a grant program for the states to make that standardization happen in their systems. And if they don’t comply with the standards, they don’t get the money. So that’s one thing. The other thing it does is it sets up a pilot project at the National Center for Health Statistics, which is inside the CDC to start doing data linkages, to start getting at these issues. And it puts some money in, and it’s not a lot of money to do these kinds of linkages, you know, to take data from other parts, not only of HHS, but to get, maybe get some census data and some BLS data and these other pieces to start looking at these things. And it actually sets up that as a demonstration project, where you can start to prove out some concepts for this national secure data service. So I’m really excited about that. It’s a bipartisan bill. I hope it’s on point that it can find a vehicle to really get enacted because it would be a huge leap forward for health data. And also for understanding some of these dynamics around health in other parts of the economy.

Robert Hahn:

So that was very helpful. So you’re saying that, or are you saying, being a good interviewer I won’t put words in your mouth, that in your view, these linkages would not be that expensive to sort of get going and the payoffs could be potentially huge because it allows you to see the big picture on these problems?

Dr. Nancy Potok:

Yeah, I’m saying on a relative scale, it’s not billions of dollars. I mean, one of the things you didn’t mention, that I will mention, is that prior to being the Chief Statistician, I was the Deputy Director at the Census Bureau. And I pushed very hard to do a lot of data linkage projects because in my view, while there is a role for surveys and big surveys, they’re super expensive. People are not responding. They don’t trust the government. Response rates are dropping. It costs a fortune to send people out, to go door to door. I mean, you can see, the census cost over $15 billion. So, it costs pennies to link records. Especially, if you already have the data. I mean, you have to have some people who know what they’re doing, and you have to do a little cleanup, and you have to get the data. But relatively speaking, I can’t remember what the numbers are now, but when you compare doing the census, the total cost of the census, for example, we have to go door to door, It was somewhere between $90 and $100 per household. And then if you linked records to get the same information, it was pennies. I mean, it was under a dollar, it was under 50 cents. It was ridiculously cheap. So you can get better information for a lot less money, and you can get a much more depth and breadth in what you’re looking at, because you can look across multiple dimensions.

Robert Hahn:

If you could whisper in Congress’s ear, in terms of priorities for fixing some of these data problems, it sounds like you’ve, you’ve sort of identified a couple of problems. One, we need to link the data better. So what would you beyond that piece of legislation that you talked about to try to get Congress to do something, assuming that they want to listen to your collective wisdom gained over 30 years or whatever.

Dr. Nancy Potok:

If I could get Congress to do whatever I wanted, I would, first of all, set up an independent national statistics office. Because to me, data quality is something we haven’t talked about, but from my vantage point of view, having been the Chief Statistician, I think it’s really critical that we have an independent statistical and evidence function here so that we don’t get political interference in the results. And there’s not biases that are intentionally built in to support, whatever administration is in power. So I think it’s, you know, we should look internationally at some models of national statistical offices that are independent. And when I say, what would that look like? I think you’d take census, you’d take the Bureau of Labor Statistics, you’d take the Bureau of Economic Analysis and for one thing, put them together and then they could run this kind of a national secure data service. I don’t know how realistic that is. I mean, it’s an idea that people have posed for a long time. So if we put that to the side, then next on my wish list would be to set up the national secure data service and let’s get going. You know, it’s not going to be perfect at first, but you learn, you know, you want to start with something. And I would go with what Nick and I recommended in our report, which is setting up what they call an FF RDC. It’s a federally funded research and development center that sort of runs by contract. And so it has a lot of the benefits of being outside of government, but it’s actually a full partner with government. The things that I worry about with the secure data service are can you hire the right people? Can you pay enough to keep them? And if you have a, an FFR DC, the answer is yes, you can. And you have all these flexibilities that you don’t have by a federal government agency. But at the same time, you need a federal agency to provide the oversight and to be a partner, a full partner in that. So that would take a long time to go into all of the pros and cons of that. But I would set that off immediately first. That would be the first thing. The second thing is that I would implement the recommendations that we’re going to be releasing next month on how to restructure OAB to really prioritize and push this through the agencies. And I would then have the agencies focus on their high value data sets, so they don’t get diverted or just so bogged down. Let’s identify, you know, what are the three most important things we have to do right now? We really have to fix the economy. We can’t just keep going like this. And we have to look at health issues, but we really have to look at all of these ancillary issues that have been the fallout of the pandemic. You know, and then we have a lot of emphasis on things like racial equity. Is that a priority for federal governments, for state governments? If it is, then, you know, we should be focusing on information that will help inform that because, if people are talking about things like police reform, that’s not something that you just rushed into kind of without evidence, without data, without understanding what works and what doesn’t. And there is a lot of data, some of it needs improvement, but, you start with what you have, you figure out what you need better. And then you structure policies around what is actually going to achieve the results you want, not what someone’s ideology is promoting, whatever that ideology is, if the ideology is not fact-based.

Robert Hahn:

That makes a lot of sense to me. We’ve been talking with Dr. Nancy Potok, who served as Chief Statistician of the United States and was also on the Evidence Based Policy Commission. Nancy, I want to thank you for joining us and for a very illuminating conversation.

Dr. Nancy Potok

Well, thank you. I love talking about this as you can see, so I appreciate the opportunity to be on your podcast. Thank you.