Sign up to get updates from us
On this episode of Two Think Minimum we welcome P…
On this episode of Two Think Minimum we welcome Professor Kristina McElheran. We explore the impact of artificial intelligence and related technologies on workers, workplaces, and productivity. Dr. McElheran shares her expertise on the economics and strategic management of technological change and how digitization is changing how firms innovate and make data-driven decisions.
Liked the episode? Well, there's plenty more where that came from! Visit techpolicyinstitute.org to explore all of our latest research!
Hello and welcome back to Two Think Minimum, the podcast of the Technology Policy Institute. I'm Scott Wallsten, president of TPI, and today is Tuesday, March 28, 2023. Today we'll be talking about the effects of automation and prediction analytics on workers, workplaces, and productivity. And if you're thinking that sounds a lot like a current hot topic, you're right. We're talking about artificial intelligence and related technologies. So far, most of the discussion has generated more heat than light, and we're hoping to shed some light in today's discussion with Professor Kristina McElheran, whose research spans these topics. Dr. McElheran is an assistant professor at the University of Toronto, following six years of teaching and doing research at Harvard Business School. Broadly speaking, she studies the economics and strategic management of technological change. And more specifically, she studies how digitization affects firm strategy, performance and organizational design, innovation in the digital age, data driven decision making and cognitive technologies which we might call AI and the future of work. Kristina, thanks for joining us today.
Hi, Scott. Thanks for having me.
So you've been studying lots of these issues for years now. Start off by telling us how you came to this research and take us through some of it and how it kind of led up to this moment in time.
Sure. Well, this is where I date myself a little bit, because I came to this first working in Silicon Valley during the.com era and just becoming so captivated by the promise, the potential, and some of the risks of what was then a cool new technology, the Internet, to transform so many aspects of our lives, and how firms work, and the organization of production. I took all that excitement and interest and brought it into an academic career with the goal, really, to unpack a lot of the nuances that take, first of all, lots of data, lots of training, lots of models, to make sense of, but always with this view of someone who'd experienced what it was like to develop new things and bring them into the world and try to get them to work. (And by build new things, I was not the programmer, just so we're clear!) But I was always interested in what I think of as sort of the integration layer between the technology, and the processes that they enable, and the people who have to work with them, and the organizations that shape how this all plays out.
[00:03:53.660] - Scott Wallsten
If I can interrupt, what were you doing in Silicon Valley at the time? I was also there during the .com era and mostly I just learned that I was never going to be rich because I didn't know what it took. So what were you doing and how did it continue to lead you into that?
I worked at a couple of firms, but I think the one that shaped my research interest the most was working for a startup called Exemplary Software that spun out of HP (Hewlett Packard). We were designing web-based supply chain management software, and the exciting thing was the ability of the Internet to enable real time communication, alerts, and automation. (You're going to hear some familiar words for the AI conversation)., A lot of things that had historically taken lots of time, lots of people, lots of analysis and could speed things up. And if we could replace “stuff” with information – we could really make things better for firms and people and economies. It was exciting –and then difficult, because to make all these new technologies work. It turns out lots of things have to change in the way firms operate, in the way firms compete, in the way they collaborate with their value chain partners, and then how workers interact with and engage with the technology. That turned out to be a lot stickier problem than getting the tech to work. So that's really what's been captivating my attention as a researcher, is getting into more of that organizational side and the economics and strategy of how that plays out
So the.com boom sort of began to show us some of the possibilities, but lots of aspects weren't there from computational power, the right kinds of connectivity, and then the bust, I guess, left a lot of people without much, but it did leave us with a lot of useful infrastructure, both physical and intellectual capital. Then you started doing research, and so what did you begin to look into? How did you start?
Well, I mean, first I did the hard, serious work of just the PhD training. I didn't jump right into research, but I think, unlike a lot of folks who pursue academia, I really went in with a set of questions in mind and tailored my training towards that. It's always been about understanding the Internet and production chain reorganization and understanding big software implementations in firms and how that changes how workers are paid or how value chain relationships play out. What the competitive implications are. I've done some great, fun collaborations with Erik Brynjolfsson (formerly at MIT, now at Stanford) where we've dug a lot into data driven decision making and predictive analytics. I've caught the wave before this wave, trying to understand cloud computing. (And I have an ongoing research agenda on the cloud.) Then most recently, the hot topic of the day, trying to better understand cognitive technologies, machine learning, natural language processing, what lots of people are calling AI today.
Before we get into what people are calling AI today and what we're all doing with the Chat GPT and the funny ways, it makes things up. So tell us a little bit more about the research that you did before, because that gives us the foundation to understand what research is today. One of your papers, for example, was on how workers of different ages and skills are affected by automation and I guess vice versa.
This was fun work with Erling Barth at the Institute for Social Research in Norway, Richard Freeman at Harvard University, and Jim Davis, who was then at the Boston BRDC, one of the Census Bureau research data centers where I have spent a lot of my time. What we did in this project was basically just we took a howitzer of data and we went in and matched up investment in big software projects. (So capital investment in software, not day to day operating expenses, but capitalized software investment) with detailed worker wage data and we basically stripped out everything we could. In fact, this came out last year in the Journal of Econometrics because we (and by we, I mean my smart co-authors), innovated in the econometric modeling of how we could control not just for the firm in which this was happening, or the industry in which it was happening, but also for the workers in particular job spells.
So we were stripping out that match quality and kind of putting everything aside, what was left (and it's striking, first of all, that there was much variation left after you do that), and we see very robust implications that when these big automation events happen in firms, that they share the productivity benefits, the wage benefits, essentially, with their workers. But it has this particular quality of complementing and working with and reinforcing mid age workers who probably have a lot of tacit organizational knowledge experience. And that this drops off. On the one hand, that makes a lot of sense because if this is technology that requires skills, and workers have skills that become less relevant over time, you might think that older workers don't have as many new skills to bring to that technology organizational mix and they don't have a lot of incentives to gain a lot of them. But what it means is that they don't lose out per se, but in a relative fashion they're disadvantaged and they leave firms more.
So this opens up in a very grounded way, in a very evidence-based way, this idea that we have to be careful about who gets left behind when all these transformations take place. I'm not in that age range yet, but I don't want to live in a society where we let all our older workers fall off the back of a fast, speeding, technology driven bus. I think we should think hard about how we help people who need to reskill or can't reskill or need a different approach to reskilling than anything we've tried before.
So with older workers there's the problem of the general problems of reskilling and how you do that also there's not a lot of time for anybody who's investing in that to make up the investment. I don't know if you can tell this from that, you probably can't tell this from the data but when they leave the firms, what do they do? Do they tend to find other jobs or is it early retirement and they're worse off than they would have been otherwise?
We can't pin it down dispositively in the data even though we have incredibly poor census data, we don't have EVERYTHING and we can't say for sure whether they leave the workforce entirely. Some for sure find other jobs, but this is consistent with a lot of anecdotal evidence that there's earlier retirement or they leave for jobs that are not as well paid or not as satisfying. To the extent that that's happening and it's not what people would choose for themselves, there's a basis for some concern.
Before we get into the kind of the reskilling question and, and what it, what this means overall it, it helped people kind of in the mid range of their careers. What about the people at the early stages of their careers? Why can they not just adapt easily to automation? You would expect them to just roll with the wave and be able to do it right.
So that was a cool twist on this project, which is that if you go about this without a lot of care, the first glimpse of the data suggests that the younger workers are hurt as well. But that turns out to be more of a sorting story, where younger workers are sorting more into different types of occupations, types of jobs, different types of firms. And once you account for all that, that early disadvantage (for younger workers) goes away. And that's left us with this interpretation that they are too young to have the complements that really pay off. Here we're starting to get away from this one paper into some of my other research. But my takeaway from all these years of studying it is this sort of headline story of heterogeneity – or unevenness. And it's just one way of saying that this doesn't play out the same everywhere and the averages actually kind of muddy the waters. And if we look at the places and the people and the applications that are well set up to exploit these technologies, they either have unique complements or particularly valuable compliments. It's the “two great tastes that taste great together”.
My students don't always know this reference anymore.
I know it's really sad. They need to.
But it's the “Reese's Peanut Butter Cup Theory of Technology,” which is that if you have these important things that go together, that's where you see transformation, that's where you see gains, you see benefits. And when that's not there, it's either kind of “meh” or it even is misaligned in a way that can pull away value or productivity or wages. I think the young folks, they just don't have the experience yet and that's why we're seeing it in the middle.
Let me give you the “person answer,” not the “academic answer” to your question.
Go ahead.
It really seems to be a story of tacit knowledge and intangible firm specific abilities that really come with age and to some extent experience in that particular environment.
I think you kind of have the same problem I do. When we say tacit knowledge, that's not sure that's really person explanation. But for younger people, it's like you said, it's sort of a sorting mechanism. So they'll be okay over time, right.
In the data so far they're fine. I'm not losing sleep over them yet. But the thing that keeps me awake as a parent is that it's really difficult to have a crystal ball saying well, this is the trajectory that we're going to be on and we can sort of anticipate what's coming. The efforts I've seen along those lines so far have left me pretty cold because they require so many assumptions or the data is kind of old or we're just fundamentally trying to drive while looking in the rear view mirror. And that's fine for academic research. But if you're a young person trying to choose a college major, that's anxiety provoking.
So we've got two issues. One is how to, even if all of this is just tremendous for society and it sounds like the benefits seem to at least be large. We've got a group of people who either need to be retrained or there's not enough time to retrain them and they will lose out. Then people who will soon be entering the workforce or have recently entered it and don't know how to prepare. Now we're beginning to move outside of your research, but what do you recommend for them? Either group.
Well, when I boil down to what I'm comfortable saying, what I feel sure saying is that figuring this out almost always requires 1) inspiration, 2) cash and 3) time –and those are not always available to all people or all firms. And thinking about how we can ease that seems to be what folks like us should be in the business of doing. I'm going to also toss a lot of this over the fence onto the policymakers. But I think we need to really reimagine how we go about training and credentialing workers, how we think about the incentives for this and who pays for it. And I don't think the solution is always dumping lots of money onto this from the outside because the workers themselves often have a better sense of what they would like to do, what they are passionate about, what they can best tailor their innate abilities and interests to. I think being creative and putting the inspiration closest to the insight seems like a good approach. I'm worried that we don't have a lot of time and I don't know how expensive some of this will turn out to be and for whom.
I think that's a big open question.
[00:19:03.760] - Scott Wallsten
I don't see really how we solve the problem for older workers because there's so little time to train them and then for them to work again. How do we handle that?
Well, I think again, it comes down to being very thoughtful and innovative in our understanding of what the right skills are. So there's this huge literature on skill biased technical change and routine biased technical change. And what happens a lot in this literature is we quickly start to substitute “jobs” and “skills” and “tasks” and “wages” in ways that I'm not sure always resolve our confusion. Let me give you an example that's really on my mind right now. I used to program a ton in Stata. I was very good at programming statistical models in Stata and achieving insights. And then R came through and I was busy teaching other things and busy writing other things and I wasn't being forced to learn the new language and so I sort of was lazy and didn't figure it out. Now it looks like maybe I don't have to! Because there's an AI there that's going to help me take what I knew and translate it into functional code that I can use. The nature of the insight is still there, rooted in all my training. But the skills that I need to get it done have been utterly transformed and they're actually less technical than they were even just a few years ago.
I'm so right there with you. I grew up on Stata and even some other things before that, SAS and so on. And then I was used to being the best programmer in the office. Then a colleague came on who was like a generation behind me, and I suddenly wasn't the best programmer anymore. And then we hired another person who was a generation behind her, and I'm just such an antique. I wasn't going to take the time to learn R so I still use Stata. But now we ask Chat GPT to fix your code. I ask it to make a PowerPoint presentation using Python. I could never have done that. It's astonishing.
But the insights needed to do good empirical research are code-independent. So one of the things we should think deeply about are the amassed wisdom and experience in our older workers that maybe is much more accessible than it ever was before. And maybe sticking them back into a coding classroom isn't what we need to be doing.
Right, that's interesting. I mean, that's a really good point. It's not code. If you understand the logic of programming or just logic in general. You may not know the code, but you know the process.
You have a lot of wisdom and insight and this is what's so fascinating, exciting, and for some terrifying about cognitive technologies - is that we're talking about automation of activities that used to be solely done by white collar, educated workers. We had a model for how to get those skills, how to extract value from those skills, what to call those skills, how to test for those skills. That's being shaken up a bunch. And I think that's what's challenging for so many is that we now have to wade into a world of relabeling, reimagining, remeasuring how technology and people come together to produce things and invent things and get the day-to-day work done.
Let me ask another question. I don't know whether this actually makes any sense. The standard economics answer for something that is beneficial for society but not beneficial for everybody when there are winners and losers, is that you use some of the new surplus to compensate the losers. Right? That's kind of Economics 101, although we never get to the 102 of actually how to do it.
There's a lot of assumptions that go into that.
Yes, exactly. But one simple way could be that for some people earlier retirement is the right answer as long as they're compensated somehow by this increased productivity. But all of the policies kind of across the world are making retirement later because people live for so much longer. Is that a reason to believe we're kind of moving in the wrong direction on retirement support? I mean, I know that there are all these macro issues that we aren't going to go into, but are we kind of thinking about as a policy matter? Are we thinking about retirement the wrong way when intersecting with this?
I'm very nervous to wade into policy, in particular, because even though I'm trained as an economist, I work in a business school and I teach strategic management, which focuses much more on the decision making at a less macro level. And I spend a lot less time, as a result, thinking through the thorny public policy issues. But my instinct is ( and I'm not the only one who has this thought…so I know that Erik Brynjolfsson writes about this as well, and he and I discuss this sometimes) is that in general, when you take groups of people and you sort of sideline them and put their gifts and interests and aspirations in a box - no matter how comfortable it is, we as a society are losing out. And those groups also lose their leverage to shape the society in which they live. I'm deeply uncomfortable with the idea that we would just make lots of people comfortable but irrelevant.
That's a good answer.
I’m getting a little bit “on in the years” and some of this is self interested. I don't have any anticipation of kicking it at the beach “before my time.”
No, I think that's a great answer. And the idea of using large language models as a way for people to use their wisdom is really intriguing. But before we get more into those, there's another strand of your research which is sort of the firm effects, right? And how firms use and don't use automation and how it has to become integrated into their production processes before we see its effects overall. So I'd like to hear a little bit about that and then also whether you think that as we can then move into sort of more of the current AI. Is this version that what people now think of AI? Will it fall into the same category of thinking?
Let me answer your first question about the role of the firm. I'm passionate about this because I think a lot of the conversation has missed this important layer that shapes how the technologies play out in daily life in society over time. When we think about this at the worker level, it's very easy to have these undifferentiated people-shaped objects with a set of easily-classifiable skills and we're going to sprinkle some fairy dust that's going to make some of those skills more productive and some less productive. And then we'll sort of turn the crank and it'll all play out… and then terrifying things tend to happen if you want to get attention because he or she with the scariest statistic seems to be winning right now.
Right.
But what we know from lots of academic research is that what happens to workers is very much driven by the workplaces in which their jobs are designed, in which their compensation is determined, in which they have differing amounts of bargaining power and all these things. (I do a lot on decision rights and who has authority over what types of activities.) What we've learned from this body of research is that this is wildly uneven. In fact, if we're worried about wage inequality, a place we need to look very carefully is firm inequality because the unevenness and firm performance and profitability and productivity is feeding into what's happening at the worker level.
Wait, talk a little bit more about firm inequality because that's kind of a phrase we don't hear a lot because we do want firms to be different. What exactly do you mean by that?
But when we do the type of activity that we talked about in the older workers paper where we start to try to figure out how much of wage inequality is due to different factors, a lot of it loads on the differences between firms and what they can pay. And so when firms do very differently in terms of what they're able to produce, what they're able to pay their workers, how they're able to persist (because lots of churn and losing of firms means lots of churn and losing of jobs), that redounds onto people and if we don't pay attention to that layer, comprised of firms in competition, with different innovation aspirations and different productivity capabilities. I do a lot of work on the management practices in these firms. If we don't pay attention to that, we're going to miss where the technology actually meets the worker to have very different effects. And so if you're in a low variance, productivity focused workplace, then having lots of technology that makes things run smoothly will be great for productivity and for worker pay and all sorts of things that we care about.
It's not true in a design firm or I do a lot of manufacturing. So like in a prototyping, job-shop sort of activity, the technology behaves differently, the workers behave differently, their productivity looks different. And so this unevenness that you can't see from “on high” becomes really obvious once you zoom in and look at firms as entities, as distinct, as varied in ways that we can characterize – but we definitely should not lump together.
So what do you do with that? Because you're just stating this as a positive analysis. It's just fact. It's a good place to live. We wish more people lived there. But so how does that affect your analysis? Is there a normative aspect to it too? Do you think that firms should be divided in particular ways? That there are better or worse ways for firms to be unequal?
Well, I think firms do well and people in firms do well when they function to the best of their capabilities, when they are productive. I'm going to get outside some of my research but I'm going to lean on my pal Claudine Gartenberg's work where they often have a sense of purpose or things that aren't maybe just about pure compensation. I think having good functioning firms in good functioning markets is good for individuals and to the extent that we can help –and this is my job as a management professor – help clarify what types of bundles of activities promote productivity or innovation, worker thriving, firm survival, all these good things that we care about. I don't think they all have to be the same, but they could be thriving differently.
If I'm lucky, I'll push you there.
I probably need to come up with a fancier academic term for this, but I think of them as recipes. So, there are a lot of different kinds of bread out there. So I was one of the Pandemic Sourdough Baking People. I learned how to make sourdough, but before I did sourdough, I made regular bread and I made great bread with industrial yeast and certain types of flour and certain types of machinery. Sometimes I used a bread machine. It did not look pretty, but it turned out great. And then when I moved into sourdough, I had to do lots of different things. I had to have a different timing, I had to have different ingredients, and they had to come together in a different way. Both good, but if I mixed and matched, it was universally terrible. If I put the heavy flour with the wrong yeast. (My husband calls it surface-of-the-Moon bread, and that tells you all you need to know about the look and texture of my baking output).
Well, we're spending billions trying to go back to the moon, so maybe it's a good thing.
Light analogies aside, I think this holds true in firms that there are certain things that go together well. And understanding that recipe, both in terms of the inputs and the proportions, is something that we still have a lot of work to do. And we're learning lots all the time. I'm learning all the time and helping make choices around the technology that firms use and the tasks and jobs that are made up of these tasks, how those are designed and come together. I think we could really do innovative, important things without telling everybody that they have to be the same.
So, I mean, you're talking about it as if it's still a puzzle, and I understand that it is. But of course, you've done research on this and some of your work. has found that productivity was significantly higher at firms, or I guess it was plant level that used predictive analytics, but only when combined with certain other things. Right?
So we see plants, we see firms that do great. If they are really efficiency focused, they probably have to have a pretty stable demand signal. They have technology and processes that help them push variance out of the system. There's a lot of physical technology, there's digital technology. They'll have workers with very specific skill sets, and that bundle goes great. It's a “Reese's Peanut Butter Cup” of technology and workers and predictive analytics and data.
A Reese's Peanut Butter Cup with four things in it. I want that.
My 11-year-old would think that's amazing.
I think that's amazing.
But if we look at places that are very focused on high-variance activities, and they're innovating and there's a lot more going on, they have to be very flexible. I've seen this with young firms that are trying to learn about their market or what kinds of things they're capable of doing, or they haven't grown yet, so they don't have all that experience and capability. They need a different set of inputs, they need outsourced IT in the cloud, and they need different types of management practices that are focused not on being really structured but on being more innovative and attaching skilled workers to that workplace over time.
So now, of course, coming to what everyone's talking about, what we see in the Chat GPTs and Bard and Bing and all the rest, how does that fit into what you've been studying? Or maybe I should put it the other way, I'm not sure. But is this a technology that will fundamentally change workplaces, the nature of work? How long will it take to find its way into our productivity statistics? Because some technologies never did. Is it really a game changer? Or are we just going to continue to read New York Times articles about somebody who said, I typed this into Chat GPT, and look what it gave me?
I knew you were going to ask me this question. So every time we've had a big technological change in history, there's been excitement and pain, and every time it's “going to be different.” I think it's been “different” – but similar in key ways. And so you had my friend and colleague Avi Goldfarb on your show a little while back talking about how long it took for steam, for electricity to move from steam into electricity and to reconfigure how manufacturing plants in the US were built. It took a very long time for that to show up in productivity. Solow famously said, “you can see the computer everywhere except in the productivity statistics.” My friend Dan Gross at Duke has this great paper showing how it took like 80 years for AT&T to automate telephone switching. Sometimes it's not what the technology can do so much as what organizations and processes can do with the technology. And so no matter how many cool things we can see in the lab or on our desktop that are “amazing”, and I hope you hear the sarcasm there, like “amazing essays at the push of a button” Chat GPT can produce –until we figure out the processes and the products and the applications that really generate value.
They happen on different time frames. The technology is moving exponentially, organizations move linearly, and there's just almost no way to close that gap in a way that's predictable on a time frame that I can give you, except I'm pretty sure it's going to take longer than the technologists will tell you and it's going to be very uneven. So it's going to happen in fits and starts. It will probably happen slowly and then quickly, sort of how Hemingway described going broke. And there's going to be real winners and losers that we may not even be able to predict ahead of time. But complacency, I think, is not the answer. And I think naive techno-optimism isn't going to solve it for us either.
So that's kind of a complicated view.
I need to work on my elevator pitch.
No, that's fine. This is a long elevator ride. It seems like you think some of the current AI is overhyped because it will take organizations a long time to figure out how to integrate it into their production processes. But in other ways it may quickly upset certain sectors, maybe small bits. Like anybody who makes a living parsing emails, I think has got to be done for, right? Because you can just ask this kind of AI to do that for you.
But Jim Bessen has this great example. I recall the only occupation that he could identify as having been rendered completely obsolete by technology was Elevator Operator. So there are going to be changes, for sure, and there will be losers, for sure. But I still feel that even the amazing potential of the technology is not the same thing as “it's going to transform immediately everything we understand about production and work and life in society.”
So is a version of AI that people see now, that the normal people see now, and that we play with, is it significantly different from the types of AI that firms have been using so far in their processes? Is it some kind of step function for them? Or are we just kind of, for the first time, seeing behind the curtain?
That's a difficult question to answer. And the reason it's difficult – I'm going to lean on my research again because that's what I do when I'm trying to find a good answer – is that measuring this has been a very difficult thing to do for a long time. And the newer the technology, the more difficult it is. I worked with the US Census Bureau and Erik Brynjolfsson to try to understand the prevalence of AI-related technologies in 2017, and we sent out this giant survey that had been rigorously tested. It went to 800,000 firms. So this is not a small sample; it's carefully stratified. It represents the entire economy. We asked firms about their use of particular technologies with relatively generic names like machine learning or machine vision. And this could encapsulate a lot of things that don't look anything like what we're seeing with Chat GPT. And the adoption rates were very, very low. We're talking single digits. I think machine learning was the top.
Wow, that is low.
So that was a while ago. And things have changed and the survey is going to run again next year and I'm sure there's going to be much more uptake and there's going to be much more positivity bias around saying, “Yeah, my firm does AI!” But I think that there's a continuum that underlies this. There have been firms dipping their toe into the technological stream at different points, and they're not going to jettison everything they've ever done to hop on the next new thing, especially when it comes out every two weeks.
So there'll be different flavors of this across firms, some that've been quietly chugging along. Algorithms have been determining all sorts of things in our daily lives for a long time. Predictive analytics of various kinds have been sometimes AI analytics, sometimes logistic regression, sometimes it depends on whether the firm is trying to get funding or not.
Right.
I view it as more of a technology continuum, though I know that's not a very “technologist-y” sort of answer because the folks who are programming these things have a much closer view onto what this looks like in code terms.
But it could be a huge advancement in code terms and not in societal terms. So, I mean, both can be right.
I think it's going to be messy and difficult to understand for a while. And my main concern right now is that ( you'll notice I'm not one of these folks), is that there seems to be a lot of airtime and attention going to whoever can tell sort of the biggest, scariest story. Which is easy to sell, but very difficult to deal with when you go to work on Monday.
But nothing happens super fast at work on Monday, especially before a couple of cups of coffee.
And then it's lunchtime.
No, but I think that what we've seen in the past of difficult and painful adjustment that eventually unlocks amazing opportunities is what I think we're going to see again.
And ask me again in two weeks. :-)
No, but it's a good point. You can say Skynet and everybody already has a whole narrative in their head, even if it has no relationship to reality.
[00:46:35.730] - Kristina McElheran
It's interesting to me how much of the conversation has been shaped by science fiction. I was a good nerd in high school and consumed my share of science fiction. But I want to make sure that we don't march forward with all sorts of implicit assumptions and biases and imaginations that aren't rooted in reality, no matter how exciting the artifact looks. I'm a little bit more skeptical about what the practical implications, at least in the near term, will be.
So having a wrong narrative or a fantastical narrative is bad if it ends up having policy implications. If you have Senators worried about Skynet. So do you see this affecting policy? And then, beyond that, what are the policy questions we should be asking and answering? Or is this a time to say, just let's just hold on a little bit and let things happen for a while?
So again, notwithstanding my instinctive allergy to making bold policy recommendations, I will note that there have not been a lot of blockbuster books or movies about utopian, happy AI outcomes. They're always scary and negative and full of tension, and that's exciting when you're in a movie theater, but it's not exciting at work on Monday. And we don't want to be pushed by our darkest, most negative biases and fears when in fact, humans are capable of tremendously optimistic and wonderful and altruistic impulses and imaginations. So I guess I'm going to put a plug in to listen to our better angels. And the other thing I'll say is that when in doubt, when we don't have data, I'm a big fan of running experiments. I'm a big fan of saying well, rather than just make a bunch of assumptions and extrapolate and pour concrete on it and hope for the best, what if we, as a policy matter, pursued experimentation, pursued different types of ways of retraining, pursued different ways of compensating, pursued different ways of commercializing and collected a lot of data around what truly works – and then built policies around that.
So, I mean, that's music to my ears, but experimentation is never popular. You can't cut a ribbon on an experiment, so they tend not to be politically popular. Although there was the Job Training Partnership Act right from a long way back that included experiments. But this seems like then this is the time to be advocating for experiments because we're at the beginning of this new stage, and it's at the beginning that you need to try to figure out what you want to measure and how to do it right.
That would be the one thing that I would feel confident that we could put some energy and political capital and fiscal capital behind is, what I say to my kids is, “When in doubt go find out.” That, instead of just arguing past each other without facts, why don't we go collect more better facts as quickly as possible to be sure? I don't think deferring this until we've run endless studies is going to get us anywhere great either, but I don't know that flailing around in a vacuum, which is where I feel we currently are with our data at the moment, is going to be super productive.
I think advising Congress, when in doubt, go find out is excellent advice and it would be nice if they would follow that. I think we've been going on for quite a while. We have to wrap up now, and it's good, I think, to end this on an optimistic note, which I think we have. So, Kristina, thank you so much for joining us. I really enjoyed this conversation.
It's been my pleasure, Scott.
