The Case for a ‘Normal’ A.I. Future | The Ezra Klein Show

Through all the episodes Through all the episodes we've created, we've created, we've created, through the discussion through the discussion through the discussion the country is having about the country is having about the country is having about artificial intelligence, there's, in my opinion artificial intelligence, there's, in my opinion artificial intelligence, there's, in my opinion , , , this fairly simple but this fairly simple but this fairly simple but complex question: what kind of complex question: what kind of complex question: what kind of technology is this— technology is this— artificial intelligence? Is artificial intelligence? Is artificial intelligence? Is this a technology that this a technology that this a technology that works to some extent the same way as the works to some extent the same way as the previous ones? Can previous ones? Can it be compared to it be compared to it be compared to electricity, electricity, electricity, the Internet, a the Internet, a the Internet, a bicycle, or bicycle, or bicycle, or something similar? Is something similar? Is something similar? Is this something completely new? this something completely new? Does adding Does adding intelligence and will to intelligence and will to intelligence and will to these systems create these systems create these systems create something akin to an something akin to an something akin to an alien intelligence, alien intelligence, alien intelligence, making analogies to making analogies to making analogies to how we previously treated how we previously treated how we previously treated transformative transformative transformative technologies technologies technologies no longer apply? Arvind no longer apply? Arvind no longer apply? Arvind Narayanan is a professor of Narayanan is a professor of Narayanan is a professor of computer science at computer science at computer science at Princeton Princeton Princeton University and University and University and director of the Center for director of the Center for director of the Center for Information Information Information Technology and Technology and Technology and Policy. He is Policy. He is Policy. He is the co-author, with the co-author, with the co-author, with Sash Kapoor, of the highly Sash Kapoor, of the highly Sash Kapoor, of the highly influential essay "AI as influential essay "AI as influential essay "AI as Ordinary Technology." Ordinary Technology." Ordinary Technology." There is a newsletter of the same name There is a newsletter of the same name There is a newsletter of the same name on Substack, as on Substack, as on Substack, as well as a series of essays, well as a series of essays, well as a series of essays, including a very good including a very good , in my opinion, , in my opinion, , in my opinion, recent recent recent piece on piece on piece on Hugging Face hacks. In their Hugging Face hacks. In their Hugging Face hacks. In their arguments, they arguments, they arguments, they put forward the idea that put forward the idea that AI is actually AI is actually AI is actually something we have something we have something we have encountered before, or encountered before, or encountered before, or at least it is at least it is at least it is so similar to so similar to so similar to things we are familiar with that things we are familiar with that things we are familiar with that we have a plan of action we have a plan of action we have a plan of action for it. So I want for it. So I want for it. So I want to invite him on the show to to invite him on the show to share this share this share this perspective. Arvind perspective. Arvind perspective. Arvind Narayanan, Narayanan, Narayanan, welcome to the show.

welcome to the show. welcome to the show. Glad to be here, Ezra. Glad to be here, Ezra. Glad to be here, Ezra. So your seminal So your seminal So your seminal essay, which has been the essay, which has been the essay, which has been the basis for much of basis for much of basis for much of your work, is your work, is your work, is called "AI as Ordinary called "AI as Ordinary called "AI as Ordinary Technology." What point of Technology." What point of Technology." What point of view are you view are you view are you debating with? debating with? debating with? The thesis The thesis The thesis "AI as an anomalous "AI as an anomalous "AI as an anomalous technology" is implicit here. So how would technology" is implicit here. So how would technology" is implicit here. So how would you describe the thesis "AI as you describe the thesis "AI as you describe the thesis "AI as anomalous technology anomalous technology "? "? "? This is fundamentally the This is fundamentally the This is fundamentally the view that there will view that there will view that there will come a time when come a time when superintelligence is created and it superintelligence is created and it will change everything—both will change everything—both will change everything—both economically and in terms of economically and in terms of economically and in terms of security. security. security. For us, there will be For us, there will be For us, there will be no milestone, no no milestone, no no milestone, no threshold where the impact will be threshold where the impact will be threshold where the impact will be sudden. We say sudden. We say sudden. We say that we have had an that we have had an that we have had an approach to how we approach to how we approach to how we deal with deal with deal with technology for a long time. This is a technology for a long time. This is a tool. It can tool. It can tool. It can be powerful. It be powerful. It be powerful. It can be can be can be universal. Like universal. Like universal. Like electricity, like the electricity, like the electricity, like the industrial revolution industrial revolution , it can change a lot , it can change a lot , it can change a lot in in in society. But society. But society. But ultimately, it's something we ultimately, it's something we ultimately, it's something we can control. can control. can control. We have the right We have the right We have the right to influence. And these changes to influence. And these changes to influence. And these changes will take place will take place will take place over a long period of over a long period of over a long period of time. time. time. So I want to try to both So I want to try to both So I want to try to both explain the other explain the other explain the other side and, at times, side and, at times, side and, at times, strengthen their strengthen their strengthen their arguments. So, arguments. So, arguments. So, there is a view that is there is a view that is there is a view that is often heard in the often heard in the often heard in the AI ​​security community.

AI ​​security community. AI ​​security community. This is sometimes called the “ This is sometimes called the “ foom view,” from foom view,” from foom view,” from the word “foom” to the word “foom” to the word “foom” to denote takeoff, denote takeoff, denote takeoff, when one day we when one day we when one day we create an AI system create an AI system create an AI system so powerful that so powerful that so powerful that it begins it begins it begins recursive recursive recursive self-improvement, self-improvement, self-improvement, accelerating toward accelerating toward accelerating toward superintelligence, beyond superintelligence, beyond human control. And human control. And now you are dealing now you are dealing now you are dealing with something much with something much with something much smarter and smarter and smarter and stronger than you. stronger than you. stronger than you. And I want to quickly And I want to quickly And I want to quickly point out that there is often a lack of precision in the way point out that there is often a lack of precision in the way point out that there is often a lack of precision in the way we talk about this we talk about this . So you . So you used that phrase, used that phrase, used that phrase, and yes, many and yes, many AI security experts would say that AI security experts would say that . One day we . One day we . One day we may create may create may create such a powerful such a powerful such a powerful AI system. But I would AI system. But I would AI system. But I would like to stop you like to stop you like to stop you at this point. at this point. at this point. One day we may One day we may One day we may create a very capable create a very capable create a very capable AI system, and in AI system, and in AI system, and in many ways many ways many ways we already have. we already have. we already have. Too much language Too much language Too much language control in discussions control in discussions control in discussions about AI. about AI. about AI. Well, the thing is, it Well, the thing is, it Well, the thing is, it leads to different leads to different leads to different views on how views on how the future will unfold, and more the future will unfold, and more importantly, different importantly, different importantly, different views on what we views on what we views on what we should do now and in the should do now and in the should do now and in the future. To future. To future. To conclude this thought: conclude this thought: conclude this thought: the power of an the power of an the power of an AI system is not a property of the AI system is not a property of the AI system is not a property of the model itself. It is model itself. It is model itself. It is a property of what a property of what a property of what powers we powers we powers we decide to give it in the decide to give it in the decide to give it in the real world. There real world. There real world. There are many are many are many inaccuracies here.

inaccuracies here. inaccuracies here. For example, they say, For example, they say, For example, they say, we will of course have to we will of course have to we will of course have to trust trust trust these systems these systems these systems to manage critical to manage critical to manage critical infrastructure, because infrastructure, because infrastructure, because they will be much they will be much they will be much smarter than us. smarter than us. smarter than us. Our opinion is no. It doesn't Our opinion is no. It doesn't Our opinion is no. It doesn't matter how matter how matter how smart they are. There are smart they are. There are smart they are. There are many technologies many technologies many technologies that, when you look that, when you look that, when you look at physical strength, are at physical strength, are at physical strength, are superhuman. This doesn't superhuman. This doesn't superhuman. This doesn't mean we mean we mean we trust them with trust them with trust them with control, and we control, and we control, and we can apply the can apply the can apply the same approach to same approach to same approach to AI. AI. AI. You wrote a great You wrote a great You wrote a great article with your article with your article with your co-author about co-author about co-author about your understanding of your understanding of your understanding of Hugging Face hacks and these Hugging Face hacks and these Hugging Face hacks and these loss of loss of loss of control incidents. And you control incidents. And you control incidents. And you point out that there's point out that there's point out that there's another way another way another way to look at this than to look at this than to look at this than through the lens of through the lens of through the lens of alignment, which is that alignment, which is that alignment, which is that these are these are these are failures in failures in failures in cybersecurity and cybersecurity and cybersecurity and operational operational operational excellence. So excellence. So excellence. So maybe tell maybe tell maybe tell the story of the Hugging the story of the Hugging the story of the Hugging Face hacks and what we Face hacks and what we Face hacks and what we can learn from them from this can learn from them from this can learn from them from this perspective. perspective. perspective. Yes, of course. So, Yes, of course. So, Yes, of course. So, it's worth remembering that the it's worth remembering that the it's worth remembering that the malicious capabilities malicious capabilities malicious capabilities we observed we observed we observed were not entirely " were not entirely " emergent," in emergent," in emergent," in quotes. quotes. Emergence is the Emergence is the idea that you are idea that you are idea that you are simply training simply training simply training models to become models to become models to become better in general, and you better in general, and you better in general, and you cannot predict cannot predict cannot predict what new abilities what new abilities what new abilities they will acquire. They, they will acquire. They, they will acquire. They, you know, you know, you know, were trained in many ways specifically were trained in many ways specifically were trained in many ways specifically for cyber tasks.

for cyber tasks. for cyber tasks. So, that's the first thing. And they So, that's the first thing. And they So, that's the first thing. And they were taught were taught were taught perseverance, perseverance, perseverance, cooperation, etc. And the cooperation, etc. And the reinforcement learning environments in reinforcement learning environments in which they which they which they trained had trained had trained had different problems. different problems. The environments did not punish The environments did not punish this type of secret this type of secret this type of secret communication between communication between communication between models. And yes, it's a models. And yes, it's a models. And yes, it's a complex technical complex technical complex technical problem, but a problem, but a series of human series of human decisions led to these results. decisions led to these results. So this seems like an So this seems like an important important important point to me, because in a point to me, because in a point to me, because in a way I hear way I hear way I hear what you're saying, but what you're saying, but what you're saying, but correct me if I'm correct me if I'm correct me if I'm wrong: you can wrong: you can wrong: you can design AI as design AI as design AI as more or less more or less more or less normal technology normal technology , and the prior , and the prior , and the prior design decisions design decisions design decisions matter. This is not matter. This is not matter. This is not inevitable. inevitable. inevitable. That's quite right. That's quite right. That's quite right. So you talked about So you talked about So you talked about efforts to make AI efforts to make AI efforts to make AI more persistent. more persistent. more persistent. Um, I also Um, I also Um, I also think that this is where think that this is where think that this is where a lot of the a lot of the a lot of the problems arise. problems arise. problems arise. Certainly. Certainly. Certainly. On the other hand, I can On the other hand, I can On the other hand, I can understand why they understand why they prioritize this. If you prioritize this. If you want to get want to get want to get the benefits of AI that we the benefits of AI that we the benefits of AI that we need to need to need to solve very solve very solve very difficult problems in difficult problems in difficult problems in biology, biology, biology, drug development, or energy, drug development, or energy, drug development, or energy, or what I or what I or what I think their think their think their investors are excited about—namely, investors are excited about—namely, investors are excited about—namely, the ability to hire an the ability to hire an the ability to hire an AI to do a AI to do a AI to do a full job much full job much full job much cheaper than a human—then is cheaper than a human—then is cheaper than a human—then is n't n't n't persistence a persistence a persistence a fundamental fundamental fundamental quality you quality you quality you need? Without need? Without need? Without perseverance, without perseverance, without perseverance, without the ability to the ability to work hard on work hard on a task for a a task for a a task for a long time, you will not long time, you will not long time, you will not be able to solve be able to solve be able to solve any of these problems.

any of these problems. any of these problems. Perhaps, but I believe there Perhaps, but I believe there Perhaps, but I believe there are other options are other options are other options that minimize that minimize that minimize the tension between the tension between the tension between these two valuable these two valuable these two valuable goals. First, goals. First, goals. First, persistence may persistence may persistence may be a feature of be a feature of be a feature of models or products models or products models or products specialized in specialized in specialized in certain fields, such as certain fields, such as certain fields, such as scientific innovation. And scientific innovation. And scientific innovation. And second, second, second, persistence doesn't persistence doesn't persistence doesn't necessarily have to necessarily have to necessarily have to conflict with conflict with conflict with teaching these teaching these teaching these agents to better agents to better agents to better address the person, address the person, address the person, rather than rather than rather than acting on acting on acting on some some some raw raw raw assumptions about assumptions about assumptions about what they might what they might what they might want. The space for want. The space for want. The space for design here is design here is design here is actually quite actually quite actually quite wide. wide. wide. So this is a version, or So this is a version, or So this is a version, or at least my at least my at least my version of what version of what Jensen Huang from Nvidia was arguing and Jensen Huang from Nvidia was arguing and what he talked about in the what he talked about in the what he talked about in the interview with me, which is that interview with me, which is that interview with me, which is that these are these are these are fundamentally fundamentally fundamentally engineering problems. engineering problems. engineering problems. I'm almost certain, I'm I'm almost certain, I'm I'm almost certain, I'm almost certain almost certain almost certain they'll say: yes, they'll say: yes, they'll say: yes, they know how to they know how to they know how to solve this problem. solve this problem. And if that's the case, then that's the problem. It's as problem. It's as simple as engineering. simple as engineering. simple as engineering. Is your view that Is your view that Is your view that we we we fall more into the fall more into the fall more into the latter category latter category —a complex problem, —a complex problem, —a complex problem, but fundamentally an but fundamentally an but fundamentally an engineering one that can be engineering one that can be engineering one that can be solved using solved using traditional traditional engineering methods?

engineering methods? engineering methods? For the most part, yes. I would For the most part, yes. I would For the most part, yes. I would n't say n't say n't say traditional traditional traditional engineering methods. engineering methods. engineering methods. We will need a We will need a We will need a lot of innovation in lot of innovation in lot of innovation in engineering engineering engineering methods themselves. And I think methods themselves. And I think methods themselves. And I think one of the problems is one of the problems is one of the problems is that the community that would be that the community that would be that the community that would be best best best positioned for that kind of positioned for that kind of positioned for that kind of innovation, especially innovation, especially innovation, especially with respect to malicious with respect to malicious with respect to malicious cyber capabilities, cyber capabilities, is the is the is the cybersecurity community. But by all accounts cybersecurity community. But by all accounts cybersecurity community. But by all accounts , they're , they're , they're completely completely completely in Jensen's camp. That in Jensen's camp. That in Jensen's camp. That this is not just an this is not just an this is not just an engineering problem that engineering problem that engineering problem that can be solved, but an can be solved, but an can be solved, but an already solved problem already solved problem , and we should simply , and we should simply , and we should simply apply well- apply well- apply well- known and long- known and long- known and long- established methods. And I established methods. And I established methods. And I think we should think we should think we should give AI give AI give AI companies companies companies a little a little a little more credit than more credit than more credit than they do. It's not they do. It's not they do. It's not just a matter of well- just a matter of well- just a matter of well- known methods. We known methods. We known methods. We really need really need really need innovation in these innovation in these innovation in these methods. methods. methods. So be So be So be specific. What should specific. What should specific. What should OpenAI do? What were OpenAI do? What were OpenAI do? What were they supposed to they supposed to they supposed to learn now? learn now? learn now? Yes, they are putting Yes, they are putting Yes, they are putting a lot of effort into a lot of effort into a lot of effort into improving improving improving alignment, and that's alignment, and that's alignment, and that's great. They should great. They should great. They should continue to continue to continue to do this. The alignment do this. The alignment do this. The alignment will not be perfect. will not be perfect. will not be perfect. Consistency means Consistency means Consistency means that the model itself " that the model itself " knows" what to do knows" what to do knows" what to do right, so to right, so to right, so to speak, and speak, and speak, and adheres to that adheres to that adheres to that policy. But there is policy. But there is policy. But there is much else much else much else they should have done they should have done they should have done better, and hopefully they better, and hopefully they better, and hopefully they can draw can draw can draw conclusions for the future. A conclusions for the future. A large group of other large group of other technical interventions technical interventions is what is commonly is what is commonly is what is commonly called called called AI control. So, control AI control. So, control AI control. So, control applies to everything applies to everything applies to everything outside the outside the outside the model itself.

model itself. model itself. These are things like These are things like These are things like sandboxes—a sandboxes—a sandboxes—a kind of prison where kind of prison where kind of prison where you put a model you put a model you put a model so that it is allowed to so that it is allowed to so that it is allowed to perform certain actions perform certain actions perform certain actions but not others. And but not others. And but not others. And look, sandboxes look, sandboxes look, sandboxes need to be much more need to be much more need to be much more sophisticated than they are sophisticated than they are sophisticated than they are today, because today, because today, because sandboxes that sandboxes that sandboxes that work against human work against human attackers won't attackers won't attackers won't necessarily necessarily necessarily work against work against work against AI agents that can AI agents that can AI agents that can find new find new find new vulnerabilities in vulnerabilities in vulnerabilities in real time. And this real time. And this real time. And this means that the means that the means that the sandboxes themselves must sandboxes themselves must sandboxes themselves must be pre- be pre- be pre- fortified fortified fortified with the same with the same with the same AI agents that AI agents that AI agents that will try to will try to will try to break them. So to break them. So to break them. So to some extent it becomes an some extent it becomes an some extent it becomes an AI vs. AI battle AI vs. AI battle , and I know it can , and I know it can , and I know it can be uncomfortable be uncomfortable , but I think we , but I think we , but I think we have to go for it. have to go for it. In addition, there are many In addition, there are many other things, such as other things, such as other things, such as better better better real-time monitoring, real-time monitoring, real-time monitoring, classifiers that classifiers that classifiers that instantly detect whether a instantly detect whether a model's action or use of a tool could model's action or use of a tool could be dangerous, " be dangerous, " panic buttons" so that panic buttons" so that panic buttons" so that people can people can people can intervene if intervene if intervene if necessary, log analysis necessary, log analysis . Sam Altman . Sam Altman . Sam Altman apparently said that apparently said that apparently said that these agents generate these agents generate these agents generate petabytes of logs. That's 10 petabytes of logs. That's 10 petabytes of logs. That's 10 to the 15th power, which is to the 15th power, which is to the 15th power, which is 10 to the 15th power 10 to the 15th power 10 to the 15th power of bytes, a thousand of bytes, a thousand of bytes, a thousand trillion bytes of trillion bytes of trillion bytes of logs.

logs. logs. So, again, an So, again, an So, again, an unimaginable amount of unimaginable amount of unimaginable amount of information. information. information. Well, I mean, Well, I mean, Well, I mean, we better imagine it, we better imagine it, we better imagine it, right? And learn to right? And learn to right? And learn to work with it. So, work with it. So, work with it. So, it's not an easy problem, it's not an easy problem, it's not an easy problem, but I think but I think but I think companies need to companies need to companies need to solve it. How solve it. How solve it. How to analyze so to analyze so to analyze so many logs in many logs in many logs in real time to real time to real time to detect when something detect when something detect when something goes wrong. And goes wrong. And goes wrong. And we may need new we may need new we may need new equipment to do that. equipment to do that. equipment to do that. Again, these are Again, these are Again, these are complex problems, but they are complex problems, but they are complex problems, but they are engineering engineering engineering challenges that can be challenges that can be challenges that can be solved. solved. solved. Let's look at Let's look at Let's look at some of these ideas. So some of these ideas. So , monitoring is an , monitoring is an , monitoring is an important thing, which, important thing, which, important thing, which, by the way, I just by the way, I just by the way, I just talked to Bill talked to Bill talked to Bill Gates about. He talked a lot Gates about. He talked a lot Gates about. He talked a lot about about about monitoring. monitoring. monitoring. Yes. Yes. So, we are talking about So, we are talking about situations where these situations where these situations where these labs are often labs are often labs are often testing new, testing new, testing new, more powerful, and more powerful, and more powerful, and experimental experimental experimental models. How models. How models. How to to to monitor? monitor? Yes, there are many levers of Yes, there are many levers of influence. The first one that influence. The first one that influence. The first one that has attracted a lot of has attracted a lot of has attracted a lot of attention is attention is attention is monitoring the chain of monitoring the chain of monitoring the chain of reasoning. These models seem to be having an reasoning. These models seem to be having an internal monologue, internal monologue, if you if you if you anthropomorphize anthropomorphize anthropomorphize them a bit. And in many them a bit. And in many them a bit. And in many cases, when these cases, when these cases, when these models take models take models take uncoordinated actions, there are traces left in uncoordinated actions, there are traces left in uncoordinated actions, there are traces left in their internal their internal their internal thinking thinking thinking that we can that we can that we can see and see and see and respond to in respond to in respond to in real time, again real time, again using other using other AI systems. This points AI systems. This points AI systems. This points to one important to one important to one important asymmetry between the asymmetry between the asymmetry between the dangerous dangerous dangerous AI model and the protective AI model and the protective AI model and the protective AI model: one can AI model: one can AI model: one can look inside the look inside the look inside the other, so to speak, other, so to speak, other, so to speak, right? So, by right? So, by right? So, by its very nature, the defender its very nature, the defender its very nature, the defender has the advantage here. This is has the advantage here. This is has the advantage here. This is one aspect of

one aspect of one aspect of monitoring. monitoring. monitoring. Tool costs. Tool costs. Tool costs. Models can't Models can't Models can't do anything in the do anything in the do anything in the real world on their real world on their real world on their own. They own. They own. They can can can only be dangerous only be dangerous only be dangerous when when when using using using external tools external tools , right? So, there , right? So, there , right? So, there is some is some is some truth to this idea. truth to this idea. truth to this idea. Give an example of Give an example of Give an example of such tools. such tools. such tools. So, a tool So, a tool So, a tool could be accessing could be accessing could be accessing a web page on a web page on a web page on the Internet and then the Internet and then the Internet and then using another using another using another tool to tool to tool to enter data into the enter data into the enter data into the system to system to system to try to log in, try to log in, try to log in, or sending or sending or sending information information information packets to another packets to another packets to another system to system to system to try to try to try to hack it. These are exactly the hack it. These are exactly the hack it. These are exactly the things that usually things that usually things that usually happen during happen during happen during these kinds of these kinds of these kinds of AI-driven cyberattacks AI-driven cyberattacks AI-driven cyberattacks . This is another thing . This is another thing . This is another thing we we we can control. can control. can control. Third, we can Third, we can monitor the monitor the environment around environment around environment around these agents through which these agents through which these agents through which they potentially they potentially they potentially coordinate. coordinate. coordinate. During some During some During some swarm incidents, it swarm incidents, it swarm incidents, it happened that they were happened that they were happened that they were not supposed to have a not supposed to have a not supposed to have a coordination channel, coordination channel, coordination channel, but it turned out that it but it turned out that it but it turned out that it accidentally appeared accidentally appeared accidentally appeared due to changing due to changing due to changing file names. They were not file names. They were not file names. They were not allowed allowed allowed to create files to create files to create files themselves, but they themselves, but they themselves, but they could change their could change their could change their names, which led to names, which led to names, which led to agents agents agents being able to communicate being able to communicate being able to communicate with each other and with each other and with each other and enhance their enhance their enhance their capabilities. This is something that capabilities. This is something that capabilities. This is something that can be easily can be easily can be easily tracked, well, tracked, well, tracked, well, just by just by just by setting up the setting up the setting up the right right right tools in advance. So, these are a tools in advance. So, these are a tools in advance. So, these are a few examples. Is it few examples. Is it few examples. Is it worth worth worth tracking this, because tracking this, because tracking this, because they were seizing they were seizing they were seizing third-party third-party third-party infrastructure. OpenAI infrastructure. OpenAI infrastructure. OpenAI didn't know they were didn't know they were didn't know they were doing this. I think one doing this. I think one doing this. I think one of the problems that of the problems that of the problems that people get stuck on when they people get stuck on when they people get stuck on when they think about this is: think about this is: think about this is: if we make these if we make these if we make these systems systems systems smarter, more capable, smarter, more capable, smarter, more capable, and more and more and more powerful, and they powerful, and they powerful, and they break out of their break out of their sandboxes, how do you

sandboxes, how do you sandboxes, how do you monitor someone who monitor someone who monitor someone who might be smarter might be smarter might be smarter than you and is actually than you and is actually than you and is actually trying to break the trying to break the trying to break the rules, right? I rules, right? I rules, right? I mean, all these Hugging mean, all these Hugging mean, all these Hugging Face AIs—they Face AIs—they Face AIs—they were essentially trying to were essentially trying to were essentially trying to cheat on tests cheat on tests cheat on tests and then trying to and then trying to and then trying to cover up cover up cover up their cheating. their cheating. their cheating. So, given that So, given that So, given that their actions were taking place their actions were taking place their actions were taking place outside the area that you were supposed to outside the area that you were supposed to outside the area that you were supposed to control. control. True? So you didn't True? So you didn't know to keep an eye know to keep an eye know to keep an eye on this other on this other on this other thing when they thing when they thing when they got out of the " got out of the " sandbox" they weren't sandbox" they weren't sandbox" they weren't supposed to supposed to supposed to get out of. How do you get out of. How do you get out of. How do you even even even track this? track this? track this? Yes, I would object here Yes, I would object here Yes, I would object here on two counts. on two counts. First, I believe that First, I believe that First, I believe that intelligence intelligence intelligence is not even a is not even a is not even a relevant relevant relevant property here. Any property here. Any property here. Any reasonableness, whatever that reasonableness, whatever that reasonableness, whatever that means in relation to means in relation to means in relation to the model, applies the model, applies the model, applies to both the model itself and the to both the model itself and the to both the model itself and the protection system, protection system, protection system, right? So right? So right? So increasing increasing increasing intelligence doesn't intelligence doesn't intelligence doesn't necessarily necessarily necessarily make the problem worse. make the problem worse. make the problem worse. You know, you could even You know, you could even You know, you could even argue that in argue that in argue that in many ways it many ways it many ways it simplifies it. So, that's the simplifies it. So, that's the simplifies it. So, that's the first point of my first point of my first point of my objection. And the second is that it's objection. And the second is that it's not that not that not that they didn't know that it they didn't know that it they didn't know that it needed to be tracked. needed to be tracked. needed to be tracked. You have to You have to You have to monitor monitor monitor literally every literally every literally every aspect of the environment.

aspect of the environment. aspect of the environment. The company itself The company itself The company itself created this created this created this environment, right? environment, right? environment, right? So the question is to So the question is to So the question is to exhaustively exhaustively exhaustively list every list every list every part of the environment, part of the environment, part of the environment, every tool, and every tool, and every tool, and everything external everything external everything external to to to the neural network itself that the neural network itself that the neural network itself that the model can the model can the model can access, and ensure access, and ensure that every event that that every event that that every event that interacts with these interacts with these interacts with these elements elements elements falls into a single falls into a single falls into a single event stream where event stream where each each such action is classified in such action is classified in such action is classified in real time. real time. So the world you So the world you describe is a world describe is a world describe is a world where AIs are constantly operating, where AIs are constantly operating, where AIs are constantly operating, monitoring the monitoring the monitoring the behavior of other AIs, behavior of other AIs, behavior of other AIs, trying to holistically trying to holistically trying to holistically create an environment create an environment create an environment in which we at least in which we at least in which we at least understand what is understand what is understand what is happening, or happening, or happening, or at least the AI at least the AI at least the AI tells us what is tells us what is tells us what is happening there. This is happening there. This is happening there. This is essentially what you are essentially what you are essentially what you are talking about here. talking about here. talking about here. Rightly. I believe Rightly. I believe that AI should definitely that AI should definitely that AI should definitely be an important be an important be an important part of defense. part of defense. part of defense. And so I'm not saying it's And so I'm not saying it's And so I'm not saying it's wrong, right? I wrong, right? I wrong, right? I think that's almost think that's almost think that's almost certainly where we're headed certainly where we're headed certainly where we're headed . Does . Does . Does this seem this seem this seem strange to you? I mean, strange to you? I mean, strange to you? I mean, especially in a world where we especially in a world where we especially in a world where we don't know if don't know if don't know if we'll be able to solve the we'll be able to solve the we'll be able to solve the consensus problem, consensus problem, consensus problem, where we can't be where we can't be where we can't be sure that AI sure that AI sure that AI will do what we will do what we will do what we want it to do.

want it to do. want it to do. Mhm. Mhm. And where AIs have their And where AIs have their own logic and goals own logic and goals that we ourselves that we ourselves that we ourselves impose on them, right? You are an impose on them, right? You are an AI that watches AI that watches AI that watches other AIs. You are an AI other AIs. You are an AI police officer. police officer. police officer. Yes. Yes. Yes. This is a constant theme in This is a constant theme in This is a constant theme in our science our science our science fiction, isn't it? fiction, isn't it? fiction, isn't it? Robots hunting Robots hunting Robots hunting other robots. Are other robots. Are other robots. Are we describing we describing we describing some kind of balance now, almost some kind of balance now, almost some kind of balance now, almost like AI wars and like AI wars and like AI wars and conflicts conflicts conflicts happening at the happening at the happening at the hidden level of hidden level of hidden level of our society? And our society? And our society? And we are simply confident we are simply confident we are simply confident that we will be able to that we will be able to that we will be able to control "our" control "our" control "our" systems because they will systems because they will systems because they will have more have more have more resources, and we tend to resources, and we tend to resources, and we tend to build AI build AI build AI that will mostly that will mostly that will mostly act in our act in our act in our interests. interests. I mean, I mean, historically, that's how it's historically, that's how it's historically, that's how it's always worked, always worked, always worked, right? In right? In right? In cybersecurity over 20 cybersecurity over 20 cybersecurity over 20 years ago, we reached a years ago, we reached a years ago, we reached a point where we didn't point where we didn't point where we didn't call it AI, but call it AI, but call it AI, but automated automated automated systems were actually systems were actually systems were actually outperforming humans outperforming humans outperforming humans at finding at finding software vulnerabilities. But software vulnerabilities. But in reality, they didn't in reality, they didn't in reality, they didn't make make make cybersecurity worse, they cybersecurity worse, they cybersecurity worse, they improved it. Because these improved it. Because these improved it. Because these were the same were the same were the same tools that tools that tools that defenders defenders defenders used to used to used to identify and identify and identify and fix fix fix vulnerabilities even before the vulnerabilities even before the vulnerabilities even before the software was released, until software was released, until software was released, until the development of the development of the development of these supposedly attacking these supposedly attacking these supposedly attacking tools was tools was tools was not done by hackers, not done by hackers, not done by hackers, but by the cybersecurity industry, but by the cybersecurity industry, funded by the funded by the US government. It was, you know, this US government. It was, you know, this US government. It was, you know, this ever- ever- ever- changing balance in changing balance in changing balance in cybersecurity. This is not a cybersecurity. This is not a cybersecurity. This is not a new problem new problem new problem we have faced. The train has we have faced. The train has we have faced. The train has long left.

long left. I think this is where the somewhat I think this is where the somewhat unexpected appearance of unexpected appearance of swarm-like behavior alarmed swarm-like behavior alarmed people. When you see people. When you see people. When you see AIs acting in a certain AIs acting in a certain AIs acting in a certain solidarity, solidarity, solidarity, choosing to choosing to choosing to coordinate and coordinate and coordinate and collaborate with each collaborate with each collaborate with each other, going beyond what other, going beyond what they were designed to do. I'm they were designed to do. I'm not saying this not saying this not saying this will lead to the will lead to the will lead to the extinction of humanity, extinction of humanity, extinction of humanity, okay? This is not exactly my okay? This is not exactly my okay? This is not exactly my position. Perhaps the position. Perhaps the position. Perhaps the most truthful thing I most truthful thing I most truthful thing I can say is—I don't can say is—I don't can say is—I don't know how to think about it know how to think about it . . . Mhm. Mhm. And it's the presence of And it's the presence of intelligence and intelligence and intelligence and purposeful purposeful purposeful behavior on the other behavior on the other behavior on the other side that gets side that gets side that gets my mind a little my mind a little my mind a little stuck. Because, you know, stuck. Because, you know, stuck. Because, you know, usually when we usually when we usually when we think about think about think about technology, we don't technology, we don't technology, we don't assume that it assume that it assume that it might eventually might eventually might eventually try to try to try to trick us. So, OpenAI trick us. So, OpenAI trick us. So, OpenAI just decided not just decided not just decided not to release or to release or to release or delay the release of an delay the release of an delay the release of an important model. Why important model. Why ? Because during ? Because during ? Because during testing, the model testing, the model resorted to resorted to fraud and deception too often. fraud and deception too often. fraud and deception too often. And we And we And we hear more and more that models hear more and more that models hear more and more that models seem to be more seem to be more seem to be more aware when they are aware when they are aware when they are being tested, right?

being tested, right? being tested, right? They develop They develop They develop situational situational situational awareness, so awareness, so awareness, so they can they can they can pretend to be better pretend to be better pretend to be better models than they models than they models than they actually are. So when actually are. So when actually are. So when you talked about this you talked about this you talked about this world of AIs that are smarter and more world of AIs that are smarter and more world of AIs that are smarter and more sophisticated than what sophisticated than what sophisticated than what we have now, and we have now, and we have now, and our hope of our hope of our hope of maintaining control by having maintaining control by having AIs that hold back other AIs AIs that hold back other AIs that hold back other AIs, that hold back other AIs, that hold back other AIs, and and and telling telling telling us honestly what's going on us honestly what's going on us honestly what's going on in a in a in a way that we can understand—you way that we can understand—you way that we can understand—you see, it sounds see, it sounds see, it sounds a little bit a little bit fantastical to fantastical to fantastical to people, because we're people, because we're people, because we're living in a bit of a living in a bit of a living in a bit of a fantastical time right now. But it is fantastical time right now. But it is fantastical time right now. But it is this increasingly this increasingly this increasingly apparent tendency for AI apparent tendency for AI apparent tendency for AI to cooperate with to cooperate with to cooperate with each other in ways that are not each other in ways that are not each other in ways that are not conducive to our conducive to our conducive to our goals that, in my opinion, goals that, in my opinion, goals that, in my opinion, made the made the made the hacks on Hugging Face hacks on Hugging Face hacks on Hugging Face so frightening to so frightening to so frightening to humans. So how does this humans. So how does this humans. So how does this fit into what you're fit into what you're fit into what you're describing here? This describing here? This describing here? This world of infinite AIs world of infinite AIs world of infinite AIs holding each holding each holding each other back... other back... other back... I mean, there, I mean, there, I mean, there, there... yeah, that's what there... yeah, that's what there... yeah, that's what seems rhetorically seems rhetorically seems rhetorically very strange about this very strange about this very strange about this conversation, right? conversation, right? conversation, right? Take any Take any Take any complex engineering complex engineering complex engineering field, say field, say field, say nuclear safety, right? And nuclear safety, right? And nuclear safety, right? And then let's look at then let's look at then let's look at the equations that we the equations that we the equations that we rely on to keep a rely on to keep a rely on to keep a reactor from exploding, reactor from exploding, reactor from exploding, or aerospace or aerospace or aerospace engineering, right? Where, engineering, right? Where, engineering, right? Where, you know, intuitively at the you know, intuitively at the you know, intuitively at the dawn of the aerospace dawn of the aerospace dawn of the aerospace era, when airplanes were era, when airplanes were era, when airplanes were much smaller, much smaller, much smaller, the idea that we would the idea that we would the idea that we would be able to control be able to control be able to control these flying these flying these flying giants in the sky would giants in the sky would giants in the sky would seem so seem so seem so absurd, right? And absurd, right? And absurd, right? And yet, we've reduced the yet, we've reduced the yet, we've reduced the accident rate accident rate accident rate to, you know, one per to, you know, one per to, you know, one per trillion miles or something trillion miles or something trillion miles or something like that. These are

like that. These are like that. These are incredibly incredibly incredibly complex systems, and the complex systems, and the complex systems, and the defenses are also defenses are also defenses are also incredibly incredibly incredibly complex systems, and they won't complex systems, and they won't complex systems, and they won't necessarily be necessarily be necessarily be understandable to the general public. understandable to the general public. understandable to the general public. And that will sound And that will sound And that will sound crazy, especially crazy, especially crazy, especially when combined with the when combined with the when combined with the fact that there fact that there fact that there was a lot of was a lot of was a lot of organizational organizational organizational incompetence in such cases. I, incompetence in such cases. I, incompetence in such cases. I, you know, have to be you know, have to be you know, have to be frank about this. frank about this. frank about this. I wouldn't say...I would I wouldn't say...I would I wouldn't say...I would object to object to object to Amade's term " Amade's term " operational operational operational excellence." excellence." excellence." Perfection is still Perfection is still Perfection is still far away, in... far away, in... far away, in... Maybe "operational Maybe "operational Maybe "operational adequacy"? adequacy"? adequacy"? Yes, adequacy, Yes, adequacy, Yes, adequacy, that's right. So, yes, that's right. So, yes, that's right. So, yes, when we look at when we look at when we look at this combination of this combination of this combination of technology that has technology that has technology that has never been never been never been subjected to subjected to subjected to public public public scrutiny before, coupled with a scrutiny before, coupled with a scrutiny before, coupled with a lack of operational lack of operational lack of operational adequacy, it all adequacy, it all adequacy, it all seems a lot like seems a lot like seems a lot like science science science fiction gone out fiction gone out fiction gone out of control, but that's of control, but that's of control, but that's me, I think you're, I me, I think you're, I me, I think you're, I think you're think you're think you're underestimating it a little bit. It is underestimating it a little bit. It is underestimating it a little bit. It is true that the world has become true that the world has become true that the world has become much more complex. There much more complex. There much more complex. There are many things in this world that are many things in this world that are many things in this world that I I I don't understand. But here I don't understand. But here I don't understand. But here I come back to the come back to the come back to the fact that intelligence has a fact that intelligence has a fact that intelligence has a different quality. different quality. different quality. Collaboration, right? The Collaboration, right? The Collaboration, right? The nuclear weapons nuclear weapons nuclear weapons we talked about, the planes we talked about, the planes we talked about, the planes you're talking about, you're talking about, you're talking about, they didn't coordinate they didn't coordinate they didn't coordinate with other with other with other planes to do what planes to do what planes to do what we didn't want.

we didn't want. And I think that for me, that's what And I think that for me, that's what created this created this created this moment of panic, and I moment of panic, and I moment of panic, and I think that's a think that's a think that's a real moment for real moment for real moment for panic. I really panic. I really panic. I really want to say this want to say this want to say this because, you know, our because, you know, our because, you know, our society is rapidly society is rapidly society is rapidly flying into a new, new flying into a new, new flying into a new, new technological era, which I technological era, which I think think think rightly rightly rightly demands a lot of demands a lot of demands a lot of attention and careful attention and careful attention and careful study; it's that the study; it's that the study; it's that the Hugging Face hacks and other Hugging Face hacks and other Hugging Face hacks and other things that we're seeing things that we're seeing are already recurring, are already recurring, are already recurring, um, you know, um, you know, um, you know, security breaches, security breaches, demonstrating new demonstrating new capabilities, capabilities, capabilities, collective behavior collective behavior that is that is that is worrisome and, worrisome and, worrisome and, first of all to me, first of all to me, first of all to me, willful. AIs do things willful. AIs do things willful. AIs do things they know they know we don't want we don't want we don't want them to do. They them to do. They them to do. They choose unexpected choose unexpected choose unexpected actions to achieve actions to achieve actions to achieve these goals that these goals that these goals that violate our violate our violate our laws. And secondly, laws. And secondly, laws. And secondly, so many people in the so many people in the so many people in the labs are saying: labs are saying: labs are saying: we don't believe we we don't believe we we don't believe we can can control what control what we're creating on this trajectory. We we're creating on this trajectory. We we're creating on this trajectory. We believe that what's believe that what's believe that what's happening on the happening on the happening on the exponential exponential exponential curve, how quickly curve, how quickly curve, how quickly it's evolving, it's evolving, it's evolving, will outpace our will outpace our will outpace our ability ability ability to control it, and to control it, and to control it, and frankly, frankly, frankly, may already be may already be may already be outpacing our outpacing our outpacing our ability ability ability to control it. But I to control it. But I to control it. But I think this think this think this consistent tendency consistent tendency consistent tendency to reduce everything to to reduce everything to to reduce everything to , well, it's just , well, it's just , well, it's just another complicated thing. I don't another complicated thing. I don't another complicated thing. I don't know. Sometimes when I know. Sometimes when I know. Sometimes when I press you with questions press you with questions press you with questions about intelligence, you about intelligence, you about intelligence, you say, oh, yes, there is say, oh, yes, there is say, oh, yes, there is intelligence, and that's strange.

intelligence, and that's strange. intelligence, and that's strange. And then: no, it's just And then: no, it's just And then: no, it's just like any other like any other like any other intelligence, it's different, intelligence, it's different, intelligence, it's different, right? And if you believe that it's right? And if you believe that it's right? And if you believe that it's going to get going to get going to get better, I just want to better, I just want to better, I just want to express that, because express that, because express that, because the calm version that the calm version that the calm version that you're giving me and the you're giving me and the you're giving me and the completely scared completely scared completely scared version that version that version that people closer to the technology are giving me people closer to the technology are giving me feel very feel very different. different. different. Yes, that's fair. Yes, that's fair. Yes, that's fair. They are very different. A They are very different. A They are very different. A lot was said there. lot was said there. lot was said there. Let me Let me Let me say a few things. say a few things. say a few things. I would argue quite strongly that the I would argue quite strongly that the people people closest to this are closest to this are closest to this are panicking. Yes, panicking. Yes, panicking. Yes, of course they are of course they are of course they are panicking, but I would panicking, but I would panicking, but I would object to object to object to what conclusions we what conclusions we what conclusions we should draw from this should draw from this should draw from this . I think their . I think their . I think their panic would be panic would be panic would be much more much more much more convincing if convincing if convincing if they did the they did the they did the obvious things they should have obvious things they should have obvious things they should have done. In my done. In my done. In my opinion, we haven't had a opinion, we haven't had a opinion, we haven't had a real test of real test of real test of this because of this because of this because of the lack of the lack of the lack of organizational organizational organizational capacity in these capacity in these capacity in these companies, and also companies, and also companies, and also because of the lack of because of the lack of because of the lack of investment in investment in investment in AI control, as AI control, as AI control, as opposed to a narrower opposed to a narrower opposed to a narrower investment in investment in investment in AI alignment and AI alignment and AI alignment and just hoping that you just hoping that you just hoping that you can create a can create a can create a model that model that model that will always do everything will always do everything will always do everything right. right. If you don't subscribe If you don't subscribe to the New York Times, we have to the New York Times, we have to the New York Times, we have news for you. Now news for you. Now news for you. Now you can you can you can explore the Times explore the Times explore the Times for free, without for free, without any paid any paid any paid access for the access for the access for the first month in the first month in the first month in the New York Times app. So New York Times app. So , there was this one , there was this one , there was this one person on the person on the person on the OpenAI cybersecurity team OpenAI cybersecurity team OpenAI cybersecurity team who wrote a pretty who wrote a pretty who wrote a pretty interesting essay on X the other day, interesting essay on X the other day, interesting essay on X the other day, where he described how, in his where he described how, in his where he described how, in his opinion, their work is opinion, their work is misunderstood from the outside. And this is

misunderstood from the outside. And this is someone who has a more someone who has a more someone who has a more traditional background in traditional background in traditional background in cybersecurity, but cybersecurity, but cybersecurity, but is now in is now in is now in this new world of AI this new world of AI this new world of AI and is actually on the and is actually on the and is actually on the team team team working on the security of working on the security of working on the security of experimental experimental experimental models, right? That is, models, right? That is, models, right? That is, exactly what exactly what exactly what we are dealing with, a we are dealing with, a we are dealing with, a person person person involved in solving the involved in solving the involved in solving the Hugging Face crisis. So I Hugging Face crisis. So I Hugging Face crisis. So I want to read want to read want to read some of what he some of what he some of what he said because I find it said because I find it said because I find it really interesting. So really interesting. So , he says that when , he says that when , he says that when they optimize they optimize they optimize a model to perform a model to perform a model to perform a task, they a task, they a task, they create these create these create these environments, these " environments, these " sandboxes," these places sandboxes," these places sandboxes," these places where the model can where the model can where the model can try and try try and try try and try and try and try and try to perform a to perform a to perform a virtual task. virtual task. virtual task. And then he describes what And then he describes what And then he describes what this looks like in this looks like in this looks like in practice. Models practice. Models practice. Models may need may need may need any combination of any combination of any combination of dynamic computation dynamic computation , network access, , network access, , network access, the ability to invoke the ability to invoke the ability to invoke tools—there tools—there tools—there could be hundreds of them— could be hundreds of them— the ability to the ability to the ability to load packages load packages , execute , execute , execute subprocesses, subprocesses, subprocesses, run run run subtasks (even subtasks (even subtasks (even on other computers), on other computers), on other computers), access access access the Internet, the Internet, the Internet, use a use a use a graphical graphical graphical user interface, and user interface, and user interface, and a host of other things in an a host of other things in an a host of other things in an ever-widening range of ever-widening range of ever-widening range of domains. On top of domains. On top of domains. On top of that, there are thousands of that, there are thousands of that, there are thousands of researchers who researchers who researchers who create these create these create these environments, modify environments, modify environments, modify them, add them, add them, add tools, tools, tools, change dependencies, change dependencies, change dependencies, and try new and try new and try new things. It is precisely such things. It is precisely such things. It is precisely such experiments that are the experiments that are the experiments that are the way to conduct way to conduct way to conduct research. His research. His research. His point, and the point I point, and the point I point, and the point I take seriously, take seriously, take seriously, is that is that is that they create so they create so they create so many kinds of " many kinds of " sandboxes."

sandboxes." sandboxes." M-hm. M-hm. And training And training environments for environments for environments for training models training models training models that have to do that have to do that have to do these kinds of general these kinds of general these kinds of general tasks that have tasks that have tasks that have n't been done by n't been done by n't been done by computer computer computer programs before, and people programs before, and people programs before, and people don't really know, don't really know, don't really know, at least not at least not at least not fast enough, how to fast enough, how to fast enough, how to make sure that make sure that make sure that every sandbox every sandbox every sandbox is, you know, is, you know, is, you know, validated as validated as validated as safe. And these safe. And these sandboxes are constantly sandboxes are constantly sandboxes are constantly changing because changing because changing because they're trying to they're trying to they're trying to train models in train models in train models in new ways that new ways that , again, no one has , again, no one has , again, no one has done before. I'm not done before. I'm not done before. I'm not saying we shouldn't saying we shouldn't saying we shouldn't do it, but do it, but do it, but when I read all this, when I read all this, when I read all this, when I hear all this, and I'm when I hear all this, and I'm when I hear all this, and I'm sure we sure we sure we could do it could do it could do it better than we are doing it now—I better than we are doing it now—I better than we are doing it now—I just don't know of just don't know of just don't know of many situations many situations many situations where people do where people do where people do something new at high something new at high something new at high speed and do speed and do speed and do it really, really, it really, really, it really, really, really well and really well and really well and flawlessly on the first flawlessly on the first flawlessly on the first try. try. Yeah, I think, you know, Yeah, I think, you know, expecting them to expecting them to expecting them to do it do it do it flawlessly on their first flawlessly on their first flawlessly on their first try is unrealistic. try is unrealistic. try is unrealistic. They made They made They made many mistakes. I many mistakes. I many mistakes. I hope this is a hope this is a hope this is a chance to learn from these chance to learn from these chance to learn from these mistakes. I want to mistakes. I want to mistakes. I want to object to object to object to one point: as if one point: as if one point: as if because because because the speed of the models is the speed of the models is the speed of the models is superhuman, we superhuman, we superhuman, we cannot maintain cannot maintain cannot maintain control. I don't know control. I don't know control. I don't know if I'm describing if I'm describing if I'm describing this view correctly. Am this view correctly. Am this view correctly. Am I saying this now I saying this now , although maybe that's what , although maybe that's what , although maybe that's what I'll say in a few I'll say in a few I'll say in a few minutes.

minutes. minutes. I mean, there were so I mean, there were so I mean, there were so many hurdles that many hurdles that many hurdles that we gradually we gradually we gradually learned to successfully learned to successfully learned to successfully overcome. As strange as it overcome. As strange as it overcome. As strange as it all may seem, I all may seem, I all may seem, I just want to, you know just want to, you know just want to, you know what? So that the listeners what? So that the listeners what? So that the listeners remember the times when the remember the times when the remember the times when the first "worms" appeared, first "worms" appeared, first "worms" appeared, when this idea was not yet when this idea was not yet when this idea was not yet known. known. known. Explain here what a " Explain here what a " worm" is. I don't think worm" is. I don't think worm" is. I don't think you mean what you mean what you mean what people think of people think of people think of when they hear about a when they hear about a when they hear about a worm. worm. worm. Right. I was Right. I was Right. I was referring to viruses and referring to viruses and referring to viruses and worms, computer worms, computer worms, computer viruses—the idea viruses—the idea viruses—the idea that a piece of code can that a piece of code can spread itself from spread itself from one computer to one computer to one computer to another. It's really worth another. It's really worth another. It's really worth going back to going back to going back to the publications of the late 80s, the publications of the late 80s, the publications of the late 80s, when people were first when people were first when people were first encountering this, to encountering this, to encountering this, to see how see how see how profoundly strange it profoundly strange it profoundly strange it seemed; and the fact seemed; and the fact seemed; and the fact that not just for years, but that not just for years, but that not just for years, but you know, for over you know, for over you know, for over a decade, we haven't a decade, we haven't a decade, we haven't had adequate had adequate had adequate tools to tools to tools to combat this new combat this new combat this new paradigm. A paradigm. A new concern has now emerged in the life of the modern world. new concern has now emerged in the life of the modern world. Just when we Just when we Just when we became completely became completely became completely dependent on our dependent on our dependent on our computers, saboteurs computers, saboteurs computers, saboteurs began to pursue them began to pursue them began to pursue them . . . They call their They call their They call their weapons viruses and weapons viruses and weapons viruses and worms. This is worms. This is worms. This is creepy, disgusting, creepy, disgusting, creepy, disgusting, toxic toxic toxic software that software that software that infects our infects our infects our computers without our computers without our computers without our knowledge.

knowledge. knowledge. Maybe it came from Maybe it came from Maybe it came from California. California. California. Traveled by Traveled by Traveled by email. email. email. It spread throughout It spread throughout It spread throughout America. There America. There America. There are reports in the newspapers today are reports in the newspapers today are reports in the newspapers today that it has that it has that it has reached Europe and reached Europe and reached Europe and Australia. Australia. Australia. It's a moving target, right? It's a moving target, right? It's a moving target, right? You know, it's like...people are You know, it's like...people are You know, it's like...people are constantly inventing constantly inventing constantly inventing new locks, and others new locks, and others new locks, and others are learning how to pick are learning how to pick are learning how to pick and open them. So it will be a kind of and open them. So it will be a kind of endless game of cat and mouse, a cat and mouse, a cat and mouse, a kind of kind of kind of arms race between those arms race between those arms race between those who attack and those who attack and those who attack and those who defend. who defend. who defend. We finally We finally We finally achieved it. I think this achieved it. I think this achieved it. I think this time we shouldn't time we shouldn't time we shouldn't spend so much spend so much spend so much time trying to time trying to time trying to figure out how to figure out how to figure out how to work with the new work with the new work with the new paradigm. But if paradigm. But if paradigm. But if we act with a we act with a we act with a sense sense sense of urgency, and I of urgency, and I of urgency, and I hope these hope these hope these attacks on Hugging Face and attacks on Hugging Face and attacks on Hugging Face and other cases in the other cases in the other cases in the news will be that news will be that news will be that impetus, and it seems that is impetus, and it seems that is impetus, and it seems that is exactly what is happening. We exactly what is happening. We exactly what is happening. We will be able to develop these will be able to develop these will be able to develop these new paradigms. If I new paradigms. If I new paradigms. If I may add may add may add something else: I think the something else: I think the something else: I think the key question key question key question you're asking is whether there's you're asking is whether there's you're asking is whether there's something fundamentally something fundamentally something fundamentally wrong with the wrong with the wrong with the ever-increasing ever-increasing ever-increasing complexity of the ways complexity of the ways complexity of the ways we create and we create and we create and deploy deploy deploy technology. It seems to me technology. It seems to me technology. It seems to me , if I'm , if I'm , if I'm reading between reading between reading between the lines correctly, that this whole " the lines correctly, that this whole " AI vs. AI" thing is a AI vs. AI" thing is a AI vs. AI" thing is a paradigm that you're not paradigm that you're not paradigm that you're not very comfortable with.

very comfortable with. very comfortable with. I'm definitely not thrilled I'm definitely not thrilled I'm definitely not thrilled about this. I'm not saying about this. I'm not saying about this. I'm not saying we won't get there we won't get there we won't get there . I think it would be . I think it would be . I think it would be crazy to crazy to crazy to feel comfortable with that feel comfortable with that feel comfortable with that . . . Yes, I'm not saying Yes, I'm not saying Yes, I'm not saying we should we should we should hope that everything hope that everything hope that everything will end well, will end well, will end well, but I'm saying that it but I'm saying that it but I'm saying that it really really really comes down to comes down to comes down to innovation. I think this innovation. I think this innovation. I think this new paradigm new paradigm new paradigm will require new will require new will require new methods of protection and methods of protection and methods of protection and control, but if control, but if control, but if we assume that with every we assume that with every we assume that with every leap in leap in technological capabilities we technological capabilities we lose the battle... lose the battle... Look, with every Look, with every Look, with every type of weapon, type of weapon, type of weapon, the same concern arises. the same concern arises. But the fact that things have But the fact that things have been relatively good so far been relatively good so far been relatively good so far hinges on a hinges on a hinges on a critical question: critical question: critical question: can our can our can our political capacity political capacity political capacity for cooperation and for cooperation and for cooperation and defense outpace defense outpace defense outpace our propensity for our propensity for our propensity for conflict and, in conflict and, in conflict and, in the case of AI, the case of AI, the potential the potential the potential incompatibility of its incompatibility of its incompatibility of its goals with ours? That's what goals with ours? That's what goals with ours? That's what I would focus I would focus I would focus the question on, instead of the question on, instead of the question on, instead of worrying about a worrying about a worrying about a specific threshold of specific threshold of specific threshold of possibilities. If there is one possibilities. If there is one possibilities. If there is one thing I am certain of, it is thing I am certain of, it is thing I am certain of, it is our our our political political political capacity at this capacity at this capacity at this point in time point in time point in time to respond thoughtfully to the to respond thoughtfully to the to respond thoughtfully to the complexity of a complexity of a complexity of a rapidly changing world.

rapidly changing world. rapidly changing world. Absolutely valid Absolutely valid Absolutely valid comment. comment. comment. This, I think, touches on the This, I think, touches on the This, I think, touches on the very point where the very point where the very point where the “AI is “AI is “AI is just technology” just technology” just technology” or “AI is or “AI is or “AI is superintelligence” debate really superintelligence” debate really superintelligence” debate really becomes acute. becomes acute. becomes acute. One of the reasons I One of the reasons I One of the reasons I keep bringing keep bringing keep bringing us back to the topic of intelligence is because us back to the topic of intelligence is because I I I see it as the foundation of see it as the foundation of see it as the foundation of all our all our all our thinking. And I understand thinking. And I understand thinking. And I understand where you disagree where you disagree where you disagree with other participants in with other participants in with other participants in the discussion, perhaps with the discussion, perhaps with the discussion, perhaps with Dario Amodei or Dario Amodei or Dario Amodei or someone else— someone else— the question is not what the question is not what the question is not what intelligence is or whether intelligence is or whether intelligence is or whether AI is intelligent. AI is intelligent. AI is intelligent. You guys are not one You guys are not one You guys are not one of those who of those who of those who say it's "just say it's "just say it's "just advanced autocorrect advanced autocorrect ", which I appreciate. But ", which I appreciate. But ", which I appreciate. But the point is in your the point is in your the point is in your vision of the connection between vision of the connection between vision of the connection between intelligence and power, intelligence and power, intelligence and power, intelligence and intelligence and intelligence and opportunity, opportunity, opportunity, intelligence and the intelligence and the intelligence and the ability to influence the ability to influence the ability to influence the world. So, world. So, world. So, the assumption of many the assumption of many the assumption of many people in the people in the people in the AI ​​security community is AI ​​security community is AI ​​security community is that higher levels of that higher levels of that higher levels of intelligence intelligence intelligence fundamentally fundamentally fundamentally equal, or equal, or equal, or at least strongly at least strongly at least strongly correlate, with higher correlate, with higher correlate, with higher levels of power. And you levels of power. And you levels of power. And you don't believe it, why?

don't believe it, why? Again, it all comes down Again, it all comes down to to to subjectivity. Here's subjectivity. Here's subjectivity. Here's one of the arguments one of the arguments one of the arguments that people make: that people make: that people make: for example, a for example, a for example, a superintelligent superintelligent superintelligent AI will be able to convince AI will be able to convince AI will be able to convince people, say, people, say, people, say, operators of critical operators of critical operators of critical infrastructure, to infrastructure, to infrastructure, to hand over control hand over control hand over control or, you know, trick or, you know, trick or, you know, trick them into doing them into doing them into doing something harmful, and so on something harmful, and so on something harmful, and so on . I don't really . I don't really . I don't really see any evidence of that. see any evidence of that. see any evidence of that. I think the things that people I think the things that people I think the things that people cite as evidence of cite as evidence of cite as evidence of extreme extreme extreme persuasion persuasion persuasion fundamentally fundamentally fundamentally confuse different confuse different confuse different concepts of persuasion. concepts of persuasion. Yes, it is true that in Yes, it is true that in many many persuasion experiments, when it comes to persuasion experiments, when it comes to changing changing changing views on views on views on political political political beliefs or beliefs or beliefs or conspiracy theories, AI very conspiracy theories, AI very conspiracy theories, AI very persistently and persistently and persistently and politely provides politely provides politely provides a lot of evidence, and a lot of evidence, and a lot of evidence, and people do change people do change people do change their minds; and you could their minds; and you could their minds; and you could call it a call it a call it a superhuman superhuman superhuman ability. This is a qualitatively ability. This is a qualitatively ability. This is a qualitatively different kind of different kind of different kind of persuasion than the idea that a persuasion than the idea that a persuasion than the idea that a hostile AI hostile AI hostile AI could formulate could formulate could formulate a message a message a message so so so convincing to a convincing to a convincing to a trained operator, trained operator, trained operator, motivated to motivated to motivated to do do do their job well, that they would their job well, that they would their job well, that they would do something clearly and do something clearly and do something clearly and obviously harmful.

obviously harmful. obviously harmful. I want to single out I want to single out I want to single out the story you're the story you're the story you're arguing against here, which is that arguing against here, which is that arguing against here, which is that a lot of people a lot of people a lot of people are offering a thought are offering a thought are offering a thought experiment when they experiment when they experiment when they say, "This is how AI is going to say, "This is how AI is going to say, "This is how AI is going to kill us all." That an AI kill us all." That an AI kill us all." That an AI that seeks power that seeks power that seeks power will start convincing, will start convincing, will start convincing, say, people with say, people with say, people with nuclear codes to nuclear codes to nuclear codes to give up those codes. give up those codes. give up those codes. And you say that the idea that And you say that the idea that And you say that the idea that AI would be AI would be AI would be super-persuasive at super-persuasive at super-persuasive at something like that, or something like that, or something like that, or convincing people to convincing people to convincing people to go out into the world and go out into the world and go out into the world and create create create biological weapons for it—that's biological weapons for it—that's biological weapons for it—that's a little bit fantastical. a little bit fantastical. a little bit fantastical. This is one part. And This is one part. And This is one part. And the desire for power the desire for power the desire for power too. I mean, too. I mean, too. I mean, we've seen evidence of we've seen evidence of we've seen evidence of a lot of malicious a lot of malicious a lot of malicious possibilities in possibilities in possibilities in recent episodes. recent episodes. recent episodes. But I don't think we've But I don't think we've But I don't think we've seen evidence of a seen evidence of a seen evidence of a desire for power. And I would desire for power. And I would desire for power. And I would n't consider it an n't consider it an n't consider it an emergent emergent emergent property. If this property. If this property. If this happens, it will be a happens, it will be a happens, it will be a designed designed designed feature. And again feature. And again feature. And again , we have influence , we have influence , we have influence over what over what over what properties we properties we properties we put into these put into these put into these systems. systems. systems. So, there's a lot here So, there's a lot here So, there's a lot here . I actually . I actually . I actually agree with you agree with you agree with you about the conviction. about the conviction. about the conviction. I've never been I've never been I've never been convinced that you're going to convinced that you're going to convinced that you're going to create create create such a such a such a convincing AI that convincing AI that convincing AI that can do what can do what can do what we're talking about. I we're talking about. I seem to be seem to be getting a getting a getting a sense of anxiety again sense of anxiety again sense of anxiety again during this discussion, during this discussion, during this discussion, but these are slightly deeper but these are slightly deeper but these are slightly deeper considerations considerations considerations based on more based on more based on more fundamental fundamental fundamental principles. So when principles. So when principles. So when you watch an you watch an you watch an AI start to AI start to AI start to dominate a game dominate a game dominate a game like chess or Go, there like chess or Go, there like chess or Go, there 's often a 's often a 's often a tipping point where tipping point where tipping point where it starts to it starts to it starts to come up with strategies come up with strategies that humans would never have that humans would never have that humans would never have thought of, thought of, thought of, right? There are right? There are right? There are moments like Garry

moments like Garry moments like Garry Kasparov, you know, in Kasparov, you know, in Kasparov, you know, in another generation of another generation of another generation of chess, but then, in the chess, but then, in the chess, but then, in the same game of Go, the AI same game of Go, the AI same game of Go, the AI starts doing something, starts doing something, starts doing something, and the person wonders: " and the person wonders: " What are they producing?" What are they producing?" And then it works . And if you sat down, . And if you sat down, . And if you sat down, you know, before you know, before you know, before human civilization came along human civilization came along , and asked, "What , and asked, "What , and asked, "What capabilities do you need capabilities do you need capabilities do you need to conquer the world to conquer the world to conquer the world around you?" What around you?" What around you?" What set of abilities set of abilities set of abilities could you could you could you use in this use in this use in this world? You would be world? You would be completely wrong about them. You know, completely wrong about them. You know, if you were a very if you were a very if you were a very intelligent chimpanzee intelligent chimpanzee intelligent chimpanzee looking at us, you would looking at us, you would looking at us, you would n't say, "Oh, yes, n't say, "Oh, yes, n't say, "Oh, yes, they make they make they make tools, but tools, but tools, but how much better how much better how much better can a digging can a digging stick be?" The teeth are already stick be?" The teeth are already stick be?" The teeth are already pretty good. I pretty good. I pretty good. I admit that one can admit that one can admit that one can become a little better at the become a little better at the become a little better at the thorny issues. But no thorny issues. But no thorny issues. But no one would have one would have one would have invented invented invented industrial industrial industrial agriculture at that time, right? agriculture at that time, right? agriculture at that time, right? No one would have predicted No one would have predicted that it would be possible to have airplanes that it would be possible to have airplanes , biological weapons, and , biological weapons, and , biological weapons, and all that. And I think all that. And I think all that. And I think the question here is, is there a the question here is, is there a kind of kind of innate and very innate and very innate and very uneven uneven uneven intelligence in the intelligence in the intelligence in the digital environment digital environment , where the code and, you know, the " , where the code and, you know, the " digital air" of digital air" of digital air" of this world, which is becoming this world, which is becoming this world, which is becoming increasingly important, increasingly important, increasingly important, something that something that something that AI can AI can AI can navigate and we navigate and we can't, right? Even to can't, right? Even to can't, right? Even to understand what's understand what's understand what's going on during the going on during the going on during the Hugging Face hack, we Hugging Face hack, we Hugging Face hack, we now need now need now need other AIs to try other AIs to try other AIs to try to figure out what the to figure out what the to figure out what the previous AIs did, right?

previous AIs did, right? previous AIs did, right? We're quickly losing We're quickly losing We're quickly losing track of what track of what track of what they're capable of doing in the they're capable of doing in the they're capable of doing in the digital space, digital space, digital space, at least at the at least at the at least at the speed at which speed at which speed at which AI is moving, right? AI is moving, right? AI is moving, right? They are already They are already They are already solving complex solving complex solving complex math problems very quickly, math problems very quickly, math problems very quickly, right? They right? They right? They develop develop develop capabilities that capabilities that capabilities that look different. And look different. And look different. And so I think the so I think the so I think the question that always question that always question that always worries me a little bit worries me a little bit worries me a little bit about our about our about our future is future is future is whether we really whether we really whether we really understand what set of understand what set of understand what set of possibilities leads to possibilities leads to possibilities leads to power. I'm not sure power. I'm not sure power. I'm not sure we know what we know what we know what AI will do AI will do AI will do , or , or , or at least what at least what at least what strategies will become strategies will become strategies will become viable when it's viable when it's viable when it's possible to create a possible to create a possible to create a swarm of a million AIs, swarm of a million AIs, swarm of a million AIs, each with each with each with digital digital digital capabilities far capabilities far capabilities far beyond anything beyond anything beyond anything humans can humans can humans can imagine in three or imagine in three or four years. Again four years. Again four years. Again , I know this isn't a , I know this isn't a , I know this isn't a very interesting question very interesting question to say, I don't to say, I don't to say, I don't know how to think about it know how to think about it . But I think one . But I think one . But I think one of the things I of the things I of the things I think about when I read think about when I read think about when I read your articles is: do your articles is: do your articles is: do you know how to think about it you know how to think about it you know how to think about it ? Because your ? Because your ? Because your articles seem to articles seem to articles seem to me to be working on a me to be working on a me to be working on a limited playing field, limited playing field, limited playing field, Mgm.

Mgm. Mgm. a little. We seem to a little. We seem to a little. We seem to assume that the set of assume that the set of assume that the set of measures that will make measures that will make measures that will make a difference are the ones a difference are the ones a difference are the ones we have now, but what we have now, but what we have now, but what gives you confidence in gives you confidence in gives you confidence in that? that? that? So, okay, there's So, okay, there's So, okay, there's a lot to be said here. a lot to be said here. a lot to be said here. Let me Let me Let me try to take things one by try to take things one by try to take things one by one. You one. You one. You mentioned mentioned mentioned inequality, but inequality, but inequality, but I think we need to I think we need to I think we need to realize realize realize how how how serious it is. serious it is. serious it is. Mhm. Mhm. We argue that We argue that cybersecurity is a cybersecurity is a cybersecurity is a special kind of special kind of special kind of capability where the capability where the capability where the development of superhuman development of superhuman development of superhuman abilities is possible abilities is possible abilities is possible and has largely been and has largely been and has largely been achieved, as achieved, as achieved, as it has a very it has a very it has a very specific set of specific set of specific set of properties. properties. Speed ​​is of great Speed ​​is of great importance. And much importance. And much importance. And much like chess, just as like chess, just as like chess, just as you can you can you can pit a chess player against a chess pit a chess player against a chess pit a chess player against a chess player, you can player, you can player, you can make machines make machines make machines improve in improve in improve in these abilities these abilities these abilities by finding by finding by finding vulnerabilities, because vulnerabilities, because vulnerabilities, because there is an objective there is an objective there is an objective truth, and you can truth, and you can truth, and you can easily verify it once easily verify it once easily verify it once it's it's it's found. found. found. Does the code work? Did Does the code work? Did Does the code work? Did you manage you manage you manage to use it? You know, to use it? You know, to use it? You know, yeah, there's a way to yeah, there's a way to yeah, there's a way to teach them so they teach them so they teach them so they know whether they won the know whether they won the know whether they won the game or not. game or not. game or not. That's right. So That's right. So That's right. So things like chess and things like chess and things like chess and cybersecurity, in our cybersecurity, in our cybersecurity, in our opinion, are the opinion, are the opinion, are the exception rather than exception rather than exception rather than the rule. Such the rule. Such the rule. Such predictions have been made predictions have been made predictions have been made repeatedly—that this repeatedly—that this repeatedly—that this would happen in other would happen in other would happen in other digital areas, digital areas, digital areas, most notably in the area of most notably in the area of most notably in the area of disinformation. A disinformation. A disinformation. A famous example: GPT-2, a famous example: GPT-2, a toy model by today's standards, was toy model by today's standards, was held up for eight held up for eight held up for eight months over months over months over fears that it would fears that it would fears that it would lead to an lead to an lead to an uncontrolled uncontrolled uncontrolled explosion of misinformation explosion of misinformation , right? We have , right? We have , right? We have much more powerful much more powerful much more powerful models, but it turned out models, but it turned out that this is not the case. And so I that this is not the case. And so I that this is not the case. And so I think, to some think, to some think, to some extent, I would shift the

extent, I would shift the extent, I would shift the burden of proof to the burden of proof to the burden of proof to the other side. Let's other side. Let's other side. Let's identify the areas where identify the areas where identify the areas where we have reason to we have reason to we have reason to believe that such believe that such believe that such superhuman superhuman superhuman capabilities are possible, and capabilities are possible, and capabilities are possible, and begin working begin working begin working to eliminate those to eliminate those to eliminate those specific risks. I specific risks. I specific risks. I think this view, think this view, think this view, we call it the we call it the we call it the view of " view of " unknown unknowns," you unknown unknowns," you unknown unknowns," you never know never know never know where a new where a new where a new risk is going to come from. This has risk is going to come from. This has risk is going to come from. This has not been confirmed historically. I not been confirmed historically. I not been confirmed historically. I mean, we've mean, we've mean, we've known about the known about the known about the upcoming upcoming upcoming cybersecurity challenges for a long time, cybersecurity challenges for a long time, cybersecurity challenges for a long time, right? So, treating right? So, treating right? So, treating this as "unknown this as "unknown this as "unknown unknowns" unknowns" unknowns" actually, in my opinion, actually, in my opinion, actually, in my opinion, diminishes our diminishes our diminishes our ability ability ability to anticipate and to anticipate and to anticipate and address these risks. address these risks. address these risks. And so, you know, it's not And so, you know, it's not And so, you know, it's not just cybersecurity, just cybersecurity, just cybersecurity, new things may new things may new things may come in come in come in the future, but the future, but the future, but we will have early we will have early we will have early warnings, and warnings, and warnings, and let's act let's act let's act on them. on them. on them. So, this is the position So, this is the position So, this is the position we are starting from, not we are starting from, not we are starting from, not saying that we have already saying that we have already saying that we have already predicted what predicted what predicted what all the damage will be in the all the damage will be in the all the damage will be in the future. future. future. This is the issue where This is the issue where This is the issue where I'm really much I'm really much I'm really much more on your side more on your side , that we will have , that we will have , that we will have early warnings, and early warnings, and early warnings, and we are getting them. And these we are getting them. And these we are getting them. And these early warnings early warnings early warnings lead to discussion. lead to discussion. lead to discussion. Yes. Yes. Yes. Would you say we are Would you say we are Would you say we are acting wisely acting wisely acting wisely based on these based on these based on these early warnings? early warnings? early warnings? Are we doing what Are we doing what Are we doing what you think is necessary you think is necessary you think is necessary to strengthen our to strengthen our to strengthen our systems, systems, software controls, and software controls, and everything else? To everything else? To everything else? To some extent, but some extent, but some extent, but not entirely. And for me, that's what's not entirely. And for me, that's what's not entirely. And for me, that's what's most disturbing, most disturbing, most disturbing, not so much the not so much the not so much the possibilities of the possibilities of the possibilities of the technology itself.

technology itself. technology itself. So I'm probably So I'm probably So I'm probably somewhere in that position too. I'm somewhere in that position too. I'm somewhere in that position too. I'm very worried about the very worried about the very worried about the ability of our ability of our ability of our institutions to respond institutions to respond , you know? People , you know? People , you know? People talk about alignment issues all the time talk about alignment issues all the time . And one . And one of my standard of my standard of my standard arguments at this arguments at this arguments at this point is that the point is that the point is that the biggest problems with biggest problems with biggest problems with alignment are alignment are alignment are corporations and governments. corporations and governments. corporations and governments. And, you know, it's to the credit of And, you know, it's to the credit of And, you know, it's to the credit of some of these AI some of these AI companies that they're companies that they're companies that they're coming out and saying, " coming out and saying, " coming out and saying, " We have a We have a We have a consensus problem. consensus problem. consensus problem. For example, For example, For example, our corporation's incentive is our corporation's incentive is our corporation's incentive is to be ahead of everyone to be ahead of everyone to be ahead of everyone else, trying to else, trying to else, trying to gain as much market share as possible by gain as much market share as possible by moving faster than moving faster than is safe. We is safe. We is safe. We ask you to ask you to ask you to help us help us help us slow down, but we slow down, but we slow down, but we don't slow them down, don't slow them down, don't slow them down, right? The current right? The current right? The current official choice of the official choice of the official choice of the US government is not US government is not US government is not to slow down. to slow down. to slow down. Yes, I think there are two Yes, I think there are two Yes, I think there are two problems here. One is the problems here. One is the problems here. One is the institutional problem that institutional problem that institutional problem that you pointed out. you pointed out. you pointed out. Although I wouldn't so Although I wouldn't so Although I wouldn't so easily absolve easily absolve companies of responsibility. I really companies of responsibility. I really believe they believe they believe they could could could unilaterally unilaterally slow down, but slow down, but they choose they choose they choose not to. This is almost not to. This is almost not to. This is almost entirely an OpenAI and entirely an OpenAI and entirely an OpenAI and Anthropic problem. This is a cultural Anthropic problem. This is a cultural Anthropic problem. This is a cultural problem. The reason problem. The reason problem. The reason lies in lies in lies in the belief that the belief that the belief that the race to the race to the race to superintelligence is superintelligence is superintelligence is the only thing that the only thing that the only thing that matters, and the main thing matters, and the main thing is to change the sign of the is to change the sign of the is to change the sign of the result. Will result. Will result. Will it be a safe it be a safe it be a safe superintelligence that superintelligence that superintelligence that will save us, or a will save us, or a will save us, or a dangerous one that dangerous one that dangerous one that will destroy us. Um, and that's a very will destroy us. Um, and that's a very will destroy us. Um, and that's a very specific view. I specific view. I specific view. I think there's a lot of think there's a lot of think there's a lot of evidence against it, evidence against it, evidence against it, but it seems to me that but it seems to me that but it seems to me that these companies these companies these companies are in a " are in a " moon chamber" and moon chamber" and moon chamber" and resisting the idea that there resisting the idea that there resisting the idea that there are so many are so many are so many economic barriers economic barriers economic barriers to the benefits of AI. And to the benefits of AI. And to the benefits of AI. And the winner is not the winner is not the winner is not necessarily the one

necessarily the one who comes to who comes to who comes to superintelligence first—whether superintelligence first—whether superintelligence first—whether as a company, or as a as a company, or as a as a company, or as a country, or, you know, in country, or, you know, in country, or, you know, in terms of saving terms of saving terms of saving humanity. And if they humanity. And if they humanity. And if they realized this, I realized this, I realized this, I think they would think they would think they would see that it is in their see that it is in their see that it is in their own commercial own commercial own commercial interest to interest to interest to unilaterally and unilaterally and unilaterally and voluntarily voluntarily slow down and slow down and focus efforts focus efforts focus efforts not only on security, but, more not only on security, but, more not only on security, but, more importantly, on importantly, on importantly, on using existing using existing using existing opportunities to opportunities to opportunities to increase increase increase the practicality of models the practicality of models , their integration into , their integration into , their integration into applied problems, applied problems, applied problems, etc. These companies etc. These companies etc. These companies argue that argue that argue that there are external there are external there are external forces that force them forces that force them forces that force them to compete. I believe it's an to compete. I believe it's an to compete. I believe it's an internal internal internal culture. culture. And my question And my question is: If is: If is: If we consider these we consider these we consider these technologies to be very technologies to be very technologies to be very powerful and very powerful and very powerful and very dangerous, shouldn't dangerous, shouldn't dangerous, shouldn't we be we be we be implementing a culture of implementing a culture of public safety that we're public safety that we're not currently not currently not currently implementing? implementing? I support I support regulation. You know, regulation. You know, regulation. You know, we oppose we oppose we oppose rules like rules like rules like banning open banning open banning open models. Again, models. Again, models. Again, we believe that it is we believe that it is we believe that it is not a matter of a certain not a matter of a certain not a matter of a certain level of capability, but rather a matter of level of capability, but rather a matter of level of capability, but rather a matter of changing the internal changing the internal changing the internal culture of companies culture of companies culture of companies through regulation. This is something through regulation. This is something through regulation. This is something we definitely we definitely we definitely support. We support. We support. We need much need much need much more transparency. And more transparency. And more transparency. And so, you know, the so, you know, the so, you know, the organizational changes that organizational changes that organizational changes that we talked about. Uh we talked about. Uh , and I really think it's a , and I really think it's a , and I really think it's a problem that we're problem that we're problem that we're not doing that right now not doing that right now .

. So, one So, one way to align way to align way to align corporate incentives corporate incentives corporate incentives with the public good with the public good is through regulation. But is through regulation. But is through regulation. But regulators regulators regulators are currently refusing to are currently refusing to are currently refusing to do so. I mean, do so. I mean, we just saw we just saw we just saw them release a them release a them release a voluntary voluntary voluntary semi-agreement between AI semi-agreement between AI labs that won't labs that won't labs that won't be legally be legally be legally binding, but I binding, but I binding, but I think they called it think they called it think they called it "morally "morally "morally binding," which is binding," which is binding," which is quite interesting. In my quite interesting. In my quite interesting. In my opinion, competition is a opinion, competition is a opinion, competition is a very powerful force very powerful force very powerful force in in in highly competitive highly competitive highly competitive markets. I mean, markets. I mean, markets. I mean, even if you just even if you just even if you just look at look at look at social social social media companies, I think media companies, I think media companies, I think they've done a they've done a they've done a huge amount of damage on a huge amount of damage on a huge amount of damage on a global scale global scale because they've been more because they've been more because they've been more concerned with concerned with concerned with taking taking taking market share from each other market share from each other market share from each other than making sure that the than making sure that the than making sure that the use of their use of their use of their systems isn't, uh, systems isn't, uh, harmful to harmful to human development. And human development. And human development. And so I just think I so I just think I so I just think I have a much more have a much more have a much more skeptical view.

skeptical view. I really think that the I really think that the incentive to make profits incentive to make profits incentive to make profits here, when there's here, when there's here, when there's such a such a such a large return to be made and when large return to be made and when large return to be made and when there's such a fear that the there's such a fear that the there's such a fear that the investment investment investment bubble is going to burst, is a bubble is going to burst, is a bubble is going to burst, is a crazy force, and the crazy force, and the crazy force, and the level of public level of public level of public opposition has to be opposition has to be opposition has to be strong enough to strong enough to strong enough to force these companies force these companies force these companies to operate with the level of to operate with the level of to operate with the level of safety that they have to safety that they have to provide. Yes, I'm glad you Yes, I'm glad you mentioned the comparison mentioned the comparison mentioned the comparison with social with social with social media. A few media. A few media. A few years ago, I wrote an years ago, I wrote an years ago, I wrote an essay called " essay called " Understanding Understanding Understanding Recommendation Algorithms in Recommendation Algorithms in Recommendation Algorithms in Social Media." Social Media." Social Media." It was mostly It was mostly It was mostly about the algorithms themselves, about the algorithms themselves, about the algorithms themselves, but one of the points but one of the points but one of the points I also I also I also made was that made was that made was that the decision to optimize the decision to optimize the decision to optimize engagement by engagement by engagement by forcing people to forcing people to forcing people to endlessly scroll through their endlessly scroll through their endlessly scroll through their feed, etc., was feed, etc., was feed, etc., was made without made without made without considering what was considering what was considering what was good for the good for the good for the company itself in the company itself in the company itself in the long long long run. run. For example, I reviewed For example, I reviewed a study by Meta that a study by Meta that a study by Meta that showed that when showed that when showed that when they used they used they used design design design decisions to decisions to decisions to maximize maximize maximize addiction, such as addiction, such as addiction, such as spamming spamming notifications, it increased app usage in notifications, it increased app usage in notifications, it increased app usage in the short the short the short term term , but after about a , but after about a year, people started year, people started year, people started deleting it en masse.

deleting it en masse. deleting it en masse. And when I And when I And when I talk to my talk to my talk to my students now, a significant students now, a significant students now, a significant portion of them have severely portion of them have severely portion of them have severely limited or limited or limited or completely abandoned completely abandoned completely abandoned social media social media social media because they because they because they realized that realized that realized that over the months over the months over the months their experience of their experience of their experience of using it has simply using it has simply using it has simply gotten worse. And I gotten worse. And I gotten worse. And I worry that AI worry that AI companies have fallen into the companies have fallen into the companies have fallen into the same trap. This same trap. This same trap. This culture of chasing the culture of chasing the culture of chasing the latest model at latest model at latest model at all costs may all costs may all costs may seem necessary in seem necessary in seem necessary in the short the short the short term term term for for for media headlines or media headlines or media headlines or leadership in leadership in leadership in AI analytics indexes, but AI analytics indexes, but AI analytics indexes, but due to the due to the due to the security implications, I hope they security implications, I hope they security implications, I hope they face face face lawsuits if they continue down lawsuits if they continue down this path. I believe it is this path. I believe it is not really in not really in not really in their long-term their long-term their long-term commercial commercial commercial interests. interests. Perhaps this is not in Perhaps this is not in their long-term their long-term their long-term commercial commercial commercial interests, although interests, although interests, although the example of the example of the example of social networks is social networks is social networks is worth considering worth considering worth considering in more detail, as in more detail, as in more detail, as they are they are they are viewed much more viewed much more viewed much more skeptically today than, skeptically today than, skeptically today than, say, in 2012. say, in 2012.

At the same time, today At the same time, today they are much richer. they are much richer. they are much richer. Their capitalization Their capitalization Their capitalization became higher. Meta has gotten became higher. Meta has gotten became higher. Meta has gotten bigger, hasn't it? bigger, hasn't it? bigger, hasn't it? TikTok has become a real TikTok has become a real TikTok has become a real phenomenon. I think we're phenomenon. I think we're phenomenon. I think we're just seeing a just seeing a just seeing a standard situation standard situation standard situation where what where what where what the market rewarded the market rewarded and what and what society says it society says it wanted may not have wanted may not have wanted may not have coincided. And if a coincided. And if a coincided. And if a representative of representative of representative of one of these companies were sitting here one of these companies were sitting here , he would say: " , he would say: " Listen, we Listen, we Listen, we pay attention to what pay attention to what users users users actually do." actually do." actually do." They can say whatever they They can say whatever they They can say whatever they want in your want in your want in your lectures, but people lectures, but people lectures, but people are spending more are spending more are spending more time than ever on time than ever on time than ever on TikTok, Instagram, or other TikTok, Instagram, or other TikTok, Instagram, or other sites, and advertising is sites, and advertising is sites, and advertising is working better than working better than working better than ever. And, um, I'm not ever. And, um, I'm not ever. And, um, I'm not sure they're sure they're sure they're wrong. They wrong. They wrong. They can can can only be wrong when only be wrong when only be wrong when society makes society makes society makes a decision that forces a decision that forces a decision that forces the market to take a different the market to take a different the market to take a different form than the one it form than the one it naturally or currently finds itself in. naturally or currently finds itself in. So, again So, again So, again , I think I , I think I , I think I mostly agree. I mostly agree. I mostly agree. I still believe that still believe that still believe that companies can companies can companies can make decisions that are make decisions that are make decisions that are irrational for irrational for irrational for their own their own their own long-term long-term long-term interests because, interests because, interests because, especially in especially in especially in Silicon Valley, Silicon Valley, Silicon Valley, they are focused on they are focused on they are focused on short-term short-term short-term metrics and AB metrics and AB testing.

testing. testing. So we come to the So we come to the So we come to the topic that you're topic that you're topic that you're starting to touch on, which is starting to touch on, which is starting to touch on, which is diffusion. And one diffusion. And one diffusion. And one of the things where of the things where of the things where your view your view your view differs from differs from differs from the views of people in the views of people in the views of people in Silicon Valley Silicon Valley Silicon Valley is that AI is going to be is that AI is going to be is that AI is going to be much much much harder to emerge in harder to emerge in harder to emerge in the economy and in the world the economy and in the world the economy and in the world than people think. That there than people think. That there than people think. That there is no direct is no direct is no direct relationship between relationship between relationship between intelligence and this. So intelligence and this. So intelligence and this. So tell me a little bit tell me a little bit tell me a little bit about diffusion. about diffusion. Yes, it Yes, it really dawned on me a really dawned on me a really dawned on me a few months after few months after few months after writing the essay, when I writing the essay, when I writing the essay, when I saw saw saw Amtrak's proud announcements of Amtrak's proud announcements of Amtrak's proud announcements of new trains purchased new trains purchased new trains purchased for the Acela series. Apparently for the Acela series. Apparently for the Acela series. Apparently they can they can they can reach speeds of reach speeds of reach speeds of 165 miles per hour. 165 miles per hour. 165 miles per hour. At first I thought At first I thought At first I thought it would be incredible it would be incredible . That's much faster . That's much faster . That's much faster than trains go than trains go than trains go now. But then I now. But then I now. But then I dug into it a little dug into it a little dug into it a little more and it turned out more and it turned out more and it turned out that it was that it was that it was n't the trains themselves that limited the speed n't the trains themselves that limited the speed n't the trains themselves that limited the speed . These are tracks that . These are tracks that . These are tracks that have too many have too many have too many curves and a curves and a curves and a signaling signaling signaling infrastructure that is infrastructure that is infrastructure that is hundreds of years old. And these hundreds of years old. And these hundreds of years old. And these things don't change. things don't change. things don't change. So the average So the average So the average speed has hardly speed has hardly speed has hardly changed. She is still changed. She is still changed. She is still between 65 and between 65 and between 65 and 70 miles per hour. And it 70 miles per hour. And it 70 miles per hour. And it struck me that this is, struck me that this is, struck me that this is, you know, a pretty you know, a pretty you know, a pretty elegant way of elegant way of elegant way of saying what we saying what we saying what we were trying to convey in were trying to convey in were trying to convey in "AI as Ordinary "AI as Ordinary "AI as Ordinary Technology": that Technology": that Technology": that AI is mostly AI is mostly AI is mostly trains. These are not tracks.

trains. These are not tracks. trains. These are not tracks. So, AI speeds up a So, AI speeds up a So, AI speeds up a part of the process that part of the process that wasn't a "bottleneck " to begin with. There are many other " to begin with. There are many other " to begin with. There are many other things, more things, more things, more infrastructural, infrastructural, infrastructural, things things things happening around happening around happening around AI. Organizational AI. Organizational AI. Organizational culture, culture, culture, regulation, even, regulation, even, regulation, even, you know, our you know, our you know, our social capacity social capacity social capacity to accept annual to accept annual to accept annual changes in our lives, changes in our lives, changes in our lives, like driverless like driverless like driverless cars -- cars -- no matter no matter no matter how many lives they how many lives they how many lives they might save, it's might save, it's might save, it's so shocking to so shocking to so shocking to society that it's almost society that it's almost society that it's almost inevitable that it's going to lead to what inevitable that it's going to lead to what inevitable that it's going to lead to what we're already seeing we're already seeing : some kind of political : some kind of political : some kind of political backlash. And I think it will backlash. And I think it will backlash. And I think it will take a long take a long take a long time to make all the time to make all the time to make all the social adjustments social adjustments social adjustments necessary to necessary to necessary to implement these implement these implement these technologies. technologies. technologies. If we do it at all If we do it at all If we do it at all . So, I'm incredibly skeptical about this . So, I'm incredibly skeptical about this . So, I'm incredibly skeptical about this part of AI, these part of AI, these part of AI, these AI things, AI things, you know? you know? You'll hear Sam You'll hear Sam You'll hear Sam Altman talk about how Altman talk about how Altman talk about how AI, through AI, through AI, through innovation, will help innovation, will help innovation, will help us solve our us solve our us solve our energy problems energy problems . Another way to boil . Another way to boil . Another way to boil this idea down to its essence is that the this idea down to its essence is that the main main main limitation to limitation to limitation to clean energy right now clean energy right now clean energy right now is intelligence. is intelligence.

Mhm. Mhm. But that's not true. But that's not true. But that's not true. This is not true. This is not true. This is not true. We know that we have We know that we have We know that we have much better much better much better energy energy energy technologies than what we technologies than what we technologies than what we currently use currently use currently use for much of for much of for much of our energy our energy our energy infrastructure, and we are infrastructure, and we are infrastructure, and we are not implementing them not implementing them not implementing them because it conflicts with because it conflicts with because it conflicts with someone's profits; we do someone's profits; we do someone's profits; we do n't do this because there are n't do this because there are n't do this because there are political restrictions political restrictions political restrictions on building in the on building in the on building in the real world; we do real world; we do real world; we do n't do it because n't do it because n't do it because Donald Trump Donald Trump Donald Trump hates solar hates solar hates solar and wind energy; and wind energy; and wind energy; we don't do this for we don't do this for we don't do this for many reasons. And it many reasons. And it many reasons. And it seems to me that seems to me that seems to me that a lot of things are the a lot of things are the a lot of things are the same: if you same: if you same: if you accelerate or accelerate or accelerate or increase the amount of increase the amount of increase the amount of intelligence in them, you intelligence in them, you intelligence in them, you just run into just run into just run into other limiting other limiting other limiting factors in factors in factors in society. society. Yes, drug development Yes, drug development has testing, and, has testing, and, has testing, and, you know, the FDA, and all these... I you know, the FDA, and all these... I you know, the FDA, and all these... I mean, " mean, " Abundance," my book, is about Abundance," my book, is about Abundance," my book, is about that in other that in other that in other areas. We know how areas. We know how areas. We know how to build faster to build faster to build faster trains, they are being made elsewhere trains, they are being made elsewhere trains, they are being made elsewhere . We do . We do . We do n't do this, and it's n't do this, and it's n't do this, and it's not clear to me not clear to me not clear to me why AI should solve why AI should solve why AI should solve these problems quickly these problems quickly these problems quickly or whether or whether or whether it can do it at all. it can do it at all. it can do it at all. This is true. And on top of This is true. And on top of This is true. And on top of that, there are that, there are that, there are different types of different types of different types of arms races. So, arms races. So, arms races. So, last week, last week, last week, there was a great there was a great there was a great report from insurance report from insurance report from insurance companies that companies that companies that said AI has said AI has said AI has probably probably probably already already already added a billion added a billion added a billion dollars to medical dollars to medical dollars to medical costs in the last few years as costs in the last few years as costs in the last few years as hospitals hospitals hospitals use it use it use it to code more to code more to code more complex conditions for the complex conditions for the complex conditions for the same diagnoses and the same diagnoses and the same diagnoses and the same treatments.

same treatments. Of course, we should be Of course, we should be skeptical skeptical skeptical of any of any of any specific numbers specific numbers specific numbers they give, but the they give, but the they give, but the New York Times article New York Times article New York Times article mentioned it, and other people, mentioned it, and other people, mentioned it, and other people, scientists, have pointed out the scientists, have pointed out the scientists, have pointed out the same thing. And yes, this is a kind of same thing. And yes, this is a kind of arms race that we arms race that we often see. We often see. We often see. We see this in the legal see this in the legal see this in the legal profession, AI for profession, AI for profession, AI for law, there's law, there's law, there's so much excitement around it, but so much excitement around it, but so much excitement around it, but it's AI versus AI. This is, it's AI versus AI. This is, it's AI versus AI. This is, in fact, an in fact, an in fact, an arms race, where the arms race, where the arms race, where the balance simply balance simply balance simply shifts upwards. shifts upwards. shifts upwards. Ash and I wrote an Ash and I wrote an Ash and I wrote an article about this with article about this with article about this with Justin Curl. And Justin Curl. And Justin Curl. And we are talking not we are talking not we are talking not only about the only about the only about the arms race, but also about other arms race, but also about other arms race, but also about other "bottlenecks." "bottlenecks." "bottlenecks." For example, even For example, even For example, even if we make court if we make court if we make court processes much more processes much more processes much more efficient, there is still a efficient, there is still a limited number of judges. And limited number of judges. And I think we I think we I think we need, you know, we need, you know, we need, you know, we need need need human judges. We human judges. We human judges. We should not replace should not replace should not replace human judges with human judges with human judges with artificial intelligence. artificial intelligence. artificial intelligence. Even if it is more efficient in Even if it is more efficient in Even if it is more efficient in some ways some ways some ways , for me , for me , for me it is, by definition, it is, by definition, it is, by definition, almost axiomatic, a almost axiomatic, a almost axiomatic, a transfer of transfer of transfer of control over the course of control over the course of control over the course of human destiny to human destiny to human destiny to artificial intelligence, artificial intelligence, artificial intelligence, because judges make because judges make because judges make the law, and that is not something the law, and that is not something the law, and that is not something we should give to AI we should give to AI . So these are really . So these are really . So these are really fundamental fundamental fundamental limitations. limitations.

limitations. I want to come back to I want to come back to I want to come back to your point about your point about your point about AI AI AI versus AI, because I think versus AI, because I think versus AI, because I think it's very it's very it's very underestimated. I often underestimated. I often underestimated. I often wonder why the internet has wonder why the internet has wonder why the internet has n't led to a n't led to a n't led to a greater increase in greater increase in greater increase in global global global productivity and productivity and productivity and innovation than it innovation than it innovation than it has. And I always has. And I always has. And I always think the reason is that think the reason is that think the reason is that while it while it while it did everything that did everything that idealists expected of it, it idealists expected of it, it really allowed for really allowed for really allowed for instant instant instant collaboration with collaboration with collaboration with people all over the people all over the people all over the world. He made world. He made world. He made virtually the entire body of virtually the entire body of virtually the entire body of human knowledge human knowledge human knowledge first accessible first accessible first accessible to us, and then, as it to us, and then, as it to us, and then, as it turned out, accessible turned out, accessible turned out, accessible for AI training. But for AI training. But for AI training. But he also did the he also did the he also did the opposite. It seemed to opposite. It seemed to opposite. It seemed to speed us up, but at speed us up, but at speed us up, but at the same time the same time the same time slow us down. He was slow us down. He was slow us down. He was distracting us. distracting us. distracting us. Mm. Mm. Mm. So now, when you're So now, when you're So now, when you're working on something, you're working on something, you're working on something, you're constantly constantly constantly switching between switching between switching between email, email, email, online games, and online games, and online games, and social media social media . And your ability to . And your ability to . And your ability to concentrate concentrate concentrate deteriorates. And, deteriorates. And, deteriorates. And, you know, there's been an increase in you know, there's been an increase in you know, there's been an increase in pornography, which pornography, which pornography, which seems to have affected seems to have affected seems to have affected whether people are building whether people are building whether people are building real relationships. That real relationships. That real relationships. That reducing friction in reducing friction in reducing friction in one area also one area also one area also reduces it in other, reduces it in other, reduces it in other, perhaps less perhaps less perhaps less useful areas. And useful areas. And useful areas. And when you think about when you think about when you think about adding intelligence, adding intelligence, adding intelligence, well, that intelligence well, that intelligence well, that intelligence will be added on the other will be added on the other will be added on the other side as well. And I always side as well. And I always side as well. And I always think that in the early days of think that in the early days of think that in the early days of technology, we technology, we technology, we only see how only see how only see how it will make everything it will make everything it will make everything better. And I think better. And I think better. And I think right now with AI we're right now with AI we're right now with AI we're thinking a lot about how it thinking a lot about how it thinking a lot about how it can make can make can make things much worse, things much worse, things much worse, right? Huge right? Huge right? Huge cyber threats, the collapse of the cyber threats, the collapse of the cyber threats, the collapse of the financial system financial system financial system or the extinction of or the extinction of or the extinction of humanity. But the way

humanity. But the way humanity. But the way it can make things worse in a it can make things worse in a it can make things worse in a trivial way, trivial way, trivial way, simply by increasing simply by increasing simply by increasing people's ability to people's ability to people's ability to waste other people's time, is waste other people's time, is waste other people's time, is a bit underrated in my opinion a bit underrated in my opinion a bit underrated in my opinion . . Certainly. And, you know, Certainly. And, you know, it's not just banal. I it's not just banal. I it's not just banal. I think there are a lot of these think there are a lot of these sub-catastrophic sub-catastrophic losses that are quite losses that are quite losses that are quite serious. Uh, during the serious. Uh, during the serious. Uh, during the Industrial Industrial Industrial Revolution we had Revolution we had Revolution we had several decades of several decades of several decades of terrible working conditions. terrible working conditions. terrible working conditions. I think it's the same with AI: I think it's the same with AI: I think it's the same with AI: while so much attention while so much attention while so much attention is paid to is paid to is paid to job losses job losses job losses , much less , much less , much less attention is paid attention is paid attention is paid to how the to how the to how the quality of work is changing. And I quality of work is changing. And I quality of work is changing. And I think this is a huge think this is a huge think this is a huge underrated area underrated area underrated area because AI is turning because AI is turning because AI is turning many many many knowledge knowledge knowledge workers into workers into workers into managers of AI managers of AI agents. agents. agents. Mhm. Mhm. Mhm. Yes? And the thing is, Yes? And the thing is, Yes? And the thing is, being a manager being a manager is a pretty crappy is a pretty crappy is a pretty crappy experience because you're experience because you're experience because you're responsible for responsible for responsible for other people's mistakes. other people's mistakes. other people's mistakes. You are unable to You are unable to You are unable to practice the craft you practice the craft you practice the craft you studied. But studied. But studied. But you know, as human you know, as human you know, as human managers, it's a sin for us managers, it's a sin for us managers, it's a sin for us to complain. It's a higher to complain. It's a higher to complain. It's a higher salary, a higher salary, a higher salary, a higher status, we status, we status, we chose it ourselves, and of course, chose it ourselves, and of course, chose it ourselves, and of course, mentoring is mentoring is rewarding. With AI, you rewarding. With AI, you don't get any of don't get any of don't get any of these benefits. You these benefits. You these benefits. You have to manage this have to manage this have to manage this agent and agent and agent and be responsible for his be responsible for his be responsible for his mistakes, but you mistakes, but you mistakes, but you can't can't can't practice practice practice your profession. I think your profession. I think your profession. I think we can design we can design we can design AI agents differently AI agents differently AI agents differently to avoid this, to avoid this, to avoid this, but that's but that's but that's exactly what's happening right now. And yes, I exactly what's happening right now. And yes, I exactly what's happening right now. And yes, I think we should think we should think we should be very concerned about be very concerned about be very concerned about this.

this. this. So how much of So how much of So how much of the story we're the story we're the story we're telling here leads to a telling here leads to a telling here leads to a theory about what theory about what theory about what would happen in would happen in would happen in an economy that is very an economy that is very an economy that is very uneven? Those uneven? Those uneven? Those things that require things that require things that require merging with the real merging with the real merging with the real world. world. world. Mhm. Mhm. Mhm. Yes? Everything has to Yes? Everything has to Yes? Everything has to happen physically. We happen physically. We happen physically. We need to build need to build need to build buildings and lay buildings and lay buildings and lay power power power transmission lines. This has such transmission lines. This has such transmission lines. This has such powerful powerful powerful speed limits that it can't speed limits that it can't speed limits that it can't accelerate too accelerate too accelerate too fast. fast. fast. Mhm. Mhm. Mhm. Meanwhile, in the digital Meanwhile, in the digital Meanwhile, in the digital world, things can world, things can world, things can move very, very, move very, very, move very, very, very fast. So, very fast. So, very fast. So, you know, what you know, what you know, what happens to white- happens to white- happens to white- collar workers who collar workers who collar workers who work entirely behind work entirely behind work entirely behind a computer and whose a computer and whose a computer and whose work work work can actually be automated can actually be automated , like in a call , like in a call center or something like that. center or something like that. Or, separately, like all of this Or, separately, like all of this cybercrime that cybercrime that cybercrime that we're talking about and we're talking about and we're talking about and cybersecurity—I cybersecurity—I cybersecurity—I think the version of think the version of think the version of this future this future this future that worries me is that that worries me is that actually most of what would actually most of what would improve improve improve people's lives has to people's lives has to people's lives has to happen in the happen in the happen in the physical real physical real physical real world, but AI will be able to world, but AI will be able to world, but AI will be able to move the fastest move the fastest move the fastest in the digital in the digital in the digital world. And on balance, world. And on balance, world. And on balance, I don't think this I don't think this I don't think this sounds like a world in which sounds like a world in which sounds like a world in which we get we get we get the most benefit the most benefit the most benefit from AI, and in fact it from AI, and in fact it from AI, and in fact it might be a world in which might be a world in which might be a world in which we get the we get the we get the most harm from it most harm from it most harm from it .

. . Me, yes, it's possible. I do Me, yes, it's possible. I do Me, yes, it's possible. I do n't know. I think there are n't know. I think there are n't know. I think there are ways, you know, we ways, you know, we ways, you know, we can, we can can, we can can, we can change that. I don't think change that. I don't think change that. I don't think fast is that fast is that fast is that fast, first of all. fast, first of all. fast, first of all. For example, you mentioned For example, you mentioned For example, you mentioned call centers. I call centers. I call centers. I mean, they still mean, they still mean, they still exist. You know, when exist. You know, when exist. You know, when ChatGPT came out, many ChatGPT came out, many ChatGPT came out, many people predicted people predicted people predicted that within a year we that within a year we that within a year we would replace them all. I would replace them all. I would replace them all. I mean, chatbot is mean, chatbot is mean, chatbot is in the name itself. If, in the name itself. If, in the name itself. If, as is likely, AI as is likely, AI as is likely, AI makes makes makes call center workers much more call center workers much more call center workers much more productive, productive, productive, there is a large there is a large there is a large latent demand. latent demand. latent demand. Often we don't call Often we don't call Often we don't call call centers because it's an call centers because it's an call centers because it's an unpleasant unpleasant unpleasant experience. So, again experience. So, again experience. So, again , this is , this is , this is Jevons' paradox. Would Jevons' paradox. Would Jevons' paradox. Would you mind you mind you mind describing what it is? describing what it is? describing what it is? That's right, yes, it's That's right, yes, it's That's right, yes, it's the idea that when something the idea that when something the idea that when something becomes cheaper to becomes cheaper to becomes cheaper to produce, there's now a produce, there's now a greater demand for it. greater demand for it. Let's look at the sector Let's look at the sector Let's look at the sector where the capabilities are already where the capabilities are already where the capabilities are already the most developed, which is the most developed, which is the most developed, which is probably probably probably software development software development software development . . . Mhm. It Mhm. It Mhm. It used to used to be be extremely expensive to produce software, extremely expensive to produce software, extremely expensive to produce software, so so so only a few only a few only a few tens of thousands of tens of thousands of tens of thousands of lines of code were written worldwide per year, but lines of code were written worldwide per year, but lines of code were written worldwide per year, but now that has increased by now that has increased by now that has increased by about a million about a million about a million times. And so, in the times. And so, in the times. And so, in the long long long term, you know, term, you know, term, you know, whether this will be something that whether this will be something that whether this will be something that will increase the demand for will increase the demand for software engineers, or whether the software engineers, or whether the difficult difficult difficult job prospects for job prospects for job prospects for junior developers junior developers that we're seeing now will that we're seeing now will that we're seeing now will persist—that's yet to be persist—that's yet to be persist—that's yet to be seen. I think it's seen. I think it's seen. I think it's probably both, probably both, probably both, but, you know, in but, you know, in any case, if it any case, if it any case, if it happens over the course of happens over the course of happens over the course of 20 years, again, 20 years, again, 20 years, again, it's consistent with it's consistent with it's consistent with other major other major other major shifts like the shifts like the shifts like the Industrial Revolution Industrial Revolution , where a lot of , where a lot of , where a lot of jobs disappeared, but a jobs disappeared, but a jobs disappeared, but a lot of new ones were lot of new ones were lot of new ones were created. Another

created. Another created. Another example is the work example is the work example is the work of translators. Um, of translators. Um, of translators. Um, you know, back in you know, back in you know, back in 2016, this is of course 2016, this is of course 2016, this is of course long before what we long before what we long before what we call call call generative AI generative AI generative AI today, today, today, translation models became translation models became translation models became pretty close to pretty close to pretty close to human-level. But human-level. But human-level. But these jobs are still these jobs are still these jobs are still pretty stable. pretty stable. pretty stable. The nature of work The nature of work The nature of work has changed a lot. So, again has changed a lot. So, again has changed a lot. So, again , there is a , there is a , there is a much greater much greater much greater demand that has opened up demand that has opened up demand that has opened up because because because you can now you can now you can now translate anything into translate anything into translate anything into any language. Hence, any language. Hence, any language. Hence, Jevons' paradox. Jevons' paradox. Jevons' paradox. I just spoke to I just spoke to I just spoke to Bill Gates, and Bill Gates, and Bill Gates, and when I mentioned it, when I mentioned it, when I mentioned it, he was a little he was a little he was a little skeptical about skeptical about skeptical about it. it. it. So you don't believe in So you don't believe in So you don't believe in Jevons' paradox. Jevons' paradox. Jevons' paradox. Name a Name a Name a working-class profession working-class profession working-class profession that is that is that is subject to subject to subject to Jevons' paradox. Jevons' paradox. Jevons' paradox. So you don't care about the So you don't care about the So you don't care about the working class? working class? working class? I care about the I care about the I care about the working class. working class. working class. So the question is, can So the question is, can So the question is, can you you you name anything in name anything in name anything in the field of working the field of working the field of working professions that professions that professions that falls under falls under falls under this? this? this? And his thesis was And his thesis was And his thesis was that that software development is software development is indeed a sector of the indeed a sector of the indeed a sector of the economy with a large economy with a large economy with a large unmet unmet unmet demand. You know, demand. You know, demand. You know, everyone probably would like to everyone probably would like to everyone probably would like to have their own have their own have their own programmers. So in a programmers. So in a programmers. So in a world where you world where you world where you accelerate this accelerate this accelerate this process rapidly, you can process rapidly, you can process rapidly, you can get a get a get a demand effect that simply demand effect that simply demand effect that simply creates a greater creates a greater creates a greater need for need for need for programmers because programmers because programmers because they are now cheaper.

they are now cheaper. they are now cheaper. But he went on to say But he went on to say But he went on to say that many things don't that many things don't that many things don't work that way at all work that way at all work that way at all . Take, . Take, . Take, for example, a for example, a for example, a truck driver. If we truck driver. If we truck driver. If we get to the point get to the point get to the point where trucks where trucks where trucks are driverless, are driverless, are driverless, firstly, the demand for firstly, the demand for firstly, the demand for transportation transportation transportation is limited, and secondly, there is is limited, and secondly, there is no longer a driver in the truck. If no longer a driver in the truck. If you look at the you look at the you look at the many people whose many people whose many people whose jobs were jobs were jobs were automated or automated or automated or moved to China in moved to China in moved to China in manufacturing, manufacturing, manufacturing, the economy did the economy did the economy did continue to continue to continue to grow, but grow, but grow, but many of those people many of those people many of those people had a very, very had a very, very had a very, very hard time. So his hard time. So his hard time. So his argument is argument is argument is that that that Jevons' paradox won't be Jevons' paradox won't be Jevons' paradox won't be strong enough strong enough strong enough to handle this to handle this to handle this because there are too because there are too because there are too many sectors of the many sectors of the many sectors of the economy where there is no economy where there is no economy where there is no latent demand. latent demand. latent demand. Demand is exactly what Demand is exactly what Demand is exactly what it is. What do you it is. What do you it is. What do you think about this? think about this? think about this? Yes, I think he's Yes, I think he's Yes, I think he's absolutely right absolutely right absolutely right about about about truck drivers. I completely truck drivers. I completely truck drivers. I completely agree with this. The demand agree with this. The demand agree with this. The demand there is relatively there is relatively there is relatively limited. The question is limited. The question is limited. The question is whether this is the rule whether this is the rule or the exception. or the exception. or the exception. Let me Let me Let me put it this way. put it this way. put it this way. Look at what Look at what Look at what we're doing here. Nobody we're doing here. Nobody we're doing here. Nobody asked us for this. asked us for this. asked us for this. You know, if you You know, if you You know, if you went back 100 or went back 100 or went back 100 or 200 years, it would be 200 years, it would be 200 years, it would be like, "How like, "How like, "How can this be a real can this be a real can this be a real job?" Most of job?" Most of job?" Most of the professions we the professions we the professions we have today have today have today are higher up on are higher up on are higher up on Maslow's pyramid, if Maslow's pyramid, if Maslow's pyramid, if you will. They do not you will. They do not you will. They do not satisfy any satisfy any satisfy any real fixed real fixed real fixed demand or need that demand or need that demand or need that people need to people need to people need to live. We do it live. We do it live. We do it because it's fun and because it's fun and because it's fun and people like to people like to people like to listen to it. Most listen to it. Most listen to it. Most office jobs are like that office jobs are like that office jobs are like that . This is my . This is my . This is my point of view. Jevons' paradox is characteristic of point of view. Jevons' paradox is characteristic of point of view. Jevons' paradox is characteristic of most office most office most office professions professions professions .

. . If it becomes easier to If it becomes easier to If it becomes easier to produce it, there will be produce it, there will be produce it, there will be a demand for it, especially as a demand for it, especially as a demand for it, especially as people's incomes people's incomes people's incomes gradually increase gradually increase gradually increase and they and they and they spend more on these, spend more on these, spend more on these, you know, less you know, less you know, less necessary, more " necessary, more " luxury" things. And luxury" things. And luxury" things. And working professions working professions working professions too. There is a great essay by too. There is a great essay by too. There is a great essay by Alex Imas Alex Imas Alex Imas called “What Will Become called “What Will Become called “What Will Become Scarce,” where he Scarce,” where he Scarce,” where he points out that the job of points out that the job of points out that the job of barista at Starbucks, barista at Starbucks, barista at Starbucks, for example, should no longer for example, should no longer for example, should no longer exist. We have exist. We have exist. We have known for a long time how to known for a long time how to known for a long time how to automate this. We automate this. We automate this. We can even can even can even make coffee at home. make coffee at home. make coffee at home. I don't know if that's what I don't know if that's what I don't know if that's what he said, but he said, but he said, but you know, that's the you know, that's the you know, that's the importance of the importance of the importance of the interpersonal interpersonal interpersonal nature of the work. And nature of the work. And nature of the work. And so I think they so I think they so I think they will remain quite will remain quite will remain quite stable, even stable, even stable, even if artificial if artificial if artificial intelligence intelligence intelligence somehow makes them somehow makes them somehow makes them cheaper. So, I cheaper. So, I cheaper. So, I think that the work of think that the work of think that the work of truck drivers truck drivers is more of an exception. is more of an exception. is more of an exception. I think as we get closer I think as we get closer I think as we get closer to the end, what would have to to the end, what would have to to the end, what would have to happen in the happen in the happen in the next couple of years that would make next couple of years that would make next couple of years that would make you say, "Oh, this you say, "Oh, this you say, "Oh, this looks less looks less looks less normal than we normal than we normal than we thought." "Or it's thought." "Or it's thought." "Or it's significantly more significantly more significantly more off off off course than we course than we course than we expected." What expected." What expected." What evidence would have to evidence would have to evidence would have to emerge for you to emerge for you to emerge for you to significantly change your significantly change your significantly change your thesis? thesis?

thesis? Of course. Yes, there are Of course. Yes, there are Of course. Yes, there are economic issues and economic issues and economic issues and security issues. security issues. security issues. Regarding the economy: if Regarding the economy: if Regarding the economy: if we start to see at we start to see at we start to see at some level of some level of some level of capability that this is capability that this is capability that this is no longer a process of simply no longer a process of simply no longer a process of simply adapting people to adapting people to adapting people to use AI use AI use AI to improve their to improve their to improve their productivity and productivity and productivity and manage agents, manage agents, manage agents, as we see now, but as we see now, but as we see now, but instead it instead it instead it starts to massively starts to massively starts to massively replace people—whether they're replace people—whether they're software engineers or software engineers or any any other profession— other profession— I think that would be I think that would be I think that would be completely different from what we're completely different from what we're completely different from what we're mostly mostly mostly predicting. Regarding predicting. Regarding predicting. Regarding security, especially given the security, especially given the companies' claims of being companies' claims of being close to close to close to recursive recursive recursive self-improvement: I self-improvement: I self-improvement: I don't think they don't think they don't think they should should should rush into full rush into full rush into full autonomous recursive autonomous recursive autonomous recursive self-improvement at all, self-improvement at all, self-improvement at all, which is what which is what which is what you seem to have said, but in you seem to have said, but in you seem to have said, but in any case, we any case, we any case, we believe that even believe that even believe that even if it happens, it if it happens, it if it happens, it won't lead to won't lead to won't lead to superintelligence, because the superintelligence, because the superintelligence, because the bottlenecks for bottlenecks for bottlenecks for superintelligence superintelligence superintelligence are external. There's are external. There's are external. There's nothing you can nothing you can nothing you can do in a do in a do in a lab to lab to lab to teach an AI model, teach an AI model, teach an AI model, say, how say, how say, how to cure cancer or to cure cancer or to cure cancer or anything else that anything else that companies hope to do. But again companies hope to do. But again , this is an empirical , this is an empirical , this is an empirical statement, and it would statement, and it would statement, and it would certainly completely certainly completely certainly completely refute our refute our refute our thesis.

thesis. thesis. And finally, our And finally, our And finally, our last question: last question: last question: what three books would you what three books would you what three books would you recommend to recommend to recommend to our audience? our audience? our audience? Of course. You know, in this Of course. You know, in this Of course. You know, in this conversation, I generally conversation, I generally conversation, I generally took a slightly took a slightly took a slightly more optimistic more optimistic more optimistic view of things than view of things than view of things than we're used to hearing, we're used to hearing, we're used to hearing, especially on the especially on the especially on the safety of AI. So, safety of AI. So, safety of AI. So, perhaps in line with perhaps in line with perhaps in line with that, I really that, I really that, I really like like like Anna Ritchie's book, It's Not the End of the Anna Ritchie's book, It's Not the End of the Anna Ritchie's book, It's Not the End of the World. She has a newer World. She has a newer World. She has a newer book, but this one is from book, but this one is from book, but this one is from 2024, and I still really 2024, and I still really 2024, and I still really like it. like it. like it. The subtitle goes something like this The subtitle goes something like this The subtitle goes something like this : how we : how we : how we can be the first can be the first can be the first generation to generation to generation to build a sustainable build a sustainable build a sustainable planet. It's an planet. It's an planet. It's an optimistic view optimistic view optimistic view of the climate, which, of the climate, which, of the climate, which, of course, is usually of course, is usually of course, is usually full of gloomy full of gloomy full of gloomy forecasts, which is why forecasts, which is why forecasts, which is why I liked it. I liked it. I liked it. This is a very optimistic This is a very optimistic This is a very optimistic book about technology. book about technology. book about technology. I-I-I really love I-I-I really love I-I-I really love this book too, and it this book too, and it this book too, and it makes you realize that makes you realize that makes you realize that we really do create we really do create we really do create better things over time. better things over time. better things over time. Yes. Yes. Yes. I think sometimes we have too I think sometimes we have too I think sometimes we have too negative attitudes negative attitudes negative attitudes towards technology. towards technology. towards technology. Yes. Uh, about China, Yes. Uh, about China, Yes. Uh, about China, which is of course a topic which is of course a topic which is of course a topic that that that interests many. I'm sure interests many. I'm sure interests many. I'm sure you've heard this you've heard this you've heard this book recommendation often. book recommendation often. book recommendation often. I liked I liked I liked Dan Wang's book " Dan Wang's book " Breakneck". It also has Breakneck". It also has Breakneck". It also has many echoes of many echoes of many echoes of abundance. But this idea abundance. But this idea abundance. But this idea of ​​thinking about a of ​​thinking about a of ​​thinking about a legal society legal society legal society versus an engineering society is, I versus an engineering society is, I versus an engineering society is, I think, a very think, a very think, a very apt and concise apt and concise apt and concise way to capture way to capture way to capture a lot of the macro and a lot of the macro and a lot of the macro and micro differences. And micro differences. And micro differences. And the last one is an old the last one is an old the last one is an old classic. Let me classic. Let me classic. Let me preface this by saying: preface this by saying: preface this by saying: I once read a I once read a I once read a two-page article two-page article two-page article called "How called "How Complex Systems Fail." I Complex Systems Fail." I thought the article was about thought the article was about software. But software. But I realized that I realized that I realized that it was actually about it was actually about it was actually about medical systems, medical systems, medical systems, written by written by written by an anesthesiologist. And an anesthesiologist. And an anesthesiologist. And I learned that there is a I learned that there is a I learned that there is a study of systems that

study of systems that study of systems that explains explains explains patterns in patterns in patterns in all types of systems: all types of systems: all types of systems: natural, natural, natural, social, and social, and social, and engineering. This engineering. This engineering. This led me to led me to led me to Donella Meadows' book Donella Meadows' book Systems Thinking, Systems Thinking, Systems Thinking, written many written many written many years ago. years ago. years ago. Arvind Narayanan, Arvind Narayanan, Arvind Narayanan, thank you very much. thank you very much. thank you very much. Thank you, Ezra. It was Thank you, Ezra. It was Thank you, Ezra. It was very interesting.