Jack chats with Sebastian Mallaby, senior fellow at the Council on Foreign Relations, about his new book The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence. They discuss current challenges in AI safety, the U.S.-China race and prospects for cooperation, and the emerging risks posed by powerful new models like Anthropic’s Mythos. They also talk about tensions between frontier labs and the U.S. government, and the trajectory toward greater government control.
Mentioned:
Sebastian Mallaby, The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence (2026)
Thumbnail: President Trump delivers remarks at the White House AI Summit in Washington, D.C., Wednesday, July 23, 2025. (Official White House Photo by Joyce N. Boghosian)
This is an edited transcript of an episode of “Executive Functions Chat.” You can listen to the full conversation by following or subscribing to the show on Substack, Apple, Spotify, or wherever you get your podcasts.
Jack Goldsmith: Today I’m chatting with Sebastian Mallaby, who’s a senior fellow at the Council on Foreign Relations and an acclaimed biographer and writer. And we’re going to be talking about his newest book, which is called The Infinity Machine. Sebastian, thanks for talking with me.
Sebastian Mallaby: Thank you, Jack. Nice to be with you.
So tell us what the book is about. Who is Demis Hassabis, and why did you write a book about him?
So the book is about artificial intelligence, and it’s centered on this character, Demis Hassabis, who is, in a way, the OG sort of AI lab leader, right? He starts DeepMind, this startup in London, back in 2010, before AI could even recognize the photograph of a cat—like nothing worked. It was full AI winter.
So this is five years before Sam Altman and Elon Musk start OpenAI. It’s fully 11 years before Anthropic gets started. So he was extremely early.
So if you wanted to tell the story of the making of modern AI through a personality, you know, Demis’s career and intellectual development maps perfectly onto that story.
So the thing that’s most interesting to me about him is that, as you emphasize in the book, his real interest in this, I think it’s fair to say, is scientific and not profit-making. And he, at least at the outset, and I think even today, has a rather idealistic—to me anyway, idealistic or optimistic—conception of the technology and how it can be used.
But the story I also see is someone who—and I don’t mean this uncharitably—but who has basically engaged in a series of compromises or fudges with regard to those values as he’s gotten deeper and deeper into the AI competition.
So is that fair? And can you talk about that arc?
Yes. I mean, he started DeepMind in 2010 with an absolute focus on AI safety. In fact, he met his scientific co-founder, Shane Legg, at a safety lecture in which Shane projected that by 2030 or so, AIs would be sophisticated enough—cleverer than humans—have their own sort of objective functions, and would maybe start to threaten humans.
And this was the lecture over which they bonded. And then in 2014, Demis Hassabis sells his company DeepMind to Google. And part of the sale condition was that AI would not be used for military purposes, that it would be safeguarded by a sort of ethics oversight committee that would be separate from the corporate leadership of Google.
So he took it very seriously. And then this continues for a while. Between 2016 and 2019, he wages a secret battle, a thing called Project Mario, where he’s trying to put pressure on Google’s leadership to have this independent safety oversight board, because Google kind of reneged on the deal at the point of sale in 2014.
And then after 2019, it kind of fades away. And, you know, by now you have Google being willing to provide AI to the national security establishment. In the US, there is no safety and ethics oversight board.
And Demis is left explaining to me, well, you know, I feel as if, you know, if I lean into Google and I’m part of the team there, and I, you know, understand the different pressures that a corporation is under, then I have a seat at the table. And so when push comes to shove, I can chime in in favor of safety. And so I’m a good person—trust me—is kind of the bottom line, which is a sort of flimsy scaffolding of reassurance for an alarmed world.
Especially since—I mean, this was also a time—a lot of this is happening at a time before ChatGPT amazes the world a few years ago with whatever model it was, I can’t remember. And suddenly there’s this massive competition among several frontier models that has been extremely fierce.
And now we’re in a massive competition among those labs and with Chinese firms, and the countries are in fierce competition. And he’s now leading—you talk in the book about how they combined, how Google combined its AI efforts—and he’s leading it.
So he’s really leading, in some sense, this fiercely competitive charge, which isn’t taking—doesn’t appear to be taking—safety all that seriously. Is that fair?
Yeah, it’s fair. And, you know, I think there’s a slight caveat in that his style is to pursue safety ideas secretly. I mean, he doesn’t talk about them.
And Dario Amodei, the leader of Anthropic, is extremely public when he picks a fight with the Pentagon, when he releases this new model called Mythos, where he’s publicly said, you know, this is too dangerous to release generally, so I’m going to release it to a sort of restricted list of people. He likes to be very out there in public with it.
Demis Hassabis, on the other hand, did two important things, to my knowledge, about safety. One was this secret battle I described before, which he was so unkeen to have sort of move into the public sphere that when I discovered it through leaks from other people, you know, I had to talk to his general counsel, who was trying to tell me I wasn’t allowed to publish that. So he really didn’t want that to be public.
And then secondly, he told Rishi Sunak in 2023, after ChatGPT came out, “Mr. Prime Minister, you know, I have an idea for you, which is you could have an international discussion on AI safety—invite the Chinese, invite everybody—start a process that might lead to some kind of understanding internationally on AI safety.”
Demis never told me that he told the Prime Minister that. I only know this because other people, like the Prime Minister’s advisers, told me. So he didn’t advertise what he was doing.
So I think he’s trying to do things now, but they’re not in the public view. So that’s a slight caveat. But basically, you’re right. I mean, he’s leading one of the major labs, Google DeepMind, in frontier AI, racing as fast as he can, even releasing, by the way, open-weight models, which by his own analysis are dangerous because you can’t control them once they’re out there.
And so there is this contradiction—you could call it even hypocrisy—between his stated beliefs about AI safety and what he’s actually doing. And so then the question is, well, how harshly does one judge him? And I’ve just floated the word hypocrisy.
But on the other hand, were he to quit his job and go off and become a professor somewhere and pursue research, which I think is the alternative path for him, it wouldn’t make the world safer, right? There’d still be this race dynamic.
To be clear, I wasn’t judging him. And he seems—I’m trying to understand—he seems like a thoroughly decent, honorable, brilliant guy. I’m just trying to understand the mindset of someone who, from a very young age, had these extraordinary scientific ambitions, which he’s been as important as anyone in making possible.
And—but safety and this kind of benign vision has always been part of it, and it just seems to have been overtaken by reality—mostly competitive, financial, and global competition reality. And I’m just wondering how he processes that. That’s what I’m getting at.
Absolutely. I mean, I was exactly trying to do the same thing—to kind of figure out how you process it and sort of portray that. And, you know, at the end of the book, he tells me, you know, I’m in a paradoxical situation.
On the one hand, Shane Legg and I projected back in 2009, 2010 that by around 2030, AI would be very powerful. And that’s kind of what’s going to happen. And we’ve been central to building it. So, you know, I’ve delivered on this vision in an amazingly gratifying way.
On the other hand, I had this hope that I could control the technology somehow and make it safe, and that hasn’t worked.
And, you know, when you want to ask, you know, why did it turn out so contrary to his expectations? You know, it’s the Oppenheimer syndrome. Oppenheimer led the Manhattan Project, built the amazing technology, and was an incredible scientific leader as well as a scientist, and thought he could sort of go and sell Truman not to use the bomb or to give the technology to the UN or whatever.
Truman just kicks him out of his office and says, “Don’t bring that guy in here again.” So scientists think that they can control their inventions, but often the inventions have their own will.
Okay, you’ve written a lot of interesting essays closer to the topics of this Substack in connection with the publication of the book. And I just want to talk about some of these policy and governance themes that are implicated—that are talked about in the book—but that you’ve talked about, I think, more in connection with the publication of the book.
First of all—and you’re, you know, the keenest of observers of these various relationships and where we are in these AI races—so I just want to get your temperature on, first, what is the state of the relations between the U.S. government and the frontier labs?
I mean, we know about the DOD confrontation with Anthropic and then with Mythos, the government trying to apparently get its hands on that technology, or at least try to reach some kind of accommodation with Anthropic.
How do you see—I want to go through different pieces of the relationships and the competitions at stake here—how do you see the relationship right now between the US government and the frontier labs?
Well, I think it’s just worth saying at the beginning that, you know, the background—if one goes back to 2023, 2024, the two years after ChatGPT came out, it made all this feel urgent—the background was that there was a lot of collaboration.
And if you talk to people, I think, you know, Ben Buchanan, who was at the National Security Council doing AI policy, is on the record about this. He says, you know, whenever I talked to the labs, there was no resistance to the idea of regulation. It was more the opposite—that they were telling me, “Hey, this is serious. This is powerful. This is scary. This is coming. You need to do something.”
And so when he wanted to, you know, set up the AI Safety Institute, which happened in the Biden years or so on, there wasn’t much pushback, contrary to sort of what one might suspect. There wasn’t much pushback from the objects of the regulatory impetus.
Now, the regulation at that point was extremely, you know, emergent. It wasn’t really biting very much. But still, it’s worth noting that the baseline here is collaboration.
Then you move into 2025. President Trump is in office. He essentially doesn’t want to talk about regulation. He’s more keen on competition, acceleration—just make sure the U.S. has the most powerful AI. And so that kind of goes off the agenda.
Now, in 2026, it comes back, both because of this fight over Anthropic’s models and how they are used by the national security establishment, and because of Mythos, the cybersecurity model.
And I’d say that right now you have this sort of, you know, weird and sort of unstable contradiction of impetuses, where on the one hand, you know, the administration has deemed Anthropic to be a supply chain risk, which is an extraordinary name to attach to a U.S. company. I think it’s the first time that’s ever happened.
And so it’s extremely antagonistic. And you had this Slack message that Dario Amodei, the head of Anthropic, wrote, and which got leaked. It was a Slack message to his own staff. And what he said was something like, you know, “The Trump administration wants dictator-level obeisance from me, and I’m not going to give it.” So that shows you the temperature of the relationship there.
But on the other hand, they are talking together about Mythos, and, you know, Amodei went to the White House to speak to people about it a couple of weeks ago.
So I think it’s a mixture of suspicion and sort of a distant relationship, but then at the same time, a need created by Mythos that—wow, you know, we have no choice but to talk to each other, and we’re going to have to do that.
So two questions following up. One is, I could never really tell how serious—I think some more serious than others, maybe Anthropic—how serious the labs were when they said, “Please regulate us.” Sam Altman testified to that: “Please regulate me.”
I don’t believe he really thought that. It’s clearly—but it’s a good thing to say when you’re developing this massively dangerous, consequential technology in the private sector. In case something goes wrong or, you know, it’s always nice to have said we asked to be regulated and you didn’t do it.
I’m just wondering how serious that request is, especially given in the last presidential election, a lot of the big money behind—or at least in connection with—the AI labs, the kind of Silicon Valley big money, went all in for Trump. And, you know, David Sacks gets installed in the White House and implements this kind of libertarian policy, hands-off policy.
So first question: how serious is that request to be regulated?
Well, I think when we talk about Silicon Valley in broad-brush terms, we need to actually break that down a bit. And you’ve got someone like Marc Andreessen, whose commercial interests as a venture capitalist are that he wants to back, you know, upstart challenger AI labs. He’s not going to back, you know, the big hyperscalers because they don’t need his money, right?
So he wants to back essentially startups that are probably going to use open-weight models. And so open-weight, by definition—you cannot be in favor of regulation and be in favor of open weight. You know, open weight is the least regulated type of AI distribution mechanism.
And so, yes, when you’re talking about Marc Andreessen, he’s extremely pro-Trump and extremely deregulatory and laissez-faire. And so there’s no seriousness whatsoever in any comment to the fact that you need more safety if it’s coming out of his mouth. I think, to be fair, it’s not coming out of his mouth.
True, correct.
So then you move to other people who are saying that they want to be regulated. And I think in some cases, someone like Sam Altman probably is straightforwardly, you know, insincere.
But I’d say that, you know, all of these characters who are running the frontier labs have said things at different times, and they’ve both wanted safety. And when Sam Altman created OpenAI in 2015, the rationale for creating it was safety and for the AI to be used for the public good. And who knows? I think he might even have meant it in 2015.
But as time has gone on and the race dynamic has become so white-hot, any instinct he might have had to be sincere about wanting to be slowed down has been overwhelmed by his desire to win the race.
But I think there’s both sides of that. Inside one human being, there could be two personalities. And I think, you know, there’s just a different balance in these different people.
So, you know, probably Sam Altman is the most prone to going for acceleration rather than regulation kind of when a stress point comes. I’d say Dario Amodei is the most prone to be safe when a stress point comes. Demis is somewhere in between. Elon—I don’t think he’s really been tested.
So last question in this vein—it just, how likely is it—let me put it this way—it seems inevitable, given the rapid changes in these technologies and the growing perceived dangers in the technology—Mythos being one example, but there are lots of examples.
And given the stakes of these technologies in private hands in the United States, so given the dangers they’re spewing and the stakes vis-à-vis the competition with China and national security more generally, it seems inevitable to me that—set aside legality for a second—it seems inevitable to me that the United States gets its hands on this.
And I don’t know what the mechanism looks like or what the institution looks like, but that the U.S. government cannot afford not to, in some sense, own this technology, both for its own purposes and for safety purposes. True or false?
Broadly true. I mean, by “own,” you would include, I think, control. I mean, it could be a regulatory model.
Yeah, let me be clear: I don’t mean technically own. I mean control. That’s a better word.
So then I think true, yes. I think, in fact, we’ve just run that experiment in the last few weeks, because you had a deregulatory, pro-accelerationist administration in power. And all of a sudden, an actual case of an AI that threatens stability emerges in the form of Mythos.
And it looks as if it could unravel all kinds of building blocks of the internet and cyberspace. Banks would have their bank accounts emptied, et cetera, et cetera. And they turn on a dime.
And all of a sudden, the Treasury Secretary is saying to the Fed chairman, “We need to call the heads of the banks and tell them to take this seriously. And, you know, we need to get our hands on this.” He’s—you know, Scott Bessent, the Treasury Secretary, reportedly has floated the idea of, you know, sort of—I forget what the term is—but you’ll know—essentially requisitioning or having power over the technology.
And Bessent just said, okay, this is so important in terms of statecraft that we’ve got to mention it, bring it up in the summit with Xi Jinping in the middle of May.
So they’ve flipped. They’ve done a 180, exactly as you predict. So I think you’re correct.
And—okay—and one reason they’re doing that is because of the larger China competition. So talk about the U.S.–China competition in AI, please. Where do you see it today? I know that’s a big question.
Well, maybe—I mean, I think actually that it’s—maybe I want to push back on your premise. I think that the reason they flipped is less about China and more about domestic chaos, right? They don’t want the internet to be hacked and, you know, the banking system to crash. It’s not—that’s not necessarily about China. That’s about domestic security.
Don’t you think it’s about both? I mean, if these technologies are—so fair enough, it’s about domestic security, but it’s also about China being able to defeat government systems, China threatening both to steal and to disrupt things in the cyber realm. I mean, I think technology has an advantage as a national security advantage.
Yes, but I think—okay, we’re going to—I think this conversation is going to go in a direction where we need to clarify one thing right now, which is that there are two kinds of worry about powerful AI in terms like—the big worries, I would say.
One is that—and essentially this is about bad guys getting it and doing bad stuff with it. And there’s two kinds of bad guy, right? There’s China. And in 2022, when the Biden team—before ChatGPT, by the way—saw this coming, they put the semiconductor export controls in place because they didn’t want China to have cutting-edge AI, because in their view, the bad guy to worry about was China.
And there’s a whole second category of bad guys, which is sort of rogue states, terrorists, criminals, et cetera, et cetera. And I think it’s just very important to clarify that there are two kinds of threat.
And I think it’s important to distinguish two ways of dealing with these threats, because in the Cold War analogy, the way to control the danger of nuclear war between the Soviet Union and the United States was mutual destruction and the balance of deterrence.
The way to control nukes from being loose and falling into the hands of terrorists and rogue states was a totally different mechanism, which was the Non-Proliferation Treaty, which wasn’t perfect, but it worked kind of for a while.
So, you know, that’s an important distinction because of, I think, where we’re going to go.
And so, going back to Mythos—Mythos is in the category of both, right? It could be that the Chinese government gets it, but more immediately, it could be just the criminals get it.
And the criminal threat or the terrorist threat is posed not merely by the prospect that China gets this technology and releases it on an open-weight basis, but also that Meta does, right? Or Mistral in France, or Cohere in Canada. There’s a bunch of other labs that could open-weight this stuff, or even just not open-weight—just release it on a proprietary basis, but much more expansively than Anthropic did.
Or have it stolen.
Yeah, right. And there was a hack of Anthropic’s Mythos.
Okay, fair—perfectly fair distinction. I want to come back to it, but can we talk about—you wrote a piece in The New York Times about U.S. policy toward China, chip restrictions, and why you thought that was a bad idea. And then I wanted to get to the nonproliferation treaty idea that you just mentioned.
So can you just tell us why you think that the Biden approach may not have been optimal?
Sure. So at the time I supported it because I thought it had a chance of actually preventing China from getting cutting-edge AI. It turns out it didn’t stop China from getting cutting-edge AI.
And the proponents of semiconductor export controls will say, well, it’s because the controls were not tight enough. But we’ve run this experiment since 2022, so it’s four years now. And we’ve done it under two different administrations.
And in neither case, if you look at the chart of the performance gap between the top model in America and the top model in China, that gap—America’s lead—has shrunk. And it shrunk under Biden, and it shrunk under Trump.
So I’m just skeptical of any regime in which we impose semiconductor export controls or some expanded version thereof that actually works in stopping China from getting powerful AI. They may be like two months behind us, six months behind us, something like that, but that’s not very long.
But just before we go further, I always understood the goal not to stop them but to slow them. And even with workarounds, the United States maintains its edge.
And are you saying that it would be better in maintaining that edge to get rid of the chip controls? That argument, I don’t understand.
Fair. Okay, that’s very fair. So yes, I agree that the controls slow them down. My point is simply that it’s not very much—it’s six months. Does six months really make us feel a lot more secure?
I’m open to—you know, this is a finely balanced argument, which is why I’m excited to have it with you. You could argue that a six-month lead is a lot, and that, you know, if you get Mythos six months before the Chinese do, that’s enough to harden your systems such that when they have it, they can’t use it to any devastating effect.
And also enough to compromise all of their systems.
Yeah, true.
Sorry, keep going.
So I guess my view—my view has been subject to revision—my view has been that a six-month lead over China isn’t enough to feel that’s a big geopolitical win, because, you know, I was thinking, I guess, more of, you know, the conventional integration of the AI into weapon systems, drone swarms, this kind of stuff.
And whether you’re six months ahead of the adversary in that case, you know, they just pick a time to invade Taiwan when they do have—they have caught up.
Now—and, you know, if you think about the Mythos example again—wouldn’t they wait until a moment of near parity? Because these things sort of—you know, you have a jump that the U.S. does, and then the Chinese are fast followers, and then they jump up so that the gap is reduced or minimized. And then they would have a strategic parity that they could use.
So I just observe that it’s not a very big lead. And then you have to ask, what was the cost of the policy? Because if there was zero cost to imposing the semiconductor equipment ban, and there was a marginal gain, then you’d probably take it.
But I contend that there is a cost, in the sense that if we go back to my distinction between two kinds of rogue—China is a rogue, but also terrorists and criminals are rogues—we need to keep our eye on that second category of risk.
And that involves getting China involved, because if they produce open-weight models, which is what they do at the moment, then we know that terrorists will get it.
So the deal is: We give you chips, and you shut down your open-weight models. And what does the form of cooperation look like? Because I’m not so sure that the third-party rogues aren’t going to get the bad stuff, even in the absence of open-weight models. That’s a large assumption.
Yeah.
And I’m skeptical—and I’ll get to this in a second—I’m skeptical that any agreement between the United States and China, about which I’m skeptical, can have powerful effects on third-party countries and private actors.
So what does the cooperation between the United States and China look like? And what is the virtuous story about what the deal is and what cooperation looks like?
I think the best analogy is the Cold War and nuclear nonproliferation. And so in that story, you have both the International Atomic Energy Agency, which kind of keeps track of nuclear material and tries to, you know, by accounting for it, prevent it from being used in secret nuclear weapons.
And then you have later—in fact, 12 years later—the negotiation of the Non-Proliferation Treaty in 1968, which kind of makes compliance with the IAEA mandatory for countries that want to have access to civilian nuclear power.
And I think that’s the sort of model where, in AI, you keep track of big clusters of compute, which can be used to train powerful AI models. And that’s one kind of safety provision, so that you know what’s being trained.
And then secondly, you don’t have open-weight models, which, once they are released, anyone can do anything with them. And that’s just too dangerous. You wouldn’t do that with nuclear material. I don’t think you should do that with AI.
And the deal is that, you know, countries which are going to want what they call sovereign AI—that’s already a bit of a catchword—if you’re France or you’re Germany or, you know, Kenya or whoever, yes, you want to be able to train your domestic AI models on, you know, the Swahili texts or whatever—the Swahili oral tradition—that, you know, you want to make it kind of culturally friendly to your own culture. And that’s all good.
But in return for getting access to models, which you can then post-train in a way that, you know, fits your sovereign AI objectives, you agree to these safety standards: no open weight, you know, we’re going to know where you’re training it, we know what the cluster is, we keep an eye on that so that if you’re trying to make cyber weapons, we’re going to see it.
And so I think it’s something like that.
So, I’m going to state why I’m skeptical and give you the last word on this, and then I have a final question.
So the nuclear Non-Proliferation Treaty—some people think—wasn’t a terribly wonderful success. Not every country joined it. The countries that didn’t join it developed nuclear weapons. It hasn’t prevented some countries from appearing to develop nuclear weapons.
But it’s a relatively simple treaty. It basically says no development of nuclear weapons. I mean, it’s a little more complicated than that, but—and we have a sense of what nuclear weapons are. There’s a problem about pre-development and how do you control that—that’s always been a problem with the treaty.
But let’s imagine that that treaty is a success, despite the adverse evidence. It just seems massively more complicated here because there are so many things you might regulate—chips, data, data centers, know-how, the models themselves.
So you’re going to be regulating all of these things. It’s much, much more difficult to have a verification regime in this context. Dual use is prevalent everywhere. It’s very hard—it seems to me, correct me if I’m wrong—to hide bad uses, bad developments, new models, and the like.
Verification regime in this context, which involves not just government-to-government, but deeply into the private sector of both countries, to private companies of both countries, and so on and so forth. Enforcement seems to me—not clear how that’s going to work.
Let’s just imagine that it works bilaterally between China—that we somehow work this out where there’s this virtuous regime where we’re both doing safety and none of us—or none of them—are doing bad uses or acceptable bad uses within the competition.
But I don’t understand, you talk about all the dangers from these third parties, both private and public actors, how this regime is going to handcuff them and all the verification and enforcement problems there, and then on top of that, there’s a massive need for speed.
This stuff is getting very, very dangerous very quickly. And it’s inconceivable that such a treaty or such an international organization would happen in the foreseeable future.
So that’s why I’m skeptical, in a nutshell.
That’s quite a long list.
You don’t have to go through the whole list, but is there a reason not to be skeptical?
Well, I think the first reason not to be too skeptical is that it’s kind of massively defeatist in the face of a serious threat, right? You know, the Mythos vision, where you have random criminals who can steal everybody’s money from their bank accounts, is not a good one.
And so I think trying to lean into the possibility of some sort of regulation that avoids that is important. And if you’re going to do that, it’s got to be international, because obviously you can commit those crimes from some offshore base on a Chinese model. And so if the Chinese have been excluded from any such deal, it’s pointless.
So I think, you know, “don’t give up” is the first message.
And then secondly, I think there are more kind of choke points on which the US, in particular, can get a grip. And if it had China with it, it could definitely get a grip.
And the choke points essentially are the making of the chips and the operation of the chips in clusters. I mean, at the moment, overwhelmingly, the majority of compute clusters are in the United States, and the ones which are not in the United States are using U.S. chips and U.S. equipment—or the equipment of, you know—even in the case of ASML, the Dutch company that builds the lithography machines to engrave the chips—that’s a Dutch company, but they have operations in the United States. They are fully going to comply with U.S. coercion.
So I think essentially the whole of the Western camp—and you and I have talked about this in the past in the context of other things, like the regulation of the internet—all of the Western AI, including—and I would include the Middle East and stuff like that—have touch points in America, which they care about, like the customers they want to serve, the way they’re going to raise money, the chips they’re going to use, other technology around the chips, the cooling material and all that.
If you brought the full might of the United States down on them, they don’t have an independent ecosystem, and they couldn’t do anything.
So I believe the U.S. has a lot of control. The only place it doesn’t control is China. That’s the only credible technological possibility of a totally separate ecosystem that rises up and, you know, provides an alternative source of AI compute power.
So if you had two countries on board—China and the US—I believe that some form of nonproliferation is possible.
Of course, I accept that in the past, nonproliferation for nuclear stuff was not perfect, and there would be people who didn’t sign and all that stuff—but better than nothing.
Fair. Okay, that’s a fair answer. Let me ask you one last question. And I haven’t heard you speak about this, so I don’t know if you have views on this.
At least in the United States, a lot of people are—we’ve been focused on—our earlier conversation was about the danger of these technologies being in private hands and the need, perhaps, since the federal government is the guarantor of the public interest and the supplier of public goods that we can’t rely on the private sector for, there’s a natural assumption that the government should perhaps, at least for dangerous uses—as for negative externalities and the like—have some access to and control over these technologies. That’s the basic argument.
But there’s another worry, which is the worry of the government possessing these technologies and the bad things the government could do with these technologies. And there’s an endless list. You know, it supercharges surveillance. One can imagine perfect control.
You know, we’re talking a lot in this country about the unitary executive. Well, you can imagine everyone being plugged into the presidential AI and taking immediate directions, and you can also imagine that taking nefarious uses.
There’s a real worry about government having these things. And so that’s just another worry in this whole horrible calculus of trade-offs that one has to think about. And I wonder if you have any thoughts about that.
Yeah, I mean, probably fewer thoughts than you do. But I would say that, first of all, I agree that AI is a centralizing technology. So the internet was a decentralizing technology in the sense that it became easier for less powerful actors with fewer resources to have as much information as the powerful ones.
And so the state, which might have had a good monopoly on some forms of important data, suddenly those data were democratized, and lots of people in the private sector could get them. You know, satellite surveillance of what’s going on on the ground, you know, was—I mean, it’s maybe not just the internet—but you have companies like Planet, which provide excellent images of what’s going on on the ground. That’s a private company, a startup in Northern California, which makes its data available to lots of customers.
And so you get the kind of democratization, privatization, diffusion of knowledge.
But with AI, you need to have as much data as you can. And then when you train on all that data, you sort of have this ability to centrally control, centrally understand things. And if the AI is held closely and not distributed, that’s an extremely centralizing phenomenon.
So, I mean, at the moment, you know, the way AI is being built is sort of in the hands of, you know, 10 or something companies. But they release it fairly openly, and so it’s pretty democratic.
But were that to change in the future—as we’ve been discussing—in a more closed way, it’s maybe government basically controlling domestic producers to the point where they could demand that they get one of the most powerful models only for the government or something like that.
Yes, that’s very centralizing and very disturbing in terms of civil liberties. And indeed, we’ve seen that argument come up explicitly in the fight between Anthropic and the Pentagon—not the Mythos one, the one before—where Dario Amodei made two objections. One was about autonomous lethal weapons. The other one was about the use of his AI for domestic mass surveillance.
And his point specifically was, you know, the law has not caught up to deal with the prospect of this very powerful AI in the hands of the government. And until it catches up, I don’t want my system to be used for this, because democracy lags behind where the tech is.
So I do think there’s a sort of whole set of concerns there, which you’re correct to raise. I’m not sure I’m pushing the argument beyond what you said in your question.
No, that’s great. I say I share his concern. The law has not caught up to the possible uses. And I think we’re going to be learning that over the next months and years.
Sebastian, thank you very much. It’s truly an extraordinary book. It’s not just about one person—it’s really a history of artificial intelligence and an exploration of all sorts of related issues. It’s a touching personal story, and it’s told accessibly and vividly and brilliantly, as usual. Thanks very much.
Thank you, Jack. It’s been a great pleasure.
