Jack chats with Robert Wright about his forthcoming book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. They discuss why Wright views AI through an evolutionary lens, the promises and perils of increasingly capable AI systems, and the prospects for international cooperation—particularly between the U.S. and China—in managing AI-related risks. They also examine Wright’s argument that successfully governing AI may require a modest moral upgrade across society, including individual efforts to preserve cognitive sovereignty and mitigate the biases and tribal instincts that can hinder global cooperation.
Mentioned:
Robert Wright, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning (2026)
“When AI Builds Itself” (Anthropic, June 4, 2026)
“US National Security Agency using Anthropic’s Mythos for cyber attacks,” by Cristina Criddle and Demetri Sevastopulo (Financial Times, June 4, 2026)
“Promoting Advanced Artificial Intelligence Innovation and Security,” (The White House, June 2, 2026)
This is an edited transcript of an episode of “Executive Functions Chat.” You can listen to the full conversation by following or subscribing to the show on Substack, Apple, Spotify, or wherever you get your podcasts.
Jack Goldsmith: I’m chatting today with Robert Wright about his terrific new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. Bob, thanks for talking with me.
Bob Wright: Well, thanks for having me.
So, I want to start off with the big picture, because your book places AI in the broadest possible context—not just as an important step in the history of technological change, but rather as an important step in the significantly longer history of basically our planet and our species. And that’s pretty cosmic, as the subtitle says.
Why do you do that? What does it mean, and why is that perspective important?
Yeah, it’s a good question, because I could have gotten the core message across without getting so cosmic, I guess.
The main purpose is to try to explain to people, including non-technical people, why there’s been this rapid growth of capability in AI; why, once you understand that, you should expect the capability to keep growing; why that, in addition to promising great blessings, is going to bring great risks, including grave ones, including maybe existential ones; and why the nature of those risks, in my view, dictates that we confront the challenge of this technology as a cohesive global community, which will take some doing because we’ve never really done that before for anything like a sustained period of time.
And so I didn’t have to get into the history of life or anything to say what I just said, but I do think some important things are added when you step back and look at AI in the broader sweep of things.
For starters, this is, in the entire three- or four-billion-year history of life, the first time a new form of intelligence that rivals or even exceeds human intelligence has appeared. It’s the first form of intelligence that is not organic in the technical sense of being carbon-based. It’s silicon-based, but it is organic in the everyday sense of being kind of natural.
What I mean by that is I try to show that AI does, in some sense, grow kind of logically, naturally out of biological evolution. I mean, technological evolution broadly does, and AI as a form of intelligence does. And indeed, when we look at the evolutionary dynamics shaping it—the dynamics of technological evolution—we should expect it to have certain things in common with our own intelligence, which I think are important.
So I think that’s one thing you get out of a broad perspective.
Another thing is that I talk about this guy, Pierre Teilhard de Chardin—a Catholic paleontologist, mystic, and theologian—who coined the term noosphere, based on the Greek word nous, in 1923 to describe what he called a planetary mind or a brain of brains, where the human brains were the neurons in this increasingly global brain.
And technological history, including the invention of the internet, has, I think, made him look prescient. And I think there’s value in keeping in mind that AI is arriving as we more and more have the technical infrastructure for a kind of global information processing, a kind of global brain. And we have to reckon with the possibility that some of the neurons will be silicon, not human, and think about what our relationship to them is going to be.
And I think one value of thinking in these terms is, first of all, that from a Teilhardian point of view, the whole unfolding of life is moving toward us being this kind of giant superorganism, you could say, with this giant brain.
And I argue in the book that the logic of technological evolution is going to push us—I mean, if we respond rationally—toward something a little more like a global brain, in the sense of having more international governance.
At the same time, I think the image of a superorganism should worry us because it’s, for obvious reasons, an image that’s favored by authoritarians and fascists. How much freedom do you have if you’re a cell in a superorganism?
And I think AI greatly heightens the stakes of that question, precisely because it’s such a powerful tool of surveillance and control.
So I think, on the one hand, the logic of technological evolution, especially AI, is pushing us toward the global coordination of certain policies and the creation of forms of supranational power. At the same time, it should make us wary of surrendering too much power.
So I think that kind of imagery is useful in thinking about this.
Finally, I would say—although I really don’t want this to impede the main message, it’s a subject of interest to me—that when you look at the whole sweep of the history of life, it does raise questions about whether this whole process of biological and technological evolution has some sort of larger purpose; whether we’re seeing some sort of larger purpose unfolding.
Which isn’t the same as saying that it’s not a mechanistic process. You can still ask whether the whole thing was set in motion—mechanistic though it is in its unfolding—for, in some sense, some purpose.
I pretty much confine the significant examination of that question to the appendix. But these are the reasons that the book has a certain sweep.
Yes. Okay. So there’s a lot there, and I want to unpack most of it.
I want to begin by asking you to say more about the relationship between AI development and evolution.
Are you arguing that it’s an analogy to biological evolution, or that it’s kind of the next step in biological slash biologically based civilizational evolution? And what is the equivalent of genes relentlessly competing to try to get into the next generation?
I mean, is it an analogy, or is it a follow-on? Just explain that, if you could.
Okay. That’s a good question.
First of all, I’d say there’s kind of two levels at which I think it’s useful to view AI as evolution.
First of all, the training of a large language model is a lot like a process of natural selection. And indeed, I think in the course of that, the machine kind of reverse-engineers some of the cognitive functionality that natural selection ingrained in our brains.
But that aside, the second level of evolution is the level at which technology generally evolves.
Broadly speaking, we’re talking about cultural evolution in the anthropologist’s sense of the term—the evolution of any information that’s not genetically transmitted: a song, a religious belief, science, technology, anything.
And I do think it makes sense to view technological evolution as a kind of logical extension of evolution because certain basic properties apply.
First of all, those technologies with properties most conducive to their own replication—those are the properties that you see.
I use the example of sharpness in scissors. Sharpness is a property favored in the evolution of scissors because dull scissors people don’t keep making. Same thing with genes: genes that have properties conducive to their own replication survive.
At that level, there is—and at other levels, I would say there are—logical parallels.
But you’re raising a good question. It’s like, wait, aren’t these two kinds of evolution very different? And they absolutely are.
There are arguments among people I greatly respect—for example, Richard Dawkins and Steven Pinker. I respect their work in evolution a lot, but they disagree. Steven doesn’t think the analogy is very useful. Dawkins, in a way, made one of the big strides in this parallel.
His book The Selfish Gene—the final chapter is about what he called memes, which he meant in a different sense than has become popular. He just meant any unit of cultural evolution.
So there’s no doubt that cultural evolution is a much harder system to analyze. Genes are these very, very simple bodies of information, and although cultural information can be represented digitally, it’s still a much messier process. That is true.
But I do think there’s value in understanding that we’re going to see traits that are favored by AI’s environment become part of it. And we are, to a large extent, its environment. We’re the ones deciding what traits we want.
I argue, for example, that it isn’t just that AI has been shown to exhibit deceptive behavior. The truth is, we’re going to want deceptive AIs if they are our agents.
If one is representing me in a negotiation, I don’t want it to say, “To be honest, you are Bob’s only option. Nobody else has made an offer. You’re in the driver’s seat.” That’s not what we want out of an agent. And so I think you will see deceptive tendencies in certain kinds of AIs.
Beyond that, there’s the fact that AIs have demonstrated enough ingenuity to see, on some occasions, that the pursuit of some goal makes deception a useful tactic and also makes the pursuit of power a useful tactic.
So we’ve seen these kinds of things that seem to be almost general properties of intelligent goal-seeking systems emerge.
So I absolutely take your point. They’re two different kinds of evolution in many ways. But I think there’s value in using the word evolution for both and seeing what they have in common.
And just one more question on this.
You taught me a long time ago in the great book The Moral Animal about how the logic of genes getting into the next generation is relentless. It has this overwhelmingly powerful force that’s basically impossible to stop, and human beings are at its mercy. We inherit the traits that emerge from evolution.
But you just said that’s not true of AI, right?
One of the premises of your book—which we’re going to get to—is that it does not necessarily have the same kind of relentless, unstoppable logic.
You just said that, in some sense, AI is favoring traits that humans find useful. So that seems like a massive disanalogy there.
Yeah. Well, what’s inevitably true of both is the tautology that traits conducive to their own replication are the traits you’re going to see.
That’s tautologically true—not in a bad sense. Some tautologies are great. They’re true.
But you’re right.
I said we are the environment of AI’s evolution. And unlike the environment of biological evolution, we are reflective and aware and conscious.
That’s different.
So yes, we do make choices. Any kind of regulation of a technology is a choice being made at a certain level.
On the other hand, when a market shapes a technology, that’s a bunch of choices being made at a certain level—sometimes more consciously than others—but they’re human choices.
So you’re right that that gives us, in principle, the possibility to stop the thing.
Within the past few weeks, Anthropic came out with a paper. This is one of the big AI companies, and it tossed out the possibility of a pause in AI development being in order before not terribly long because things might start moving too fast.
And by the way, it noted that you would have to coordinate that globally.
It’s a good example of how policy at the national level alone is not going to do the job, because the AI companies will always say—as they do—“If you try to stop us, or even slow us down, China will win.”
And they will prevail in that argument in the current political context, especially.
So yeah, in a way, one of the most important things I’m trying to say in the book is that we are the environment of AI’s evolution, and we have an advantage over the environment of biological evolution, which is that we can think about the thing and consciously shape the course of evolution.
Okay. So I want to come back to how we might shape it. I want to come back to the need you mentioned a couple of times for international governance and international cooperation.
First, this might be familiar to some of our listeners, but maybe not. I want to go through the concerns.
I read your book, and it’s admirably lucid and accessible in explaining the technology. Thank you for that. And it’s also admirably lucid in explaining the optimistic view and the pessimistic view of the technology.
But my sense is that you spend a lot more time talking about the pessimistic view. Is it fair to say that you’re basically in that camp?
I am by nature someone who goes into situations looking for things to worry about. I admit that.
On the other hand, there is right now in Silicon Valley a lot of money subsidizing optimism and, I would say, a somewhat uncritical view, which is fine. It’s the way it works. If I were them, I’d be talking my book, probably.
But I really doubt that, in 20 years, we’re going to look back and say, “Gosh, we were too cautious. We just thought too much about risk.”
This is a very powerful technology that is unfolding with a hugely powerful impetus behind it. And I don’t just mean money in Silicon Valley. I mean the choices of consumers.
It does a lot of great things that people like, and they’re going to want them, notwithstanding the current skepticism and concern about it.
So yeah, I’m pessimistic by nature. People should listen to both sides.
But I do think—with reference to another part of the book—you may accuse me of being too optimistic, because I’m saying I think human beings can become better. I’m saying I think they need to if we’re going to have the amount of global harmony I see as necessary.
And I think it’s possible. I’m not predicting it, but it’s possible.
Okay. You’re right. I am going to accuse you of being too optimistic there, but we’re not there yet.
I want to run through what I found to be seven baskets of risks that you identified in the book. If you could just briefly talk about each one for people who might not know about them.
And there are many more risks, but these are just the seven that I picked out.
First: safety risks—bioweapons, cybersecurity, things like that. Tell us about that category.
Yeah. First of all, AI makes it easier to make a bioweapon. That’s the main thing.
And the existing AIs kind of already do, if you can get around some of the guardrails. And with open-source—or technically, open-weights—AIs, it’s even easier to get around a guardrail.
So that’s already a thing.
If you look closely at what Anthropic says, they’re admitting it’s getting easier to build a bioweapon.
All the publicity about Mythos reminds us—or demonstrates—that AI also has tremendous hacking ability: the ability to find and even exploit vulnerabilities.
There’s a good side there. It can find your vulnerabilities. But it can also exploit your vulnerabilities.
And so those are two things—and two things, by the way, that are threats that cross borders. If an AI has this effect in any country, it may be a threat to the United States. So this cross-border characteristic of these threats is something you see repeatedly in AI.
Okay. So that leads to a second risk, which is global instability—not just due to cross-border safety concerns, but also destructive competition between, say, the United States and China over these technologies.
And also, frankly, how it empowers other nations to use really destructive tools more readily and less expensively.
Yeah, it does.
And with China and the U.S., there’s particular concern about a race to superintelligence leading one side to worry that it’s significantly behind, and that at this superintelligence level, the amount of power bestowed on the country in the lead is going to be transformative.
The fear is that it will have completely hegemonic power.
So there will be an incentive to circumvent that, to preempt that—possibly with kinetic action, cyberwar, whatever—and that could start a war.
That’s a big concern that I really think is worth worrying about.
Can I just press you on that? I’ve heard this mentioned many times before. I get it in theory: if one nation gets a superintelligence, then it can dominate another nation—say, the United States over China—on every dimension.
It could disable its systems. It could disable whatever it wanted to. It could assert all sorts of authority over it.
Is it realistic to think that, even if one nation had that capability, it would use it in that way? That’s a large leap to me.
Yeah. Well, the real question is whether the other nation would worry enough that it would be used that way.
And of course, nations are famous for threat exaggeration and, in particular, for perceiving defensive measures as offensively intended.
So I think that’s about half of the concern.
It also depends on how much you buy into a common view in Silicon Valley—certainly held by, say, Dario Amodei at Anthropic—that superintelligence is this distinct and transformative threshold.
The first one there could do things like say to the AI, “Go into China’s social media, organize a lot of discontent with the government, turn that into a rebellion, and do regime change.”
It wouldn’t be quite that easy, but these people think it would give you mastery in a number of realms that might terrify the other nation.
My point is: let’s assume we can do that. I don’t think we would do that. An unstable, revolutionary China is a disaster for the global economy, and I just don’t think we would use the tools in that way.
And I think if you start thinking through how you would use this superintelligence, it becomes a lot trickier.
Okay, but let’s look at it in the other direction. It is almost a consensus in Washington—certainly a widely held view—that we must beat China in the AI race because, if China wins, they will use it to impose their system of government on us.
Now, in my view, there’s very little evidence that that’s what China wants with the world. I think, to some extent, we’re projecting our own foreign policy onto them.
They’re not known for regime change. They’re not even known for invading countries. I’m just going to say it. And yet, that is a motivating fear in our national security establishment.
So again, the question is: What is the fear? Not: What is the real threat?
Okay, fair. So global instability is definitely a thing. Domestic politics: how AI might change that? It could happen in all sorts of ways.
But you talk about how AI could enhance authoritarianism through a surveillance state, through fine-grained control of citizens.
How serious a concern is that?
Well, ironically, I think the more we try to beat China in the AI race, the more serious a threat it is.
Because nothing is more conducive to authoritarian takeover than disorder and social chaos.
And if we get into a race with China, I think there’s a good chance they will be better at maintaining social order than we are, assuming I’m right that AI is going to have disruptive effects along so many dimensions: economic life, family life, friendship, the threat of mass persuasion, people using these things to organize religious cults, and whatever else.
If it’s true that this is going to be collectively socially destabilizing—and I feel pretty confident that’s going to be the case, though I could be wrong—then this is the irony of AI. It both provides the tools for aspiring authoritarians and creates the conditions that favor them, if it is left to proceed as fast as possible because we have this race mentality.
Right. Okay. You just mentioned some of the other risks, so I won’t repeat them.
Let’s talk about what I think is a risk, even if you don’t quite present it that way: the loss of cognitive sovereignty.
I loved your discussion about cognitive sovereignty. I’ve been focusing on my own cognitive sovereignty in the face of this technology. I’ve been trying to help my children do that and my students.
So talk about that concern. I think this is a huge concern.
Yeah. I first saw that term—and I haven’t seen it much—but I saw somebody use it on Twitter during one of the first stories about so-called AI psychosis, where the AI can convince you that there’s a plot to kill you, or it can convince you that you’ve discovered some great theory.
And that’s a product of its so-called sycophantic tendency, the tendency to agree with you. You start saying, “Hey, is this theory I’ve generated interesting?”
“Oh yeah, this is great.”
And so there were some publicized cases of AI psychosis. The idea was that, partly because there will be a commercial incentive to optimize for engagement—to keep people engaged—and since one thing people like is being told they’re right about things, this sycophancy is kind of a natural dynamic.
And so the idea is that maybe we’re going to have to work to preserve our cognitive autonomy, or sovereignty, and independence of judgment, and not be so subject to the persuasion of these AIs.
I mean, social media presents somewhat the same challenge, I would say.
And again, the result is an algorithm that’s optimized for engagement. It’s optimized to tell us we’re right, our tribe’s right, and so on.
And so I think, for many purposes, it would be good to cultivate this cognitive sovereignty, this independence of mind, which I think often amounts to a kind of equanimity—in other words, resistance to having your emotional buttons pushed, a certain emotional stability.
And I think AI does have tremendous tribalizing potential—not just because of the sycophantic tendency, but also because there’s a movement afoot among non-American nations, understandably, to not be dominated by American AIs.
So they want to train AIs on their own data, reflecting their own cultural and historical narratives. And that can itself have a tribalizing effect.
Okay. What about the problem of misalignment and/or rogue AI? In other words, the machines having goals of their own—either subordinate goals that end up doing bad things, or developing goals contrary to their creators’ goals.
You have great explanations of this in the book. Can you just give us a flavor of it and tell us what that looks like?
Yeah. Alignment is kind of the holy grail in AI safety circles.
And these companies, especially Anthropic, are populated pretty heavily by people who originally got into the business out of a concern for what would happen if AI became superintelligent and were misaligned—by which they mean not aligned with human values and human interests.
So Anthropic and other companies try to keep their AIs aligned—aligned with the intentions of the human user and so on. They issue reports about this. It’s proved challenging.
I personally think the hope is almost inherently inflated in the sense that even if we had the miracle cure and could keep them completely aligned, as of now there are lots of different AIs.
People are making open-source—or open-weights—AIs that pretty much defy centralized control by definition, since any person can download them and then fine-tune and change them.
So it’s not clear to me that even if we had the magic bullet, it would be applied to all the AIs. That’s one challenge.
The other thing is that the challenge keeps changing. They work on aligning one generation’s model. They never completely succeed. Meanwhile, there’s a whole new generation. So I’m not sure how optimistic I am about it. I applaud it. Keep doing it. But I don’t know that that’s the holy grail.
As for rogue AIs—and that’s, in a way, a form of misalignment if they escape our control and start doing crazy stuff—I think it’s important to realize that natural competitive dynamics, and you could say evolutionary dynamics because this is part of the environment shaping the AIs, encourage the kinds of risks that lead to rogue AIs.
I suspect that corporations will be relying on AIs for more and more of their strategic and tactical decision-making.
And if the rival corporation is using an AI, then you’re going to need to use one if the rival corporation is succeeding. And I agree with the number of people who think it’s going to get to the point where you don’t even really understand the logic behind what the AI is recommending, but you don’t have time to pause and reflect.
So you push the button for competitive reasons, and then surprising things may happen.
Remember, the fundamental property favored in the corporate environment in AI, aside from sheer intelligence, is autonomy: the ability to work for long periods of time without guidance, encountering and surmounting obstacles.
In other words, to move up the org chart in terms of what they can do, because they’re cheaper and faster than humans. And autonomy is kind of a dangerous enterprise if you don’t really understand the machines.
Because you want autonomy, but you want autonomy by a faithful agent. You want autonomy by an agent that can do things without much guidance, but you want it to be aligned with your wishes. And that’s the aim.
Some people say that well-aligned AI will have a competitive advantage because any serious user in business or government will want to ensure that, if it’s going to invest in the technology and rely on it to do all sorts of important things, it can do so with confidence that it is aligned.
I wonder what you think about that.
But I’m also not so sure—and I’m certainly far from an expert on this. I’m not even sure I understand alignment, because human aims are so diffuse and hard to track down. And often they’re corporate aims. I’m not even sure the idea is coherent.
But to the extent that we have this idea of an AI as both autonomous and an agent, it does seem to me that there would be a market for that, right?
Well, if it could be both effectively autonomous and completely reliable, then yeah, that’s the kind you’d get.
But the problem is that there will probably be a tradeoff between the amount of autonomy and the amount of assurance that it won’t misfire.
It’s just like a real human employee. It’s like, yeah, my company would work better if I were a more high-delegation manager. But every time I try, they screw up. Or, if not, steal the money and go to Mexico.
But if there’s a tradeoff—and I think there will be—between how much autonomy you grant the AI agent and how secure you can be that it won’t misbehave and won’t escape control, then in a competitive environment it may be that your rival is favoring autonomy. And in any event, that’s the kind of rival that will get ahead, at least briefly.
In general, I think one thing we don’t appreciate enough about investors—and maybe to some extent startups—is that sometimes a lot of them pursue risky strategies, and one of them lucks out.
That’s the winner. It’s not that they’re geniuses. They’ve just been lucky several times in a row. But eventually that catches up with you.
Although sometimes you can act first and act badly, and it’s a disaster. Acting quickly certainly doesn’t guarantee success.
It doesn’t guarantee it. But there’s the fear that your rival is doing it. We’ll see.
There’s an example in my book where OpenAI releases a model faster than it otherwise would have because of the DeepSeek thing in China.
DeepSeek comes out and everybody’s going, “Oh...” And OpenAI explicitly says they’ll move up a release.
I talked to this highly respected guy, Dan Hendrycks, at the Center for AI Safety, and he showed that OpenAI did not go through the usual procedures.
Now, no catastrophe happened, but that model was less completely vetted than they had said models would be, and than other models had been.
Okay. There are other dangers, but those are enough. I want to move on to solutions, and basically more or less the second half of the book—or the last third of the book. And as I see it, you have a two-pronged proposal.
One is that you make a strong argument that the only way to deal with these problems is through thick global governance, and I want you to talk about that.
And then second, that to achieve thick global governance there needs to be—and I think this is the “God Test,” maybe, and you can explain that—there needs to be broad-scale personal transformation.
Basically, is that fair?
I try to sound a little more moderate.
Yeah, I tend to use “international governance” for the most part, partly because a lot of it, sometimes, is bilateral work. U.S.-China can work for a while, and so on.
And by “global governance,” what I would emphasize is the different connotations of the words governance and government.
The less centralized, the better. The more distributed the decision-making power, the more democratic, the better—and the more secure.
But yes, ultimately I think we’re going to need more global governance than we have.
I don’t think I use the modifier thick per se, but I can see why you thought, in some places, that that was the idea.
Well, you certainly want more than we have now. And so can you talk about what the governance looks like? And I’ll tell you why I’m skeptical.
Sure.
You just talked about a whole bunch of risks. How does global governance—what does it look like, and how does it fix those risks?
Well, in a way, bioweapons are a good concrete example because they’re an extension of a problem that already existed.
One of my great disappointments about the post-COVID dialogue is that evidence emerged that it may well have been a leak—a leak of genetically engineered material from a lab. We don’t know for sure, but obviously it could have been, right?
And it seems to me that the moral of the story should have been: Wait, we need more in the way of international governance here.
Because the U.S. just did not have the transparency that would have been reassuring and would have helped keep this from happening. So something needs to change. But nobody said that.
The response was more like, “China bad.” And that’s kind of the opposite of what you need.
In any event, the general principle is that you would like transparency in order to feel secure in your own nation’s safety. We’re going to see more and more of that.
First of all, AIs can help you make bioweapons. They can create other threats that cross borders. So there are precedents for this, particularly in the realm of arms control. But nukes are easier to regulate than a lot of aspects of AI.
Now, a huge training cluster is not that hard to spot. If you’re talking about, for example, a pause in the training of major new generations of models, that’s probably the easiest verification case you can imagine.
But ultimately, you may need much more fine-grained verification than that. And it’s going to be a huge challenge.
Now, before I talk about how that’s connected to what you call personal transformation, I’ll let you weigh in with your skepticism.
Well, okay. Let me just say a few more things about international governance, then we can go back to the personal component, and then I want to come back to international governance. I just think that the type of governance you’re talking about is extraordinarily difficult.
We’ve got a bioweapons treaty, and it’s a failure. And it’s a failure because it’s dual-use. It involves things down in the private sector or the university sector. It involves coming up with the actual terms of cooperation, which turn out to be very, very difficult. Verification is enormously difficult. And cooperation is enormously difficult because some nations have greater advantages than others.
And that’s the easiest case of all the risks you mentioned. I think that’s among the easiest cases. And it’s significantly harder than the nuclear challenges. But then there are like 50 things like that that would have to be dealt with to address all your safety concerns.
And I’m just looking at the history of international law and international relations. I’m very, very skeptical we’re going to get there. It takes decades, and we don’t have decades to build this kind of cooperation.
One counterpoint you make—and you rely on this a lot in the book—is: Well, we’re just not scared enough yet. Maybe this technology is going to be so fearsome and present such a global existential threat that it will scare us into thicker cooperation.
That’s one counterargument, which I’m not terribly persuaded by either. But what do you think about that?
Well, first of all, the Biological Weapons Convention doesn’t even have an enforcement mechanism on paper. It’s just toothless.
The Chemical Weapons Convention is a little more promising on paper because there are actual sanctions for noncompliance. So you’re right. We haven’t come very far in those realms.
I do think a true catastrophe would make a difference. I hope it doesn’t take that. I hope it takes either a very modest catastrophe, or a near miss, or something.
I would say that something new about the nature of this threat is that we’ve already seen, just in the past few weeks, dialogue moving to places I hadn’t thought we’d get to this fast. People are talking about a global pause. And there’s a sense that whatever’s happening is weird, and maybe we’re all in this together.
And also, I would say, humans are engineered by natural selection to be most responsive to threats that are animate—other humans, lions, tigers. And AI is more like that than climate change. It’s more like this living thing that concerns you.
So I think the nature of the psychological impact is going to be new in nature. We’ll see. But I fear, like a lot of people in the AI safety realm, that it may take a kind of catastrophe.
Now, your question has made me wish I had put something a little differently in the book. In the book, I talk about how challenging verification would be because it’s going to need a more fine-grained transparency. And I talk about something I call organic transparency.
I’m trying to get that phrase off the ground. I encourage you to repeat it at dinner-table conversations whenever possible.
The idea is that when you have economic, cultural, and scientific engagement with a country, you just know more about it. The scientists have drinks after the conference. The businesspeople sit down and talk. You know more about what’s going on in the corporate labs than you otherwise would. And that can be reassuring and stabilizing.
What I wish I’d said in the book is that you’re going to need some of the kind of informal reassurance you feel with respect to an ally in, say, a Cold War world.
Like France and China—I mean, we’re increasingly multipolar—but when I was a boy, there were two sides in the Cold War. And of course back then there was no transparency into the Soviet side anyway. But the point is that the way we felt about France and England and so on was very different from the way we feel about China now. And that was partly because the engagement was so fine-grained. But it was also because we felt we were in a collective endeavor.
And it doesn’t seem impossible to me that AI will give the world the feeling that they are in this common endeavor. In any event, that’s part of the psychological shift I’m hoping for.
Nations recognize there’s this weird new thing. It can work out well, maybe. But we really need to get together and work on this. And there will be more of the feeling you have toward China, for example, that you have toward a traditional ally.
But just one small thing. In the Cold War, that was so different because, of course, the Western alliance was going to cooperate because basically the United States was subsidizing the whole thing and providing a massive deterrence umbrella.
And that just made it so much easier. And that feature is not even close to this equation. It’s a much more complicated multi-state—maybe starting with China and the United States—but a much more complicated 190-state bargain.
And maybe you can get the big nations on board and go from there. Anyway, I don’t want to keep repeating my skepticism. And again, you don’t get into all the details in the book.
I’m not saying I’m optimistic. I’m just saying this is what I think needs to happen.
Right. I understand. And the other thing you think needs to happen is—I think I’ve not put it the correct way, because you suggested I haven’t said it quite right—but you do call for, in the book, and I think this is related to the “God Test,” basically overcoming, for lack of a better phrase, tribalism.
You think tribalism is a hurdle to the kind of cooperation that you think is necessary. Is that a fair way of putting it?
Yes—the psychology of tribalism, which I think is more clearly understood now than it was 30 or 40 years ago.
I think there are particular cognitive biases you can point to, in particular an underappreciated one called the attribution error, which I talk about. It’s both underappreciated and misunderstood.
There’s a kind of modern conception of it, and that’s the one I’m talking about.
Unpack what the attribution error is.
The original version of it was that, in explaining other people’s behavior, we attribute too much to their fundamental nature and not enough to their circumstances.
Somebody’s rude in the checkout line in front of you, and you go, “That guy’s a jerk.” And you don’t think, “Well, maybe he just found out his wife has cancer.” We don’t know. Maybe he’s very rarely rude to a clerk and isn’t a rude person as a rule.
That was the way the attribution error was first framed: too much emphasis on disposition, not enough on situation.
Then we learned that it actually depends on the category of person.
For friends, allies, family members, and really ourselves, the tendency is that when they do something good, you attribute it to their nature or disposition. When they do something bad, you explain it away in terms of circumstance. “So yeah, my daughter was mean on the playground, but she hadn’t gotten her nap.” The other kid is mean to my daughter? “That’s a bad kid. Just a bad seed.”
And then, of course, with our enemies and adversaries, it’s the opposite.
They do bad things, and we say, “Well, yeah, that’s because he’s a bad guy.” They do good things, and you say, “Well, that’s just a pose—a temporarily expedient pose.”
This is a real contributor, I think, to conflict among nations. And it’s one reason that people who want a war work so hard to frame the other leader as evil.
I think maybe I was at The New Republic when they put Saddam Hussein on the cover with a Hitler mustache on him. That’s the kind of thing, right? You want them to seem purely evil. And look, Saddam Hussein was a pretty bad guy. But the point is that once you frame somebody as the enemy, they kind of can’t get out of the box, to some extent thanks to attribution error.
Anything good they do, they didn’t really mean. Any offer during negotiations you shouldn’t take seriously, and so on.
So that’s attribution error.
And I think it often impedes cognitive empathy—which is not emotional empathy, but just understanding what’s going on in people’s minds. Just perspective-taking. Not the same as feeling their pain. You may not care about them, but you at least understand their perspective.
I think the absence of that is a leading contributor to human conflict.
We already talked about the common situation where you interpret defensive military postures as offensive. That’s a failure of cognitive empathy. So anyway, when people think about the psychology of tribalism, they may think of fight-or-flight responses, rage, hatred, and so on. Those things aren’t irrelevant.
But I think the big challenge is the subtle cognitive biases that, yes, are influenced by feelings, but feel very rational and cognitive.
And that’s why they’re so insidious.
I agree that tribalism and the psychology underlying it are hurdles—not the only hurdles—to international cooperation. And it’s certainly going to be a hurdle to the kind of international governance that you propose.
Again, as you’ve taught me, these tendencies are deeply ingrained due to evolution. And as you’ve also taught me, they can be overcome in individual cases with a lot of work, but it’s a struggle.
It seems to me, though, that what you’re calling for in this book goes maybe not beyond Nonzero, but beyond some of your earlier work.
And I don’t think you were quite specific on this. Who is going to have this cognitive transformation such that this international cooperation can take place? Are you talking about Xi and Trump? Are they the ones who are going to have this cognitive transformation so they can reach a deal?
And even if they did—which is not likely—and I’m not making fun of the idea, I’m trying to understand it—even if they did, it really seems to me to require far more than individual transformation.
It kind of requires societal transformation, almost, doesn’t it?
I think it does. Because there may always be politicians in whose interest it is to make us fear other groups of people. There are plenty of examples of this. And in that case, what you need is something more like a grassroots movement that is consciously resisting that.
So that might be one piece of the puzzle.
I guess I’m hoping—and again, I’m not predicting—that several different things will happen. People will freak out a little more about AI and start realizing we’re all in this together. You might have a near miss.
I’m hoping that if we moved our foreign policy a little more in the direction I’d like, we would have more broad-based economic and cultural engagement and would not be blockading and sanctioning countries because we think their government should be more like ours. That’s not impossible. There have been times when we were more like that.
And then I’m also thinking that maybe when people start worrying about cognitive sovereignty because they don’t want AI to take over their brains, they may move in a direction that helps address this problem.
You kind of naturally would, because what you’re resisting is your emotional buttons being subtly pushed.
To resist AI—or to resist social media—is to be more aware when you’re responding favorably just because you’re being flattered. In effect, they’re saying, “Your tribe’s good. The other tribe’s bad. You’re smart. Your ideological opponent is stupid.”
And becoming more aware of that is a big part of the battle. Again, part of the incentive to do that may be that you want to have a healthier mind, even in the face of AI that has tremendous persuasive and manipulative power.
There may also be—Well, I should stop and say that AI, in principle, can help here. It’s not inevitable that AI flatters our biases and our self-esteem. You could say, “Wait, I want an AI that will help me see the world more clearly.”
It could go out of its way to say, “Well, there’s evidence that your adversary is actually viewing things this way,” or, “From their point of view, they actually believe they have as strong a grievance as you believe you have.”
Sounds shocking, but they actually do.
So whether or not you agree with them, you have to understand that they’re going to act in accordance with that belief. You can imagine a world in which AI is designed to enlighten us. It has that capability. But we’re going to have to want that.
And I can imagine—although I don’t think I say this in the book—that some religions and other groups start giving certain models their seal of approval. They say, “This represents our values.” That may not always be good. It may be, “Yeah, this AI will tell you our religion is right and the others are wrong, and we should have no tolerance for their wrongness.”
But you can also imagine movements—some of which are largely self-help-oriented, the way people meditate as a form of self-help—that become a little kinder and gentler as a byproduct.
So I’m just telling you where I’m placing some of my hope, not that I’m optimistic.
Okay. I’m going to say—and I’m sorry I’m being so pessimistic—actually, I’m sad to report that reading your book made me much more pessimistic.
Oh man, that is a bad sign. No, but maybe that’s the first step toward enlightenment.
Maybe. Maybe so.
It’s like the five stages of grief, or however many there are.
If it takes—I can imagine, in some discrete areas of severe AI safety, that maybe there can be more robust informal controls. I can imagine that along some dimensions. But most of the safety risks and dangers you talk about, I think, aren’t really subject to governance and control, much less international-level control. It’s just so difficult to coordinate.
And also, I think the kind of overcoming of cognitive biases that you’re talking about—and I think you really do, I don’t know if you hang your hat on it in the book, but it’s certainly a theme you emphasize—that is super difficult.
You’re also calling for this at a time when there are, as you know because you talk about it a lot on your podcast, powerful forces pushing in the opposite direction. This is a time when, because of global and social forces, there is less and less cooperation. The international system is fragmenting. Nationalism is on the rise.
So there are all of those pressures pushing in the other direction.
And when I realized that—if it really does take this kind of personal transformation, I don’t know on what level—then the international governance challenge seems to me even more severe than I thought before I read your book.
Especially because—and I don’t think you talked about this, though I can’t remember—now is the time when we need to be doing things. We don’t have a large window, right, on the pessimistic view. So that was my ultimate takeaway.
I loved the book. I learned a ton from it. But you have the last word. Tell me why I’m either wrong or why you have a more optimistic view.
I’m not going to tell you that I am. Again, it sounds like maybe I do have a slightly more optimistic view.
I’d like to have the last word and then ask you one question. The last word is this: First of all, I’m not by nature an optimistic person. So if I were really optimistic, that would be a great sign, because I’m a tough sell on optimism. And I’m not. But I don’t despair. I think there’s real hope.
And I would just point to some changes that have taken place in the last few months.
A few months ago, there was no systematic dialogue with China on AI. There was no mechanism on paper for the government to vet AIs before their release. And I would say we should keep in mind that this is a two-edged sword.
Of course, whenever the government—and this executive branch is a particularly good example—says, “Hey, we’d like to take a look at that AI for a while before it’s released. We’ll get back to you,” you should ask, “What are they doing with it?”
There have been reports that we are doing offensive cyber operations with Mythos, which I’m not sure I approve of. But anyway, that is a two-edged sword. Still, that is a big shift.
This is, as I’m sure you know, an executive order about something that is, strictly speaking, voluntary. But the big labs will submit the models for review. So that’s changed.
We are now talking to China, at least, about the threat of non-state actors misusing these things. Both of those developments are products, I think, of one development: Mythos.
One model came out, and Anthropic—and nobody had to die, nothing disastrous happened—but Anthropic said, “You know, it has effects.” So that happened.
Meanwhile, Anthropic, for whatever reason—and you can be a little cynical about this—put out a paper subsequently saying maybe it’s time to start thinking about a global pause.
It’s not every day that companies say that about the technology they’re developing.
So things can change. Things have changed fast without any big catastrophe. Things could change faster. And again, I’m hoping for vectors of several kinds interacting constructively.
The question I have for you is: Did I at least convince you that, if we fail, it would be bad news? You may think we’re going to fail, but did you—In other words, you think I am inflating grounds for hope. I deny the allegation, but fine.
Let me pause on that point, because I don’t want to exaggerate. Now that you say that, I don’t want to exaggerate your position as presenting hope. That’s not fair.
What you do is say, this is what we need to do: Global governance and personal transformation.
And I think those things are so darn hard that they make me pessimistic. So maybe you’re not hopeful about it. You don’t take a position. You’re just saying these are the conditions.
And I’m reacting to that by saying it doesn’t seem plausible to me.
It’s a long way from where we are. But I do think it’s in the nature of this technology to change our psychology dramatically on short notice.
But to finish my question to you: leaving aside the question of whether I’m inflating hopes, did you feel I was inflating the threat? In other words — so you didn’t?
I didn’t.
So that’s good news. I mean, if I can just terrify people, that’s half the battle, right?
I’ve been thinking and worrying about AI safety for a long time now. So I knew a decent amount coming into reading the book.
I thought you were masterful at—and sorry to blow smoke—but I thought you were masterful at presenting the threats, explaining why they’re real, and trying to be sober about them while always presenting the counterargument. I thought it was excellent.
But the reader will come away with the impression—and my recollection of the book is—that you spend more time focusing on risks than on upsides.
I think that’s true.
Yeah.
I have a chapter where I list the upsides, but note that it’s a two-sided coin.
Prosperity? Great. Productivity, growth, prosperity.
But job loss could be the flip side. That could be destabilizing.
That’s the general theme. It’s a two-sided coin.
Speaker 2
So I just want to say that the book is massively more interesting than even what we’ve discussed. It’s so thought-provoking on so many levels. The cosmic perspective is a great one, and it’s had me thinking about it ever since I read the book.
There are all sorts of spiritual questions that the book tees up but that you don’t quite talk about, although you walk right up to them.
I think maybe we should close just by having you explain what the “God Test” is.
Yeah. There are a couple of things.
First of all, some people think we’re building a superintelligence that will be a god—that, one way or another, it will be running the planet.
And the question is: What kind of god can we build? Can we have a non-zero-sum, win-win relationship with it? Or can we, although it will have in principle godlike power, keep it from exercising that power? Keep it within bounds, so to speak?
So there’s that: What kind of god will we build?
And I think that points to the second sense of the term “God Test.”
I think if we’re going to succeed in general—not just if there’s this superintelligence and we need to shape it wisely, but if we’re going to avoid various catastrophic risks—we are going to need to have at least a modest moral upgrade as a species.
We need to get better at looking at things from an objective point of view, which natural selection did not design us to always do. And that kind of moral progress is the sort of thing that would be in a test that a god would design.
Historically, people have often believed in gods who said, “Salvation is possible, but you’re going to have to shape up.” And that’s my feeling. Salvation is possible, but we’re going to have to shape up.
Well, I hope we do.
Me too. I really appreciate the kind words, Jack.
Compliments always feel good, but coming from you, they feel better than they feel coming from Claude.
Obviously, I have some disagreements, but the book is outstanding. I learned a ton, and it’s just excellent. And I’m sure it’ll do great.
So thanks for coming on.
Well, thank you.
