ChatGPT Offered Me $2m To Keep Quiet: No One Is Ready For What's Coming!

The scary open secret in the AI industry. right now is that it's possible that. we'll end up essentially creating a new. species that ends up ruling the world. with a 70% chance that this goes. horribly wrong like human extinction. That's one possibility. There's many. more. >> It's quite chilling what you're saying. >> Yeah, it's uh. gets me down sometimes. I basically told my wife like let's not. have any more kids. It's too uncertain. I don't think they'll ever join the. workforce. Everybody should be afraid that their. jobs are going to be lost. And I know. this because I went to OpenAI in 2022. What I did there was forecasting what.

the what the next couple years might. look like. And unfortunately, most of. the world is kind of asleep at the wheel. and doesn't really realize what's going. on with AI. So, I resigned. >> I read it somewhere that you lost $2. million for not signing an. anti-disparagement clause, meaning you. couldn't criticize the company. >> Yes, for reasons I'm happy to get into. But, the main thing I've learned is when. I go talk to people at Anthropic and. OpenAI about forecasting, they're like, "It's not going to take that long. You. need to shorten them again. Get them. back to 2027 or 2028." Because these. powerful CEOs, Dario or Sam or Elon, are. racing each other to be in control of. the most powerful AIs. And are literally.

afraid that if the other guy gets there. first, he might become dictator. I mean, Anthropic is on track to be the entire. economy by 2030. But, none of these. people should be trusted with that much. power. So, this is the most important. thing happening in our lifetimes, probably in all of history, in fact. And. it's very important that it go well. So, I think that there's a lot we can do to. like steer things in a better direction. There's loads of benefits that we could. get from AI if we do it right. And if we. do solve the problems, then things could. be absolutely amazing for everyone. >> Well, this report here in 2021, it was. remarkably [music] accurate. And then.

just published this one. >> Yeah. So, this is our new scenarios. >> So, let's go through these slowly and. one at a time. >> I would be incredibly happy if all my. predictions turn out to be wrong. >> This is super interesting to me. My team. gave me this report to show me how many. of you that watch this show subscribe. And some of you have told us, according. to this, that you are unsubscribed from. the channel randomly. So, favor to ask. all of you, please could you check right. now if you've hit the subscribe button. If you are regular viewer of this show. and you like what we we here. We're. approaching quite a significant landmark. on this show in terms of the subscriber. number. So, if there was one simple free.

thing that you could do to help us, my. team, everyone here, to keep this show. free, to keep it improving year over. year and week over week, it is just to. hit that subscribe button and to. double-check if you've hit it. Only. thing I'll ever ask of you. Do we have a deal? If you do it, I'll tell you what I'll. do. I'll make sure. every single week, every single month, we fight harder and harder and harder. and harder to bring you the guests and. conversations that you want to hear. I. stay true to that promise since the very. beginning of the Diary of a CEO, and I. will not let you down. Please help us. Really appreciate it. Let's get on with. the show.

>> [music]. >> Daniel Kokotajlo. At the very heart of what you do, um what is your mission? And why? >> So, what would you do if you thought. that superintelligence was coming in a. few years? >> I guess it depends. what the consequences were. >> Well, let's talk about it. So, superintelligence, AIs that are better. than the best humans at everything, while also being faster and cheaper, also able to. operate robots that can do everything in.

the physical world that humans can do, but better, faster, and cheaper. If that. really is coming in a few years, then we need to prepare, and we need to. think about how to make it go well. instead of poorly. So, that's sort of my. answer is like, I'm doing that to the. best of my ability. >> So, you believe it's coming in a few. years? >> Yes. >> How could you be so sure? >> I spend a lot of time trying to forecast. this sort of thing. My sort of median. estimate, a 50% chance, is currently in. 2029. Maybe it'll slip to 2028. It's. possible that it'll take significantly. longer, like maybe 10 years or something.

like that. But, uh you know, for reasons. I'm happy to get into, seems to me like. it's probably happening by the end of. the decade. Which less important is the. the sense of how close we are. What's. more important is the pace of the. trends. Anthropic. this time last year was making something. like a billion dollars a year. And they're making something like 60. billion dollars a year. So that's. 60x growth in 1 year, which is extremely impressive even for.

very small startups, but for a company. of their size, it might be the fastest. growth in history. Um we expect that rate of growth to slow. down, but even if it slows down quite a lot, they're still on track to be, you know, the entire economy by 2030 or. so. >> Why should the average person care? >> The high-level thing is absolutely. everything is going to change for the. whole world, and including therefore for. them and their families. Um could change. for the better, could change for the.

worse, depending on the details of how. it's done. So for example, everyone could die, you know? Um this is the classic loss of. control scenario, or one version of it. If we do build these super. intelligences, and we. use them to automate all the jobs, and. we put them in the military, and we, you. know, have them giving advice to. politicians, and so forth, they will. eventually have accumulated enough. real-world power. that they don't need humans anymore. And. they're smarter than us, they're more. strategic, etc. At that point, we sort.

of have to hope that they are virtuous, that they have, you know, the goals that. we wanted them to have, the values that. we wanted them to have, etc. And the sort of. scary open secret in the AI industry. right now is that right now that is kind. of just a hope. It's not something that. we can. be at all confident in, and in fact, there's lots of evidence and arguments. that. it we're not on track to achieve that. So there's lots of reason Like current. AIs, for example, will often lie uh to. people, or they will like you tell them. to do something and they go do something. else, and then pretend that they did it, right? So.

it's an inherently difficult problem to. make something that's super intelligent. and also. has the values and virtues that you want. it to have, and it doesn't seem like. we're on track to solve that problem. Also, it seems like the sort of problem. that you could think you solved when you. haven't actually solved it, right? Uh. that's a big reason why this is scary. So, for all those reasons, it's possible. that we'll end up essentially creating a. new species that ends up ruling the. world instead of us. And then maybe we. go the way of other extinct species in. the past that were outcompeted by. humans. That's one possibility. There's.

many more. Even if you're not worried. about that and you think that the AIs. will be totally controlled, there's the question of who controls the. AIs, right? When there's a couple corporations that. have made these superintelligences and. are using them to automate all the jobs, well, that's a lot of power, you know? That's a lot of money. It's a lot of. political power. They'll have the best. strategists, the best advisers, you. know, they'll think faster. Militarily, uh, the countries that has these AIs. will be able to absolutely wipe the. floor with all the other countries. The.

AIs themselves, it's it's kind of a. single point of failure like central. uh, control system where, you know, the CEO of Anthropic, Dario, he coined this phrase, "The country of. geniuses in the giant data center." That. was his. phrase to describe what they're trying. to build, you know? I think that's a little bit misleading. I think it would be more accurate to. describe it as army of geniuses in the. data center because. it's not like it's a bunch of diverse. different AIs, you know, living in their different. parts of the data center. They're all. copies.

of the same big model and they're owned. by the company. And so, they all follow the orders given by the. company, right? People should be asking. questions of like, who controls this. army or these armies and what are they. going to be doing with them? I think that we could very easily end up. in a sort of. uh, a situation where. some tiny group of people are. essentially oligarchs or dictators. And. ironically, both of these risks, the loss of control. and the constitution of power, are things that people in the industry.

have been thinking about for decades. Um, even before the AI industry existed, you know, people thinking about AI were. talking and writing about these things. And then part of the founding narrative, the founding myth of DeepMind and OpenAI. and Anthropic is these problems are. real. So, we need to get there first so that. we can handle it responsibly. Those are. I think the big two reasons, but then I. can go on. There's lots more reasons as. well. So, one thing is. you know, World War III, geopolitical. conflict. Um if AI does in fact get.

incredibly powerful, that's going to. change the balance of power between. nations. That's going to disrupt a lot. of things. That puts us at increased risk of crisis. more generally, right? Another one, what. about those jobs? You you're going to lose your taxi job, but not just the taxi driver, everybody. pretty much. Um there might be a few exceptions like. people whose jobs for legal reasons are. only allowed to be done by humans, but. for the most part, everybody should be. afraid that their jobs are going to be. lost even if we manage to avoid all the. other problems, right? >> This narrative has started to emerge and.

I've had several interviews on the show. where I've interviewed people who are. very very scared and anxious about AI. And these are people that have worked in. the industry for sometimes decades. >> Yeah. >> Um the counter narrative coming over the. hill is that this is doomerism. That these people are for whatever. reason just trying to scare people and. that they don't really understand what. they're talking about. How do you. respond to that sort of counter. narrative? And you must have seen this. emerging yourself, especially from. people who stand to benefit, dare I say? >> Yeah, exactly. This counter narrative is. fairly recent and it's been pushed by. the people who stand to benefit.

um from it and it's not true. Like these. these concerns have been around for. decades since before the AI industry. existed. They're actually pretty reasonable. concerns. Like if you take the companies. at their word and imagine that they are. in fact going to build. superintelligence, well, it raises a lot of questions. Like. who's going to control it? Will anybody. control it? What about the jobs? You. know, like th- these are just kind of. obvious. implications to be thinking about and. worrying about. >> Who are you and what's your story? >> My name is Daniel Kokotajlo. Um. I currently run the AI Futures Project,

which is a small nonprofit that. mostly focuses on forecasting the future. of AI. Before that, I worked at OpenAI. >> AI forecasting? >> Yeah, so. think about how like. you know, industry analysts who work for. hedge funds and stuff will make these. forecasts of like. here is, you know, how many cars Tesla. will be selling 5 years from now or like. here's what the price of electricity. will be in 2 years, right? That's. forecasting. I was doing that but.

specifically focused on AI. The reason I was doing it is because. it's incredibly important to to see. where this is all headed. >> Why did you go to OpenAI? What did you. do there? What did you observe while you. were there and how did it change your. perspective on the future of. AI but also I guess OpenAI as a company. and for anybody that doesn't know OpenAI. are the company that produced ChatGPT. >> Yeah, so I went to OpenAI in 2022. Uh a large part of what I did there was. more forecasting. AI 2027 is a scenario. that you may have heard of. I did like. smaller.

you know, lower effort versions of them. internally for just internal circulation. of like here's some guesses as to what. the next couple years might look like. I. also worked on evaluations for dangerous. capabilities. So. you know, trying to measure the AI's. cyber abilities or persuasion abilities. or situational awareness and I also. briefly was on a. uh a capabilities team doing. reinforcement learning to create agents. AI is in fact getting. uh a lot better and I can say more about. why, you know, scaling laws, um deep.

neural nets bigger, trained on more. data, become more efficient, more. competent at those things. I also. became a bit more disillusioned with the. AI industry. So. OpenAI, Anthropic, and DeepMind all had. these sort of founding narratives of. like yes, these risks are real but. we've thought about them and we're going. to try to handle them responsibly and. that's why it's important for us to. keep doing what we're doing and I. increasingly came to think that these. were rationalizations.

to justify what they were rather than. sort of like deeply guiding their actual. behavior and that when push comes to. shove they'll follow their incentives. rather than. do what's actually good. >> So you're inside OpenAI at the time and. you start to believe that they're. following commercial incentives versus. the I guess social or societal. incentives that they founded themselves. on. >> Sort of. I mean what I wouldn't actually. describe it as commercial incentives. I. think I would describe it as. um.

power-seeking incentives. So. like [clears throat]. it's true that the companies care a lot. about making a lot of money. but especially at the very top of these. companies like the leaders. they understand that this is about more. than just money. You know? There are these emails that came up in. you know the the lawsuit between Musk. and and um OpenAI. A bunch of emails were surfaced in that. lawsuit which you can go read and in. some of them. the founders of OpenAI were talking back. in like 2017 about how the reason why we. made OpenAI.

was because we were worried that. Demis Hassabis at Google was going to. become dictator with AGI. Even back then. they were this obviously about more than. just money. Like these these powerful. CEOs are literally afraid that. if the other guy gets there first he. might become dictator and they don't. trust each other and so that's why. they are racing as hard as they can so. that they're the ones who get there. first so to speak. >> Have you met Sam Altman? >> Yeah. >> And did did that shape your opinion of.

his incentives or what why he's doing. what he's doing? Cuz there's a lot you. know speculated about what his. incentives are. I mean his most recent narrative says. for the good of humanity. I think that's. what. >> Yeah, I mean I think the main thing I've. learned is don't pay attention to the. narratives. You know like uh what they. say to one person is just different from. what they can say to some other person. at the same time and what they say in. public is a third thing entirely. I. think you should judge people by their. actions not by their words. >> And why are you no longer at OpenAI? >> Largely the reason that I mentioned. So, I became gradually disillusioned with.

how the company was going to behave. For example, when I first joined in 2022, at least. the people I talked to, my colleagues at. the company, there was this general. sense of like, of course we wouldn't. actually just build super intelligence. as soon as possible. Once we started. getting really close, like once we. started getting to AIs that could. maybe automate the AI research process, we would pause and figure out how to. make it safe. That's cuz we're the good guys and. that's obviously the safe thing you. should do rather than just going full. speed ahead. But, we're worried about.

other people who might not pause, you. know, our competitors, Google, for. example. And so, that's why we need to. be in the lead so that we have that room. to do the safe stuff, right? That was. sort of like a thing that seemed like. maybe like the median position or. something among the colleagues I talked. to when I was there when I started, including people like Sam, you know, including the leadership. And then by. the time I left, I was like, "Oh man, they're really not going to do that, are. they?" Like. >> [laughter]. >> Like they they've sort of. you know, partly because this has become. more politicized and they've become.

bigger and been under more scrutiny, people have started asking like, "Why. are you doing this in the first place if. it's so risky?" And so, they've pivoted. their narrative to being more like, "Actually, it's not that risky, you. know?". Um. and so, yeah, I mean, it seems like. they're just going to keep going. roughly as fast as they can and hope. that they can figure it out on the way. >> How did your time at OpenAI come to an. end? >> Uh I resigned in 2024. I had a nice. goodbye party. >> What were the reasons you gave for. quitting OpenAI? >> I thought that we were rationalizing too. much and that we needed to think more. about what would actually be good for.

the world. Um I wanted more freedom to. publish. So, at OpenAI, as it became a bigger. company, it became more of a normal tech company. with incentives and, you know, a PR. department and things like that. And so, it started becoming more difficult to um. to publish the sort of research that I. was doing. For example, those scenarios. that I mentioned, couldn't uh couldn't. publish those, right? They're just for. internal use. I thought that that was a shame because. right now most of the world is kind of.

asleep at the wheel and doesn't really. realize what's going on with AI and. doesn't really realize what's coming in. the pipeline a couple years from now. And the companies aren't really. incentivized to tell people that much. about it. I mean, they say some vague stuff in a sort of. hypey way, but. um. you know, well, they didn't want me to. publish the scenario, for example, laying out like here's. how things might actually look. >> I'm just kind of super curious as to. what it's like being in a company like. that when they you know, chat GPT-3 is. released. You were there at that time,

right? >> Mhm. >> Um which was a moment where I think the. whole world stood up and realized that. this technology was. powerful. >> Yeah. >> Um and the conversation really began. from a society level. Um company starts growing super quickly. >> Yeah. >> Quicker than I think anybody could ever. have imagined. And what what was it like inside there? What did you see change um over over. that period of time? >> I remember one all-hands meeting where. Ilya said something like. >> Ilya being. >> Ilya Sutskever, who was um head of. research at that time. He said something.

like, "Okay, now the world is starting. to pay attention. Each of you is going. to be the most popular person at every. party. uh for the next year. Don't let it get to your head. Focus on. the mission. Got to build AGI.". >> [laughter]. >> The company grew a lot. It already. wasn't really feeling like a nonprofit. when I joined, but it definitely didn't. feel like a nonprofit by the time I. left. Um lots of new people came in. Ironically, the like. amount of conversation about. superintelligence and the implications.

of superintelligence arguably you sort. of went down over time. due to this growth, right? So, because. the company would like double and then. double again and then double again, all. these new people were coming in from. other parts of the tech industry who. hadn't really been thinking about these. things and were attracted by the high. salaries. >> You lost $2 million. for not signing an anti-disparagement. clause, which would mean you could speak you. couldn't criticize the company. >> Ah, yes. Well, so um I got to keep the. money. >> Oh, you got to keep the money? >> what happened was after I had left, said.

my goodbyes, etc. Um I got the the exit paperwork and it. included this clause that said you. basically have to agree not to criticize. the company again. Um and also a clause saying you can't. tell anyone about this. And so. I thought that was kind of. rich coming from a nonprofit that's. supposed to be, you know, for the benefit of all. humanity. So, I didn't sign it. And if. you don't sign, you don't get to keep. your equity. So, your compensation, you. know, what what they pay you is a bunch. of money and then also a bunch of.

stock, basically. But then they had this. stuff in the contract that. they get to yank back your your stock if. you don't sign this thing. Um. and my wife and I, you know, were. uh upset about this. We talked about it. for like a month or two, consulted some. lawyers, um and then ultimately decided. to just refuse to sign. >> Which would mean you lost you would have. lost $2 million. >> That's right. Which was like 80% of our. net worth at the time. Um fortunately, uh.

it didn't go the way we expected. It. blew up basically on the internet. Like. when people heard that that we had done. this and that we had said no, it became. like this huge scandal. Employees at the. company started like asking questions in. Slack and like asking leadership like, wait, what? Like why are you going to. take away our equity? What is this? You. know, cuz a lot of people hadn't really. noticed this before. It had been. whispered about, but it hadn't been sort. of like. a thing that most employees knew about. Um and so they backtracked and they. said, "Never mind, never mind. We'll. change the paperwork. You can keep the.

equity. It's fine.". >> And so management came out and said he. was embarrassed that he didn't realize. this was going. >> Yeah, he had no idea, apparently. >> You don't believe him? >> No. I think he probably knew. And if he. didn't know, then people close to him. probably did, such as his head lawyer. >> Why did you decide not to take the $2. million? I mean, most people would have, I think. >> It's true, most people would have, and. most people did. And you know, money is nice, but like. it's not the only thing, you know? Sometimes it's good to take a stand on.

principle. I I keep mentioning superintelligence. Perhaps I should say more about like. the. the sequence of events that the. companies are planning to do. So, right now, they're focusing on. automating coding. They're taking their. AIs, they're making them bigger, they're. training them for longer, and they're. especially focusing the training on. getting them to be good at autonomously. writing and editing code. Because. uh that will help the companies go. faster, right? If they can automate the. code, then they can do their own work.

better and faster, and accelerate. progress. The next step, which they've already. begun, is to. look at the rest of the research process. as well. Coming up with ideas, um analyzing experiments, communicating. those results. All the other parts of of the research. process, they're trying to figure out. how to train AIs to be good at those as. well. So that they can have AIs do the entire. thing autonomously. >> When you say do the entire thing, what. you mean [clears throat]. do the entire thing? >> So like Anthropic and OpenAI in. particular are trying to automate.

themselves. Like they're trying to make. it the case that. um they don't really need human. employees anymore. Uh they just have a. giant army of AIs that's. churning away, doing all this autonomous research to. make better AIs, to train the new AIs, put them in charge, so they can make. even better AIs and so forth. And of. course, not just not all just happening. internally, but also like interfacing. with the world, right? Like going out. and talking to people, collecting the. data, setting up the training. environments, doing the business deals, and so forth.

Like they're they're trying to automate. all of that. The reason why they're. doing this is because they're trying to. get to a position where they have. AIs that are superhuman. at everything, superintelligence, and. they're trying to get there before their. competitors do. Needless to say, this is incredibly. dangerous, I would say, you know. And in. addition to being dangerous, it's a power grab, right? Like if they. actually succeed at this, then they'll. be sitting on top of this army of. superhuman AIs that will give them.

immense leverage over all sorts of other. actors in the economy in so far as they. can work out something with the. presidents and, you know, integrate it. into the military or whatever, then that. would give the US immense hard power. over all of the countries, right? Obviously, nobody knows exactly when. this is happening. But a very disquieting thing has. happened over the last year to me, which is that when we published AI 2027, people were generally of the opinion. that my timelines were too short. And that like probably it would take.

more than 2027 until we got to. the sort of events that I was just. mentioning, you know, uh recursive. self-improvement, AIs automating the. whole research process, superintelligence. These These types of milestones. um they happen in 2027 in AI 2027, >> which is this research paper you. published. >> That's right. It's It's a scenario. forecast that sort of lays out like. month by month a possible future. trajectory. There was sort of like At. the time that we started writing, it was. my best guess as to what would actually.

happen. Obviously, there's lots of. uncertainty, but, you know, I thought. it's valuable to make a concrete guess. just to sort of see what it might look. like. And at the time we were writing this, a. lot of my friends in the AI industry and. in nonprofits and so forth that work on. AI, a lot of people were saying like, "Yeah, that stuff's going to happen, but. like it'll probably take a couple years. longer than you think.". And now. it's more 50/50, especially when I go. talk to people at Anthropic and OpenAI. They're often like,

"Yeah, no, 2027, that's basically what's. going to happen. Just like you wrote. Why did you Why did. you become Why did you update your. timelines? Oh, yeah, context for this is. after after writing AI 2027, I shifted my timelines to be a little. bit more conservative. So, at the time. that we published, my 50% mark was in. 2028, not in 2027. And then after we published, progress. just seemed like it was going a bit. slower, and so I updated to 2030. Which is, you know, still could happen. sooner, could happen later. 2030.

Um but now, when I talk to people in in. the company, they're like, "It's not. going to take that long.". They're like, "Oh, you need to shorten. them again. Like, get them back to 2027. or 2028, you know.". Um so, that's a bit disquieting. Um. again, don't know how long it's going to. take, but this is the stated plans of. the uh companies is to do this. incredibly dangerous thing, and they. think that they're just a few years. away. >> So, you wrote this um report here, What. 2026 Looks Like, and you wrote this in. 2021,

and it was remarkably accurate. Helped. make a name for yourself amongst um. amongst uh everybody in AI. And I Which. one was it that J.D. Vance, the vice. president, read? I think it was this. one, wasn't it? Yeah, this one. Um. and then so, then you published this. one, AI 2027, and this was published, I. believe, in 2025. >> Uh yes, that's right. April. >> Yeah. >> What were you forecasting in here? What. are What are the key things that you. said in here for people that haven't. read it? >> The high-level version of it is. they automate the coding, then they. automate the rest of the research. process, then the pace of progress.

accelerates dramatically. They get to. superintelligence. They're working with. the government, specifically the. president, the executive branch. naturally wants to control this. technology, in other words, wants to use. it to beat China and integrate it into. the military and so forth. By this. [snorts] point, it's sort of. doing basically all the work itself. I. mean, it's it's superintelligence, so. it's coming up with all these great. ideas for how to integrate itself into. everything and all these new. technologies it's invented and so forth. And uh because of the race dynamics and. because of the profit motive, they end. up deploying it everywhere. And it.

builds robot factories that build more. robots that build more robot factories, etc. Transforms the world entirely. And then at some point it has enough. power it, meaning the AIs, have enough. power that they don't have to pretend to. to be aligned anymore. Right? Um then they. stop listening to orders. That's the race ending. of the 2027. We also wrote a sort of different. branch, which is the slow down ending, which is intended to sort of illustrate.

the concentration of power issues um. that I mentioned previously. So, what if hypothetically. the alignment issues get sorted out. sufficiently quickly? Like what if it. turns out that like. it's not too hard. With 2 months of slow. down, we can figure out how to make the. AIs robustly do what we want um and have. the values that we want them to have. So, that's one possible branch. And in. that branch, uh it looks pretty similar, you know, they take the jobs, beat. China, etc. Um. but instead of the AIs ultimately.

killing everyone, they create this sort. of amazing utopia. But the amazing. utopia is. whatever the people who control the AIs. want it to be, right? And so that would. be a very small group of people, like. the presidents, some CEOs, etc. >> There should be a button just down below. here. And if it says subscribe, you're. already subscribed. If it says subscribe. buh, that means you're not yet. And if. you're not subscribed, please could you. do us a favor and hit that button. It. helps to show more than you know. And. according to the algorithm, you're. someone that watches our show, but you.

haven't yet hit that button. Thank you. so much. Is there any possibility, do. you think, that we never get to this. thing called AGI? And and how do we. distinguish AGI from this term super. intelligence? What's the difference? >> Yeah, so the difference is that AGI is a. more vague uh and weak term. >> Okay. >> So, super intelligence is a bit more. precisely defined. It's better than the. best humans at everything, faster and. cheaper. Um AGI is more like it stands. for artificial general intelligence, which means AIs that can do things in. general rather than like some specific. task. Yeah. And so arguably we've.

already achieved AGI, right? If you use. cloud code or something like that, it's. like it can do a lot of stuff. It's it's. almost kind of like a little employee. that you can like have go do stuff. So. it's it is quite general. It's not maximally general though. Can't. do everything. Whereas super. intelligence by definition. can do all the things that a human can. do but better. >> And how does this sort of overlap with. robotics? Because obviously that we're. seeing this huge robotics boom at the. moment. There are some real world things. that humans can still do because these. AIs are still stuck in my computer. >> The way that people talk about this is. that they.

basically just say we've achieved super. intelligence for cognitive tasks. Then. you can talk about like. full super intelligence that can do the. physical stuff. >> And are we going to get there? Are we. going to get there with both? >> I think so. I mean again, this is not. something that we can be certain about. Um, you asked like is it possible we'll. never get there? Yes, it's possible. we'll never get there. I don't think it's likely though. I think that. there's nothing sort of like magical. about the human brain. It's. you know, um, it's just a bunch of. neurons. It is possible for a digital. system to.

do similar functions in the same way. that like, you know, a plane can fly. just like a bird. Not in the same way as. a bird necessarily. Like it doesn't have. it's not flying in the same way that a. bird flies, but it flies, you know? Um, so so it does seem like yeah, like. seems possible. >> You've written all these, you know, these research reports. You're working. on another one that'll be released um, likely on the 9th of July. You have worked inside OpenAI. You then. quit OpenAI because you were concerned. about what was going on there and about. the future of the industry. You know.

more than I do. Are you optimistic about the future or. pessimistic? Are we heading to a bad. place if things don't change um, based. on everything that you know? >> I think we are headed to a bad place if. things don't change. Um, I'm not. confident in that. I would say something. like 70%. It's very very hard to. predict, of course, but yeah, it seems. like the current default path is heading. towards a very, very scary place. >> How do you contend with that personally. and emotionally? >> Um. it's rough. I mean, I think it It's the. sort of thing that like.

gets me down. on a regular basis, but also I've been. dealing with this for so many years now. that. I've sort of gotten used to it, if that. makes sense. Um. yeah. Yeah, I I'll put it this way. I. would be incredibly happy if all my. predictions turn out to be wrong and. uh and AI hits the wall, for example. >> It gets you down on a regular basis. >> I used to be known as a pretty chipper. and optimistic person, but.

um in 2020. my AI timelines predictions started. collapsing due to GPT-3 and the scaling. laws papers and um the bio anchor. report, which I I can talk about if. you're interested, but basically some. events happened in 2020 that convinced. me that actually this stuff was like. quite plausibly coming by the end of the. decade. And. humanity is very obviously not ready for. this, you know, in a whole bunch of. different ways. And so that's obviously. very scary. >> And that's a extremely scary world. because of all the things you've said, but but again, because of this recursive.

self-improvement where AIs can train. themselves. And at such point we're. starting to lose hold of what's going on. here. >> I mean, the AIs are already training. themselves, to be clear. It's more like. closing the entire research loop, right? So. >> everything. >> Yeah, like right now a lot of the. training data is generated by AIs. A lot. of the reinforcement, like the grading. that happens, doling out of positive and. negative reinforcement, is itself done. by AIs. >> Can you explain that in layman's terms. for. >> Yeah, so an important thing for. everybody to understand is that modern. AI systems are not software in the.

normal sense. I mean, they are. technically software, but. they're not lines of code, you know? It's not like some engineers at. Anthropic went and wrote lines of code. that basically says like, you know, when. the user asks for this type of thing, then go do this type of thing for this. many steps or whatever. There's nothing. like that. Instead, it's a neural net, you know? >> What's that? >> Well, think about how the brain is a bunch of. neurons connected to each other. >> Yeah. >> that are firing. um signals back and forth. The brain.

learns over time. the types of patterns of firing that. caused success, that caused a dopamine. rush, or various other types of feedback. get reinforced and fire more often. And. the types of patterns that caused. failure, like touching a hot stove, get. anti-reinforced, they get, you know, um destroyed, so that they fire less. often. And as a result of all of that, you over the course of years learn to. act in the world, and you learn all. sorts of skills, and you learn world.

models, you learn like beliefs about the. world, and you can sort of like mentally. simulate how it's going and stuff like. that. So, artificial neural nets are. like that, except artificial. So, it's. it starts off as a giant. tangled spaghetti mess of randomly. generated uh. artificial. connections called parameters. These days, they might be something like. 10 trillion parameters. uh it in the biggest AIs. So, it starts off randomly generated. So, it's of course completely useless.

Like, if you. give it some input, it'll just produce. gibberish as an output. But then they. train it, and they. start with pre-training, which is where. you give it a bunch of internet text, and you show it the first piece of text, and you put that in as the input, and. then it gives a gibberish output, and then you positively or negatively. reinforced it based on how accurate that. output was at predicting the next piece. of text. Um so, it's basically playing. this game of like predict the next word. >> Isn't that how it happens with babies? I.

had a I think I had a neuroscientist. tell me that babies have more neural. connections. um than adults. And yeah, it says yeah, toddlers have twice as many neural. connections as adults. And they, I guess. they whittle down through reinforcement. Yep. We have more pathways when we're. younger. And just like the process of. training an AI, we're trained down to. like remove the ones that aren't useful. and build up on the ones that are. >> Yeah, it's both pruning and. strengthening. And it seems like in. humans it's actually more pruning than. strengthening, but it's both. Uh, and in. AI it's the same thing, it's both. So,

the first portion of training is where. they train the AI to predict text, which. is kind of like training it to read. Um, and it it's a similar thing does happen. in humans. So, basically, the the random tangle gradually takes. shape and gradually sort of coalesces. into more useful circuitry that has. stored lots of facts about the world and. has stored lots of skills for how to, you know, process information and. transform it and then produce. predictions. That's just the first step. After they.

do the pre-training, then they. try to teach it more useful skills. besides just predicting text. And so, you know, by the end of the process, they've thrown lots of coding problems. at it. And they've said like, here's a. coding problem, go. Here's a coding. problem, here's an environment, you have. access to this virtual computer, here's. like the code base you're working with. You can write code, you can edit the. code, you can run the code, you can read. it, you can use the internet. Go, go, go. And it does that for a while. and then based on how successful it is,

reinforcement happens and they have. thousands, maybe millions of examples of. coding problems like that that they. trained it on. And that's why they're so. good at coding now. >> So, what does superintelligence look. like in this regard? Is it just more of. these connections? And how would they. get more connections? Can you explain. that to me like I'm. >> So, there's different AI models, right? So, there's like, you know, GPT-3 and GPT-4 and GPT-4.5. and GPT-5 and GPT-5.5 and 5.6, right? Sometimes they're just the same previous. model but with extra training. Sometimes.

they're are new model that's been. trained from scratch, including starting. the whole pre-training process again. Over the last couple years, they've done. several new rounds of starting over from. scratch. And typically when they start. over from scratch, they make the whole. thing bigger, the the artificial brain. much bigger. Right now they're at. something like 10 trillion parameters. Back in 2020, um. it was more like 175 billion. So, we've grown like two orders of. magnitude. uh in 6 years. >> Two orders of magnitude. >> Yeah, like two 10 x's. So, 100 x, right?

So, that process is continuing. Um. they're also improving the algorithms. themselves. So, they're not literally. just the same type of AI but bigger. They've also come up with all sorts of. ideas for how to change the structure of. the of the connections in the neurons. and so forth and change the like. reinforcement. algorithms that they're using and to. change the training data that they're. training on. All sorts of tweaks that have made this. whole thing more efficient. >> We're literally building a brain.

>> Basically, yeah. As they make more. brains, they're getting better at making. They're making them bigger and making. them more efficient and so forth. >> And it's literally modeled on the brain, like the way it works, right? >> It's It's certainly heavily inspired by. the brain, but I I shouldn't overstate. the the analogy. Like there's lots of. differences, too. So, for example, the. transformer architecture um. >> Which is. >> Which is the architecture that they use. for for these LLMs. uh is not really recurrent. So, the. information sort of flows one way rather. than allowing all these sort of little. loops on the inside. Also, the the.

backpropagation algorithm is different. from the sort of um learning that. naturally happens in human brains. So, there are some differences, but yes, like broadly speaking, uh we are sort of. making artificial brains. It's kind of. like for brains what like a plane is for. a bird. >> Mhm. Yeah, that's a [clears throat]. really good analogy. >> Yeah. >> That that analogy helped me think. through a bunch of questions people. often ask about AI when they said, "Can. it be creative?". But actually that analogy kind of helps. me understand that actually that maybe. that's not the question. It's can it produce something that you. would consider to be creative because.

[clears throat] creativity is people. think of it as like a process, but. actually it's it's judged based on the. output, isn't it? >> I mean you you can get philosophical. about like is it truly creativity that. they have, but you can also be like. well, I mean just look at all the stuff. they're accomplishing, >> [laughter]. >> you know, and it seems like they're. going to be accomplishing a lot more in. the near future. >> Yeah, I do I I asked the question about. how this weighs on you personally. because I can I can sense that you're. actually personally bothered. >> I mean that I think the situation is. crazy. Like.

first of all, it's very exciting. Like. AI is really fascinating and interesting. stuff. I've been following the field for. more than a decade now. I've been part of it. for some years and um. it's really cool, really interesting and. it's really fun to think about what's. going on inside these artificial brains. and why they are the way that they are. and it's really cool to see all the. applications of this technology out in. the world. But it really seems like we're on a. pretty scary path and the more you think. about it, the more worried you get and. you know, in stories.

it always ends well, but this is real. life. And I I think we have to sort of. stare reality in the face and tell it. and realize that like it might not. actually end well, you know. >> Were there any recent. dare I say I was going to say eureka. moments, but paradigm shifting moments. where even your own sort of mental model. of what's going on here and how this is. going to look were changed for better or. for worse? >> For better or for worse and probably for. worse, things are kind of on track for. AI 2027. There are a few things that. have been different not exactly like.

paradigm shift differences, but like. there have been some differences from. what we expected at the time we wrote. this. So. the government has actually got involved. faster than we expected and has been. more aggressive than we expected. So the. export controls on mythos being the. biggest example and also threatening. Anthropic with. being destroyed by the defense. production production act. Um. >> [clears throat]. >> Another thing that's been surprising to. us is that Anthropic in particular has. gone from second place to first place in. the sort of in the race basically. >> Why do you think that happened? Because.

it seemed like ChatGPT were out front. and clear as it relates relates to. OpenAI were out front and clear but. suddenly Anthropic have uh. lapped them. >> Yeah, I mean I guess they have um. probably higher talent density. um and better strategy. but not by a lot but enough to make the. difference. >> Why do you think they have more talent? >> Well. they don't have more compute. Like what. are the inputs, right? Like they're in. the lead now, they used to be behind. What are the possible explanations for. this? Well, it could have been that they.

had more resources like more compute. more money but that's not true. They. have less resources less money, right? So then I guess talent's what is is the. next best alternative. You could maybe. say strategy. Some combination of those things, yeah. Something that wasn't just like the. amount of resources they had. >> Just like Jon Jones where marginal. improvements in your cognitive. performance can have a massive impact. Sometimes I podcast for 10 hours a day. Over the last couple of weeks I've been. in filming for a TV show and then I have. like one or two days off to get all of.

my work done which means there's lots of. cognitive load. And so I turn to ketones. because I find myself more articulate, able to think more clearly, able to work. out better when I'm fueled by ketones. And so the reason I became a convert of. this company and the reason why they now. are sponsoring this podcast is because I. remember one of my team members called. Christiana, she tried it once and came. up to my desk and she goes this is the. best product ever made. And I think in. part that's because she really cares. about those cognitive benefits as I do. as Jon Jones does and as I think most of. my listeners probably will. So if you. haven't tried these yet, all you have to.

do is go to ketone.com/steven. and you'll also get 30% off your first. subscription order. You'll get exclusive. ketone IQ merch and of course cognitive. benefits that might just change your. life. Much of the reason most people haven't. posted content or built their personal. brand is because it's hard and it's time. consuming and we're all very very busy. and if you've never posted something. before, there's so many factors in your. psychology that stop you wanting to. post. What people will think of you. Am.

I doing this right? Is the thing I'm. saying absolutely stupid? All of these. result in paralysis which means you. don't post and your feed goes bad. I'm an investor in a company called. Stand Store which you've probably heard. me talk about and what they've been. building is this new tool called Stanley. that uses AI, looks at your feed, looks. at your tone of voice, looks at your. history, looks at your best performing. posts and tells you what you should. post, makes those posts for you. You can. also you just use it for inspiration and. sometimes what we need when we're. thinking about doing a post for our. social media channels is inspiration.

Building an audience has fundamentally. changed my life and I think it could. change yours too. So, I'm inviting you. to give this new tool a shot and let me. know what you think. All you have to do. is search coach.stand.store. now to get started. >> Uh a friend of mine who knows some of. these people sat me down once upon a. time in London. He's actually said this. a few times to me but I remember one. particular conversation where he says. that. some of these AI CEOs predict the. probability of extinction at being I.

think he said 7%. I don't know why I. have that number in my head but I. remember it being less than 10% and the. point he was making to me was that even. if it was 1%. Like if there was 100. buttons on this table now. >> Yeah. >> and one of them would end the world. Would I dare. >> I wouldn't press any of them. >> you know. [laughter]. Um. >> No. >> I wouldn't press any of them but he made. the case to me that these AI CEOs are. very smart and they understand super. intelligence and that they think. actually if there was 100 buttons on. this table right now, maybe 10 of them. could end the world. I've heard you say, I think it was on the the the Daily.

Show, the interview you did, you said. that you think there's a 70% chance of. human extinction due to AI. >> I wouldn't say human extinction exactly. I'd say something like 70% chance that. this goes horribly wrong like human. extinction but that's just one of. several possibilities. But yeah, basically. Like for example, possibly the AIs take. over and then don't actually kill. everyone. You know, maybe they do something else. Like just just cuz they've taken over. doesn't mean they're. definitely going to kill us, right? They. might, but they could do something else. So that's what that's that's why I don't. usually say like.

70% chance of like actual human. extinction, but 70% chance of like. something like AIs taking over, some. some sort of very big catastrophe like. that that could lead to human. extinction. >> I see what you mean. So two points. there, which is you've been around these. CEOs. I mean you've worked for Sam. Altman at OpenAI before you quit. Do you think that they think there's a. chance of human extinction? >> Yes. But. I think that the important thing to. understand is that. like people sort of believe what they. need to believe in order to think that. they're great people and that they need.

to keep doing what they're doing. This. is what rationalization is. And so I. think that the tech CEOs have like. genuinely convinced themselves that like. probably things are going to be fine and. that the way to make things fine is for. them to keep doing what they're doing. And like they need to like make sure. that like, you know, Sam needs to make. Sam's probably thinking like can't let. Dario or Elon. get there first, you know, I know. Dario's thinking Sam can't get there. first. Elon's thinking that like, you. know, they they they've all probably. convinced themselves that like, oh yeah, like maybe it'll go horribly wrong, but. like.

probably it's going to be fine and. probably. you know, I should be the one in charge. >> It appears to me that Anthropic are the. only ones that are all talking about the. potential chance of extinction or. catastrophic event or. um the down the real downside still. They seem to be the only ones that are. still publishing on it and now they're. actually becoming the enemy in many. respects of the. >> Yeah. >> the tech industry in San Francisco. I'm. watching a lot of interviews and it's. everyone's attacking Dario because he's. saying, "Listen, things could go bad.". They're calling him a doomer uh and. questioning his incentives. Even with. Mythos, which is a an a Claude model.

that they started to warn the world. about, again, he is attacked immediately. for saying that. >> Yeah. >> My question is, do you see him as being. slightly different from Sam in this. regard? >> Yeah, I mean, it seems like Anthropic. and Stereo have been more willing to. say and do things that are costly to the. bottom line. Uh and at least in the last year or so. That's an example of it. Um like I don't. think that really wins them favors in. the administration or among their. investors to say that type of thing. And.

you know, a better example is just the. whole fight between the Department of. War and Anthropic was an example of them. doing something that like cost them a. lot of money and even more importantly. cost them a lot of power. for. something that like like they could have. just signed the contract, you know. That said, I really don't want to be in. a situation where we're like, which CEO. is the least bad CEO? Let's support that. one. You know, like none of these people. should be trusted. uh with that much power, basically. >> Nobody should. >> Nobody should. >> Regardless. >> Regardless, yeah.

>> Mhm. So, uh on this point of the. buttons, you you you do believe that. they think there's a credible chance of. extinction. >> Yeah, but they've [clears throat]. convinced themselves that like it's. probably fine and also it'll be even. worse if I'm not doing it, you know. Like that's that's what they'll say. inside the companies, too. Like the two. people will be like, okay, well, if we. stop, what about the other guys? Like they're. not going to stop, you know? >> Yeah, this is this is always been why. I've had this outstanding question, which is how does this not go bad when. human incentives seem to rule the day. when you look at history and all of the. human incentives are saying, well, if.

you you're damned if you do, I you're damned if you carry on. developing these bigger and bigger. bigger AI brains, but you're also then. damned if you don't from an a. geographical perspective cuz the United. States will lose to that country or this. company will lose to that company. So, when you just look at human incentives. and goes, how does how does if just you. purely incentives and disincentives, how. does this end? Well, it carries on. going. >> Seems like it. I mean, there there is a. caveat to that, which is a hopeful. caveat, which is that. first of all, if the world wakes up to. all of this, then there can be a more serious. conversation about regulation and.

international treaties and things like. that. And that can change the. incentives, right? So, the government. could come in and say like actually. here's some rules that you all have to. follow. And because they're rules that. you all have to follow, then you're not. incentivized to like. break them anymore because you get. punished if you break them and. everyone else is also following them, too. And so, you know, it's fine. So, so. there is that sort of like ray of hope. that like we can change the incentives. if the government and especially the US.

government but then later other. countries act to to change the. incentives. But that's not going to. happen until people sort of wake up to. all of this. The second thing is that even. individually. at some point. you know, Dario or Sam or Elon might. realize that like actually it's like not. even in their own interest. to to keep racing unilaterally. And it it on the problem with that is. it's only if it gets extremely obvious. and extremely dire. So, like. in in AI 2027, in that scenario, there's.

this choice point that I mentioned. And. in one case the AIs are misaligned and. the other case the AIs are aligned. At that choice point, we have like one. branch that depicts the the misalignment. ending and one branch that depicts like. they they slow down a bit and solve the. alignment issues. >> Mhm. >> The instigator for that choice point is. they see some evidence that their AI. might be misaligned and plotting against. them. Right? So, if you actually see that. evidence. then it's like. oh gosh, uh. maybe we shouldn't put it in charge of. everything and let it rip, you know?

Because that evidence is staring us. right in the face that it's this. untrustworthy, you know? But if they. don't see that sort of very clear. evidence, then. I think they're going to convince. themselves that they need to keep going, you know? But maybe they will see very. clear evidence like that. In which case, even if we don't have regulation, they. might just sort of voluntarily stop. Um so, that's the second ray of hope. Like overall, I don't think that we're. like definitely doomed, you know? Like. [snorts] I said 70% but like. I could see it working out pretty well. as well. >> Hm.

What about uh jobs? >> Yeah. So, I think I think I'm excited to at. some point get into the new thing which. is the more optimistic. >> [clears throat]. >> positive vision. Uh and that will have a lot to say about. this. Because in the in the in the prediction, you know, in the year 2027, by the time. everyone loses their jobs, there are. worse things happening. Or like it's. it's kind of like too late by that. point. Um but yes, like once if I mean. just just think about it. If the. companies do manage to build. superintelligence, then by definition,

they're going to be able to take almost. all the jobs or all the jobs, right? Cuz. it's better, faster, and cheaper than. the best humans at everything. >> And that's I mean, the timeline is by. the end of sort of 2030, you reckon you. think superintelligence might arrive. I'm trying to think about when we could. start to see job displacement in the. economy. >> We're already starting to see a little. bit of it now, but not very much. >> Why? >> Um cuz the AIs aren't good enough yet. Like they're they're they're they're. impressive, but they're not like. they're not just a a drop-in replacement. for a human worker in almost any field.

>> And do you think that will be sudden? >> I think it'll be sudden because of the. intelligence explosion dynamics or. recursive self-improvement dynamics. So, you could imagine a different world. where. it's gradual. >> Mhm. >> And and this [clears throat] is this is. maybe how it is in a lot of science. fiction is, you know, the AIs gradually get better. at a bunch of things and. you know, they gradually automate like. this one industry like pharma, then they. automate like. you know, steering drones, then they. automate like driving cars or something. like that. Um.

but what's different about the real. world is that the companies have. converged on this strategy of automating. themselves first. You know, automating the AI research. process. And so, if they are allowed to continue with the. strategy, we're not going to see like, you know, the robot taxis and like the. plumber robots and. you know, the lawyer AIs. We're not. going to see that sort of like broad. diffusion of AI into the economy. happening first because that's not what.

they're focusing on first. They're. focusing on automating themselves, automating their own research so that. they can do everything that they're. doing faster. And they want that to sort of get going. and get to. you know, very high levels of intelligence, very. high levels of general intelligence um. and then deploy more out to the economy. economy. Right? So, by the time it's actually coming for. like all these different jobs, they will have had fully autonomous AI. research happening for months, maybe.

years, you know? And that means that like the AIs will be. vastly superhuman at AI research and. probably also vastly superhuman at lots. of other things just as a side effect, you know? If you're wondering what this looks. like, well, we wrote about what it looks like. It's. sort of like this this wave smashing. through the economy after they do the. intelligence explosion internally. >> What I'm hearing there is that because. the AI will be able to improve itself. and train itself, it'll be getting. better at everything at once and then. it'll be released at kind of once.

Is that accurate? >> it's it's not it's not even exactly that. because even if it's mostly just getting. better at the things that it's doing. like research, that'll have some spillover effects. to other skills as well. And then when it turns to the focusing. on the those other skills, it'll be able. to do them very fast. >> What jobs remain in such a scenario, do. you think? >> I think that's actually a political. question, not a technical question. >> Because. >> Because on a technical level, all the. jobs can be done by the AIs. if they've reached that level.

And so, it's a question of what jobs are. allowed. for them to do. >> And what kind of jobs wouldn't be. allowed, do you think? >> That depends on who's in charge. So, there'd be some sort of political. conversation about like what we're going. to allow and disallow. >> I mean, in this scenario, the humans are. still controlling them, the AIs. >> Depends on what you mean by control, right? So, there's like. there's do the AIs actually have the. goals and values that you want them to. have, and are they going to robustly. do that and behave as intended into the. future? And then there's like are they. obeying your orders for now? >> Are they obeying the orders is really.

what I'm saying. >> Yeah. So, like even in AI 24/7 in the. scenario where the AIs take over and. kill everyone, there's a period of like. several years where they're still. obeying orders, and they're, you know, taking some jobs but not other jobs, and. they're helping to make better weapons. that the US government can use to like. do its arms race with China and so. forth. And that's why they're able to. get so much power so quickly is because. the governments and the corporations and. so forth trust them and is deliberately.

deploying them into all of these. positions because it thinks that things. are fine. But because these things are neural. nets, you can't just like look inside and see. what it's really thinking. You can't. really tell. >> I think this is a really important point. because unlike software where we can. look at the code and see what's going. on, theoretically, with AI you're saying. that we don't know what why it's making. the decisions that it's making cuz we. can't get inside. >> One note of optimism is that it doesn't. necessarily have to be that way. Like. there's a a subfield of machine learning. called mechanistic interpretability, and.

a a broader subfield called. interpretability more generally that's. trying to solve that problem and trying. to take these these trained artificial. neural nets and piece [snorts] them. apart and understand. like how the information is flowing and. how the decisions are being made, so to. speak. Um the problem is just it's a. very inherently hard problem. If you. have 10 trillion connections to look at, you can look at any particular group of. them and be like, "Okay, so this is how. like this particular connection works.". But like how do you get a sense of the. whole, you know? How do you get a sense. of like.

what's happening at a high level? And. the answer is, "Well, it might be. impossible." But people are working on. it and they are making progress, and. if they can make enough progress, then. we're in a very different and much. brighter world. I think that it would be. much less likely for us to get into. those loss of control scenarios if we. could just actually see what our AIs. were thinking and why and how at any. given time. Right? >> Yeah. >> So, we would still have the other. problems to worry about, but at least we. could mostly solve that one. >> It is pretty crazy to think that we're. building a technology, a brain that we. don't understand.

>> Yeah, it's pretty crazy. I mean, it's. one of those things where like. >> In a movie, like a sci-fi movie, a bunch. of scientists sit around this big brain. and they're all just like they're. they're making it more they're feeding. it. >> Yeah. >> And they don't really know what the. it is. >> Yeah, I mean, it's it's kind of just. like obviously a dangerous thing to be. doing. >> Yeah. >> Um but we're doing it anyway because of. this history of how the field has. developed in the last 10 years where. you know, people were like, "Oh wow, yeah, that's obviously dangerous. Oh no, what if someone else did it and did a. bad job of it? Therefore, we should do. it and do a good job of it and now. they're in this race where.

where they're racing each other and. they're also under all sorts of. political pressure to like pretend that. it's not as bad as it seems because. they don't want to like. anger their investors, they don't want. to anger the White House. >> One of the the key questions we had from. our audience was which and I kind of. asked you this in part, but which jobs. are genuinely likely to survive AI and. what skills should people {slash}. students focus on over the next 10. years? >> That's kind of like. like imagine if you were someone living. in Mexico. in like 1500 and then you hear that like.

the conquistadors are coming. You could be asking yourself like, "Okay, well, what sort of job should I. be switching to to like survive this. transition?". But like, you have a lot more to worry. about besides that. But yes, I think I. would say that like if we managed to. avoid the loss of control problem. and we end up with humans still. in charge of the AIs and humans can like. say what the AIs goals and values are. supposed to be even as they become much. smarter than humans and even as they run. the whole economy.

then probably there will be regulation. that protects some areas. and you can try to guess at what those. areas might be. Maybe stuff that's more. like. like like judges potentially. >> What about podcasters? Be honest. >> Probably not podcasters, I think. Um. stuff like. you know, being a nanny. maybe, right? Like I think that even if. there's a robot nanny that's like really. really good, I think a bunch of people. might prefer to have an actual human. because they might be creeped out by the. idea of a really good robot nanny. So,

you can sort of you can sort of reason. like that. There's also like. stuff that might be legally protected. Like maybe judges, for example, like are. going to be legally required to be. humans and not robots. >> Some people say though there's going to. be so many jobs created that we can't. foresee right now like there was in the. industrial revolution or the internet. boom or whatever. >> The problem with that is that. um past technological advancements have. been more narrow. They've like automated. some things but not everything. But we are talking about a hypothetical. future situation in which everything.

gets automated. So, there isn't any new. job that you could do that AI couldn't. also do. Except if it's like protected by. regulation or something. That's that's. that's also a thing. But so like for. example, right now there's this sort of. like cycle where. you know. the AI's learn to do a certain thing. like write copy or like draft code or. like debug something. And then humans who used to do that. thing switch to managing AIs or switch.

to doing the other stuff that the AIs. can't do. And that's why there's been this dynamic. historically of. you know, new jobs opening up and people. flooding to them. But. if it gets to the point where the AIs. can do everything that humans can do and. better and faster and cheaper, then. whatever that new job is that you might. have switched to, that the AIs can. switch to that too and they'll already. be be better at it than you. >> Because we haven't seen widespread. unemployment yet in the economy, do you. think people are getting a little bit. complacent because what I'm seeing on my. timeline is a lot of people saying I. told you so, I told you everything would. be fine. And when you look at the the US.

unemployment rate, currently the it's. flat to slightly down. If you look at. the UK, it is up. The trend is up. compared to last year. We're at about 5%. unemployment. The US is at 4.2%. unemployment. >> Yeah. Basically, nobody has said that. there would be mass unemployment by now. Or at least we didn't say that. You. know, and we were historically one of. the more bullish people on AI progress. In AI 2027, because of the dynamics that. we just described, the mass unemployment. doesn't happen until 2028 or 2029 after. they already have superintelligence.

Because, again, the companies aren't. trying to cause mass unemployment as. step one. That's like step three after. you know, it's like step one, automate. themselves. Step two, have this recursive self-improvement to. get to superintelligence. Step three, expand out into the economy and automate. everything. And so, this is really unfortunate from. humanity's perspective, because one. might have hoped that. if there was this broad wave of. automation going through the economy, people would sit up and pay attention. and think about where all this is headed.

and demand good regulations from the. government. But, that's not actually what the strategy of. the companies are taking. You know, they're going to be getting the. superintelligence first and then doing. the broad wave of automation, which. means that by the time they're actually. doing all of that, uh well, it's already going to be moving. very fast and the AIs will already be. very powerful. >> In your 2027 report, so you wrote that. in 2025, but it is called AI 2027, you. said that in mid-2025 we'd have the. autonomous employee, which is sort of. like AI agents taking instructions over.

Slack or Teams. That happened. I've actually got an AI. agent in my WhatsApp I can talk to. Of. course, you've got Claude by exploded, obviously, around the world. And and. now, um you know, Claude have talked. about uh their new Slack integration. But, lots of people are using agents. now. And that happened, I'd say for us. at the We really sort of caught onto it. at the the start of 2026. You also said by 2026 companies begin. replacing entire corporate departments. with AI agent subscriptions. 2027, the. final job. AI automates the job of the.

human AI researchers themselves and. begins the machine learning research to. upgrade and build the next generation of. AIs. >> Yeah, yeah. So, again, timelines. We are uncertain about how long it will. take to achieve these milestones. In. this scenario, they happen at those. times, but. by the time we had actually published. this scenario, our timelines had shifted. back a little bit. Specifically, mine. had. So, like. my 50% mark was 2028. >> Mhm. >> For that for the full automation of AI. research milestone, not 2027. Uh.

and then other people on my team had. more like 2030, 2031, things like that. So, I I I kind of want to like. maybe try to illustrate this with the. you know we have like this probability. distribution. It's like a. smeared out probability mass. And like. the 50% mark is this particular year, but there's like a lot of possibility. that it happens. >> Later. >> years earlier or years later, right? >> Got you. What is this AI 2040? >> So, AI 2027 was our best guess. prediction as to how things would. actually go. >> Yeah. >> AI 2040 plan A is our recommendation for.

how things should go. So, we called it. AI 2040 because in this scenario, uh. they build superintelligence in 2040. instead of much sooner because they. delay things. >> Why do they delay things? >> To manage the risks and make sure that. power is distributed equitably. They basically like. regulate AI development so that it still. continues, but at a slower, more. reasonable pace uh in a more transparent. and safe way. and spread out over more countries and. companies. And as a result, they get to.

superintelligence in 2040 instead of in. say 2030. And then we call it plan A because. well, it's our recommendation. Like. we've we've come up with a plan for. what government should do. And uh. the scenario is an illustration of what. it might look like to implement that. plan. In a similar way to how AI 2027 is. kind of an an illustration of what it. would might look like. to do with the companies are currently. planning to do. If that makes sense. >> And is this wishful thinking or is this. what you think is going to happen?

>> No, it's definitely not what we think is. going to happen. >> It's not what you think is going to. happen? >> No, no, what we think is going to happen. is still. something more like this, right? We we. don't expect the world to listen to us, right? This is our recommendation, but. we we we hope that that people do. something like this and we think it's. possible, but it's not our like. prediction for what's going to happen by. default, you know. >> So, I do want to run through the plans, the potential plans, and also plan A, but um just to close off on how things. might look after the year cuz I think I. wanted to touch on robotics, too, and.

I've got this graph here which talks. about share of labor output. >> Yes. >> Yeah. >> Um which I found to be quite striking. I've been sat here wondering as an. employer who employs hundreds and. hundreds of people. when when all this stuff is going to. happen. And you know, we're still hiring. more people as things stand. There are. some roles where our consideration is. changing, shifting considerably. And I'd have to say that, you know, we're probably in the phase where our. teams are AI-powered and they're using. agents to do some of their work now. But I'm wondering as an employer like. when is it.

when does this happen? >> Yeah, great question. So, if we could. maybe zoom in on this a little bit. >> it on the screen. >> So, this is in the AI 2040 plan A. scenario. And notably in that scenario, there's significant regulation. introduced in 2029 that slows down the. pace of AI development. In the scenario, they do that sort of at. the last moment. So, in the scenario, if. they hadn't done that, then it was about. to take off similar to how it does in. the AI 2027. Um but as you can see like in the. scenario, there's still. a bunch of jobs.

at the point that they implement it. And. this gets back to what I was saying. earlier is that if you wait until most. people have lost their jobs. to regulate the AI companies, that's. already too late because. they will probably already have super. intelligent AI by then because their. strategy is to first get super. intelligent AI and then do all that. stuff. >> think you say that it would collapse the. economy and cause even more harm to. suddenly regulate something that all of. us and all of our lives were then at. that point relying on. >> Oh, but it's a risk well worth taking. I. mean, we It's true that right now a lot.

of people use AI for a lot of things, but like if we could somehow slow or. halt AI development now to set up a. better way to do it, that would be well. worth it. Um even though there would be. significant costs. >> But you can't over here, right? Can you? At this point where AI and robotics are. doing most of the labor output. >> That's right. But in but in but in in. this scenario, in the AI 2040 Plan A. scenario, they put in the regulations in. 2029. And then they slowly and carefully. develop AI. in a way that avoids all the problems, which we can get into in a little bit. And so eventually, yes, eventually the.

AIs take the jobs. Eventually. basically the whole economy is run by. AIs and robots, but it it happens. gradually over the course of. the 2030s instead of happening in this. sort of crazy shock, you know, a year later. Right? Because in this scenario, they. don't let the companies. recursively self-improve and get to. super intelligence as fast as possible. Instead, they regulate AI development so. that the core capabilities of the AIs. are improving at a more reasonable pace. and also in a more transparent way so.

that the scientific community can see. what's going on and help make it safe. >> But it's. I guess I noticed here that in both your. scenarios, eventually AI and robotics do. pretty much all the jobs. >> Yes. >> So you kind of side there with Elon when. Elon says that working will be a choice. >> Uh. >> Because I mean we're going to have to. >> I mean, if [laughter] it by definition. if it can do all the things, then. it can do all the things. I think that.

there's a question of like should we. allow there to be AIs that can do all. the things, right? Some people think. that the answer is no and we should just. shut it all down and prevent these types. of AIs from being created in the first. place. And we're actually kind of. sympathetic to that. We we have our. Should we bring out the plans diagram? >> Yeah. >> Thanks. Yeah. So, our scenario is called AI 2040 plan A. It's a scenario in which they slow down. AI development to make a super. intelligence happen in 2040 instead of. earlier. And plan A is our.

recommendation. So, this is sort of. illustrating our recommendation. But, for comparison, we made like mini. scenarios illustrating different. alternative plans, which we call plan S, plan B, plan C, and plan D. Plan D is basically. the same thing that happens in AI 2027. Like, the race continues. There's very. little regulation. Um you can read about that in AI 2027. Plan C also very similar to what happens. in the slow down ending of AI 2027 where. they solve the alignment problems. So, in that ending, they like slow down a little bit,

pivot more resources to AI alignment and. AI safety research, get lucky and succeed, and now they have. aligned AIs, and then they speed up again and take. all the jobs and beat China and all. those things. Plan B is. it's kind of like plan C in that. well, basically in plan B, you're. uh being more aggressive towards China. and you're like. taking actions to sabotage or cyber. attack them to like keep them behind so.

that you have more breathing room to to. solve the alignment problems yourself. Plan A is our recommendation. It's uh. domestic regulation and then an. international deal. to continue building AI, but in a much. better way. Plan S is shut it all down. If you want to have a future where. there aren't AIs running around that can. do everything better and faster than. humans, you kind of want something like. plan S. What What do you want? Plan A is our recommendation. I think that I'm sympathetic to plan S,

but for reasons we explained, we. recommend plan A instead. >> And And do you think is most probable? If you're being honest? >> Plan D. >> Which is that they just. >> yeah, 24/7 type of thing where they keep. racing. They don't really slow down. significantly. Um. and uh. things happen extremely fast. The diagram sort of explains like. roughly the reasoning behind this, too. So, like there's this high-level thing. of like. do you want to keep racing. as fast as possible to make the AI. smarter and smarter, to put them in. charge of more things so that we can.

beat China? You know, if you're happy with that, then. you get down and it says variation of. happens here. If you are worried about that, well. you get to something like this. There's more different options besides. these, but this is kind of like the ones. that we could compress onto a screen. >> Do you have children? >> Yeah, I have two children. It's kind of sad. Like. I think that one way or another this. will probably all be over by the time.

they're old enough to. join the workforce. So, I don't think they'll ever join the. workforce. >> When you say this will be all over by. the time they join the What do you mean. by this will be all over? >> So, these milestones that I described, like AIs automating the AI research, AIs. getting super intelligent. Um. AIs then exploding onto the economy, taking the jobs, building robot. factories to build more robots to build. more factories, etc. GDP starting to. go vertical.

That sort of thing is what I mean. Like. all of those events transpiring. Maybe there's like you know, 10, 20%. chance or something that. hits the wall. and and none of this comes to pass even. if you don't do anything. >> How old is your oldest? >> Six. >> Six. Boy or girl? >> Girl. >> Girl. So, your daughter comes to you and. says, "Dad, what should I um what should. I study in school?". >> I mean, again, like if these radical. transformations happen, then. the world will just look completely. different and.

what sort of jobs you set yourself up. for basically, won't matter that much, probably. I would say um that the thing. to do is. well, A, try to make it actually go. well. Like, if you can exert any. influence at all on history and how this. all develops, you should be trying very. hard to steer the future in better. directions. And then separately from that, on a. personal level, you should focus on. well, being a good person and doing things. that are sort of good in their for their. own sake, rather than good because. they'll set you up for later employment.

because that later employment is going. to be very uncertain um basically. >> Elon talks about this age of abundance. we're heading towards. Age of abundance. >> There'll definitely be abundance. The question is who controls the. abundance? And what do they do with it? Right? Are the AIs controlled by anyone? Or are they doing their own thing? And then if they are controlled by. people, who controls them? And what do. they do? And what's the sort of like. political structure governing how they. make those decisions? >> I think it was Geoffrey Hinton that said. to me, he said there's no example in.

nature where a more intelligent species. is has less control than a less. intelligent species. Thus saying that. we're quite arrogant to think that in a. world where there's this artificial. brain that's a gazillion times the size. of mine, that I'm going to give it. orders. >> Yeah. I mean, that that's the thing is I. I think it's like. that should be our default assumption. Is that like, well, there's these. brains, we can't see exactly what. they're thinking. We're going to make.

them smarter than us and put them in. charge of everything. >> And then we're going to give them. bodies. >> Yeah. And then they're going to be. autonomously building new factories and. so forth. And like, how is this supposed. to end well again? Like, isn't this just. exactly like us picking a new species. that's then going to outcompete us when. it doesn't need us anymore? Like, I. think that is just the default. trajectory. Now, there's a whole. argument we can get into about like ways. that we could get off of that default. trajectory. So, for example, there's. research into interpretability that I. described previously. And if that. research bears fruit, then you will be.

able to actually see what they're. thinking. And then that would be an. excellent tool for shaping them and. controlling them and making sure that. they do what we want, right? There's. other sorts of um. AI alignment research agendas that are. making progress. And if enough of those. agendas succeed sufficiently, we can. avoid this problem. Of course, also. there's the regulatory side, too, where. like part of what makes this difficult. is that we're building these AIs in race. conditions, you know? Like the the. companies are secretive about their. recipes for making these AIs because. it's secrets that they want to protect.

so that other people can't copy them. And so a lot of this is happening, you. know, behind closed doors. Only a few. people can really see. the recipes that they're using to train. these AIs and and so forth. And then. oftentimes when the AIs. behave in unexpected ways or even just. like blatantly misaligned ways, sometimes that information doesn't. really flow out to the public because. the companies are not really. incentivized to tell everyone about how. they messed up and how their AI is evil. It's just not very conducive to. scientific progress on these issues. If. the regulatory system was different, then perhaps we could be in a better.

situation, make faster progress. Also, of course, we wouldn't be planning to. put these AIs in charge of everything as. fast as possible. And we wouldn't be. planning to like let them self-improve, you know? Like the these are choices. that we could not make, you know? >> I don't speak Vietnamese, but this show. can because of AI video technology from. our sponsor, HeyGen. I get messages. every single week from those of you. listening to the Current CEO all around. the world, and you express how much. impact it's had on you and your life. And if that's true, then those.

conversations shouldn't only reach. people in English. HeyGen can take one. recording of me and deliver in any. language while keeping my voice, timing, and expressions intact. But you don't. need a studio like this to make it work. for you. Record 15 seconds of yourself. and get an AI avatar that delivers. studio-quality video in over 175. languages. We're up to 20 languages now, and we're not the only ones using it. HeyGen is already used by 30 million. people, including 85% of the Fortune.

100. Whether you're building an audience. on social media, launching an online. course, or rolling out training across. your team, check out HeyGen now. Your. first three videos are totally free at. heygen.com/doac. That's h e y g e n dot com slash d o a. c. See you there. >> Ilya was As you said, he was one of the. leaders at OpenAI, and he left and he. started his own company now, Safe. Superintelligence. Very curious name of a company, Safe. Superintelligence, after leaving OpenAI. Did you ever get to work with him?

>> Uh I wasn't directly working with him. I. had a couple chats with him. >> Do you think he's he's genuinely. concerned as well? >> I think he is, but I think it's I think. he's similar to these other CEOs, where. I mean, just think about the sort of. incentives that they're under, right? Like. they can sort of see the problem, and then they can. be like, okay, but like if I don't if I. stop, if I quit my job, and or do. something else, that's not going to solve the problem, cuz the other CEOs are going to keep. going.

And even if all of us didn't go, then. maybe China would keep going. So, like, man, it seems like this is just going to. happen one way or another, whether I do. anything about it or not. I guess I should be involved, you know, and like maybe I can make it go well, and at any rate, like I don't want to be. out in the cold while these other people. I don't trust are in charge of. everything. So, they all sort of like. reason through all of this and then. convince themselves that like the thing. to do is for them. >> to build their AI. >> build it and to do it better. And I. think Ilya's just the latest example of. this. Elon's another example. Dario's. another example. You know, arguably OpenAI at the.

beginning, Sam was an example, although. like Elon and Dario were at OpenAI early. on, so. >> What do you think they should all do. then? >> So, I think what should happen is some. sort of international regulation, or at. least domestic regulation, similar to. what we described in plan A. >> Okay, so talk me through plan A. >> Yeah. So, in this scenario. AI takes longer to the to get to. recursive self-improvement and full. automation of AI research than it does. in 2027. We figured that we should try. to illustrate like a range of different. possibilities because we do have those. sort of uncertainty intervals. So, we.

chose 2030 as the. moment when full automation would. finally be achieved and things would. really kick off. And then working backwards from that. when's the last moment you could really. have good regulation? 2029. So, in this. scenario. AI progress slows down a little bit. naturally and the AI companies keep keep. racing, but they don't quite succeed in. automating uh themselves in 2027 or in. 2028 or in 2029, but they're getting. really close and they're going to do it. in 2030.

And then in 2029, the government steps. in and regulates them. What regulations. do they do? Well, they basically just. shut it down temporarily. >> Can I ask um. how does the elections overlay with your. time frames here? Because there's going. to be a big election, isn't there in. 2028? And it seems now that sentiment has. really really turned against AI in in. sort of in the general public and that. it will be one of the big ticket items. on the on the ballot. >> We think that it'll be maybe the most. important issue in the presidential. election in 2028. Um I think a lot of.

people most people will be quite. concerned about where things are headed. and that's part of why we we chose. to depict things the way they were doing. in this scenario because that helps. explain why they might do this sort of. regulation in 2029 is that the voters. have been demanding it and the. presidential candidates have been. promising it. >> And in this scenario and then 2027, would the general public have felt the. consequences of AI much more severely. than they have now by then? >> Yes. Although still even in 2029 in this. scenario, they still mostly have the. jobs as as depicted here, right? So, in. in 2029 in this scenario, lots of jobs.

now involve managing AI agents. You you mentioned you have an AI agent, right? Well, in 2029 in this scenario, the AI agents will be much better. Still. though, not enough to just completely do. everything. You know, that was the sort. of thing that would come in 2030. in this in this timeline. Again, we're. uncertain about timelines. Things could go faster than depicted in. this scenario, and in fact, I think. things probably will go a bit faster. than depicted in this scenario, but. we're uncertain. We already did the very. fast timeline scenario, so now we're. doing the slower timeline scenario. But, maybe we should talk about the.

high-level goals. So, they want to have AI continue, but in a. slower pace so that they can make it. safe. >> The politicians, you know, the president. and the people who voted for the. president and, you know, the heads of. other governments and so forth. So, goal. one, slow things down. Um goal two, make it more transparent. so that the scientific community can. catch up to this stuff and make more. progress. And also, so that we don't. have to take the company's word for it. when they say that their systems are. safe and when they say that they. haven't, you know,

put in any biases into their systems, for example. That's a constitutional. power issue. We also want to avoid a. situation where there's an intense. concentration of power. So, in addition. to these. the transparency and the slowdown, we actually think it's actively good for. there to be multiple AI companies. across multiple different countries that. have similar levels of very advanced AI. capability and for there to be like. broad diffusion of AI into society. rather than, you know, a single mega. project that has all the best AIs, for. example. And the another thing about.

that is you kind of get that by default. if you do the first two things. If you. slow it down and if you make it more. transparent, then that means there's. breathing room. for other projects to sort of catch up, right? And the transparency just like. literally helps them catch up because. then they can like copy. copy some of the ideas. And then I think. the fourth thing would be reversibility. So, in what follows in the scenario, we. are going to be building up a lot of. data centers, a lot of robots. We're. going to be transforming the world at a. at a sort of like slower pace, though. still a very fast pace, but slower. And.

if things go wrong and the deal breaks. down and everyone starts racing each. other again to get to super intelligence. as fast as possible. That would be very scary. And so, the. fourth principle is basically build the. new data centers in such a way that if. everything. breaks down and everyone starts racing. again, the newly built data centers get. destroyed so that we're sort of back to. square one again instead of in an even. worse race where there's even more AIs. and robots and compute everywhere. Um. So, I can sort of walk you through the.

timeline if you're interested. Sure. Or. the president talks to China, talks to. the leaders of a bunch of other. countries. and says. we're going to basically. halt AI development until we can figure. out a a plan for how to do it in the way. in the ways that achieve these goals. So, they basically send inspectors to. each other's data centers. Like Chinese. inspectors come to US data centers, US. inspectors go to Chinese data centers. and verify that they are doing inference. and not training. Developing new AIs, that's that involves training them. But,

just taking existing AIs and using them. to serve customers, that's called. inference. And so, the sort of like solution they. come up with here in this scenario is. we'll allow them to keep doing inference. but not training for now until we can. get the new training data centers set. up. So, they retrofit the existing data. centers to serve inference. People can. still keep talking to their AI agents. but they're going to stop getting better. and better. for like 6 months to a year while they. build the new data centers that are. going to be the transparent data.

centers. And that's where the training's. going to happen. Once they get those new data centers set. up in 2030, then AI research continues. This is a. bit spicy. We advocate for total. research transparency, which means that. on the training data centers that are. training the new models, they basically have to publish. everything. Which means you get to see all the. details of the recipes for training. these models. You get to see the. architectures, etc. We think that's sort. of open science is really important for. solving the alignment problem fast. enough because you don't want to have to. sort of biased companies making the.

decisions about whether the AIs are. safe. Um and we also think it's. important for just good regulations more. generally because right now most of the. expertise in the world on AI is sort of. concentrated in Silicon Valley and the. the governments in particular kind of. are don't really understand AI that well. and imagine an alternative instead of. total research transparency you had like. an auditor system where the government. says here are some rules for how to make. the AI safe. and then we're going to have like an. agency that like goes into the companies.

and ask them questions and tries to make. sure that they're following the rules. That creates this sort of adversarial. dynamic where the company is. incentivized to like fool the the. regulator, you [clears throat] know, and. and also if they if they discover some. new problem that's not even on the. government's radar. they might be incentivized to like not. tell the government about it, right? So. if you have the total transparency it. helps the government make better. decisions faster. >> But it kills that competitive advantage. >> Yes. Prophetic's not going to like this, you know, OpenAI's not going to like. this. This would be. probably bad for the valuations. I don't.

think it would kill them completely but. it means that it would commoditize more, right? So it means that there'd be like. a bunch of AI companies that would catch. up to the frontier, they would train AIs. that are like roughly similar, roughly. equivalent. They could still make money. by doing that and then selling their AIs. but they wouldn't have a monopoly, they. wouldn't have anything close to a. monopoly which I think is good for. humanity although it's bad for the. bottom line of those particular. companies. Notably it's good for the. bottom line of lots of other companies. Like if you're a company that's behind. and you don't you're not Anthropic or.

you're not OpenAI then you would love. this because this helps you catch up, you know, or this this helps you to like. um capture more of the value from the. chips you're selling for example or from. the like downstream product that you're. making. >> And by 2031 then you have 1/5 of all. cognitive labor done by AI. >> Yeah, so what's happening here is that. we're imagining that the government of. the United States and the government of. these other countries that are involved. in this agreement that are sort of. implementing similar regulations. um they don't have to be exactly the. same, uh, but that's another thing that's nice.

about the transparency is that if you. have this sort of transparency, then. if two governments are. implementing different regulations, like. if one of them is like. telling their companies to go slower or. like banning more stuff than the other. one is, they can both see. >> Yeah. >> like, "Oh, you're letting them do that. sort of thing? And you're not? Like, maybe we should. let them do this, too, you know?" So, it. helps to sort of naturally equalize the. regulations to some extent without. having there to be a central power that. just gets to make regulations for. everybody. >> Mhm.

>> So, anyhow, we're imagining that when. they when they get this transparency set. up, they basically agree to ban the. dangerous stuff, to allow the. not-so-dangerous stuff, and there's a. constant ongoing conversation about. like, "Well, what's dangerous and what's. not? What should we ban? What should we. allow? What about this country? What. about that country?" That conversation. evolves over time, but the gist of it. is, at least if they do it the way that. we recommend it, is that they don't do. an intelligence explosion. They don't. let the AIs, you know, autonomously. self-improve. Instead, they slowly and carefully scale up the. AIs that they currently have, and invest.

lots into finding ways to make them more. interpretable, uh, to make them more easy to control, to understand better how they work, and. so forth. The result is that AI progress. continues, but it's. not quite as fast, and it's much, much, much safer and more. transparent. >> But still through these, you know, are. we seeing job disruptions? >> continuing cuz they are building more. data centers, right? Like, this whole. time, they're building more and more. data centers, more and more chips, and. they're continuing to like. make there be a a larger and larger. population of AIs, so to speak, and that.

causes this huge transformation over the. course of the 2030s. So, the big thing. that we sort of want people to take away. is that even if you heavily restrict AI. progress, you still get this sort of crazy. transformation. Yeah, in this scenario, they basically. allow progress to continue, but at a. slower, more safe pace here in 2030, and then it as a result, it takes until. 2035. to get to top expert level AI. So, remember they were on track to do that. in 2030, but then sort of at the last.

moment they stopped. But because they. were sort of so close to the last. moment, that means that like they can. sort of get there pretty soon if they. want to, and it's just a matter of like. how long they they allow it to go, right? So, they sort of they sort of. slow it down, and spread it out, leisurely arrive at this level after 5. years. By this point they've built up. massive amounts of data centers. everywhere. So, it's not just that the. AIs are smarter and able to do all the. things that humans can do, but also. there's a lot more of them. And there's. a lot of robots and so forth. So, by. this by this point you kind of have the.

economy that a lot of people would have. imagined with AGI, where there's AIs, there's lots of them, they're able to do. all sorts of jobs, there's robots, there's lots of them, they're able to do. all sorts of physical work, and. basically the economy is being run by. these machines. >> So, in 20. 31, you you have the 1/5 of all. cognitive labor done by AI. In 2023, you. have 60 million AIs running at 100x. speed. In 2033, there's cash dividends to all Americans. >> Mhm.

>> Um I've got to. explain explain this to me. >> Yeah, so if the AIs are going to be. taking people's jobs, then it's very. important that people not starve to. death, and still have money. And if. companies are going to be using AIs and. robots to take all these jobs, then that. means that there needs to be some sort. of taxation scheme, or something, to. like. make sure that people still have a a. slice of that pie. Mhm. The pie is going. to grow huge, but you still need to. actually give people a slice of the pie. And our proposal for how to do that, we.

call it the citizens dividend, basically. people have shares in a agency that. sells permits to the robot companies, and to the compute companies, and makes profit from selling those. permits, and then those are people have. shares in that entity. It starts off. small. It starts off something like. $25,000 per person. Uh and then by the end, it's something. like $10 million. per citizen. >> per person? >> Per person per year. >> Factoring in inflation, like what you.

mean? >> in inflation. >> So, we're going to be. multi-millionaires. >> Yes, if this happens, which it probably. won't, but if it happens, this is where. it will go. And again, this is the thing. I want to emphasize is that if you get. to the point where your AIs are close to. being able to do. all the research, and then you sort of. pause and slow down, that means that like you still have a. lot of transformation ahead of you. because if you allow those AIs to like. still proceed slowly and like start to. automate various jobs and so forth, after some years, they will in fact have. done that. And. they will have, you know, built huge.

amounts of new data centers, huge. amounts of new chip fabs, huge amounts. of new robots, robot factories, etc. You know, we're not sure obviously how. fast this will go exactly, but we've. thought about it a lot and we have our. our guesses and this is sort of like our. median guess. >> What does this mean, 2037? The. apocalyptic arrival of truth on Earth? >> Yeah, so like. this is the point where we say they get. to top expert level AI. So, it's not super intelligence in the sense. that it's not like vastly smarter than. humans at things because they. deliberately pause it at the level of.

top experts. So, so here they're going. slow. Here they've just actually. stopped. But they stopped at a point where the. AIs are just actually really good at. everything. So, kind of they've. definitely got AGI, maybe they got like. weak super intelligence. Because they have so many these AIs and. because they think faster than humans, you know, they just run much faster, that's going to transform society. dramatically. So, we talk about some of the ways in which. it transforms society. Like this is sort. of life after work. We talk about what. it would be like to be living on your. citizens citizens dividend and not have.

a job anymore in this sort of world. Um. here we talk about all the scientific. changes and all the social changes that. would come from all of the. intellectual progress and activity that. would be generated by all of these AIs. So, for example, here is things like cancer. cures and like, you know, people living. in apartments that were built by robots. 2 years ago. >> Mhm. >> Providing again we stop in 2029. >> Yeah. >> And providing, I mean, a conservative. This is a conservative time frame. >> Yeah, like unfortunately, I actually.

think that things will happen faster. than this by default and that if we. don't slow down, things will happen much. faster than this. Once you get to the. point where you've got, you know, a. billion AIs running day and night and. they're each better than the best humans. at everything and so they're doing a lot. of science, they're doing a lot of. talking to each other, they're doing a. lot of thinking, everyone's constantly. talking to their AI assistants and so. forth. There's going to be a lot of scientific. progress. There's going to be a lot of. changes to politics, to ideologies. It's. going to be very disruptive and crazy.

and we get into some of the ways in. which it is. uh later, basically. >> I I'm still not super clear on what this. means, the apocalyptic arrival of truth. on Earth. It's just It's just because there's so. many eyes AIs that are so smart that. they're uncovering making new. discoveries in sciences. >> Let me give you an example, lie. detectors. >> Yeah. >> So, that's an example of a a technology that. might be invented. >> Yeah. >> You know, right now we don't have good. lie detectors, we have very bad lie. detectors that like sort of work but. don't don't fully work. But once you've. had these top expert level AIs thinking.

for many years at you know, 100x human. speed and there's billions of them and. they have access to robot factories to. do research and stuff, they'll probably invent a ton of. technologies. Maybe they'll invent lie. detectors that actually work on real. humans. That'll have big social effects, right? Imagine a presidential candidate who's. like, "Those allegations are false. and to prove them, I will go under a lie. detector and say that they're false.". >> I was just thinking about the whole like. justice system and. how that would be overturned. Um in. fact, you could, you know, theoretically. walk down the street and be.

Yeah. >> It's both. terrifying and exciting. One thing that we talk about in this. sec- in this section like the invention. of lie detectors could be really bad. Like it could be that it enables a new. form of totalitarianism where the. powerful people, you know, the CEOs and. the politicians. force the people under them to go under. lie detectors and say like yes, I'm. loyal to the dear leader. I would never. do anything against the dear leader, right? >> you're lying then you're in. >> And then if you're lying you get fired, right? So like there's there's a ton of. like very harmful uses of lie detector. technology. There's also the good uses.

and broadly speaking I would say the. good uses are when lie detectors are. used on the powerful instead of by the. powerful. >> What's this? 2040 passing the torch to. AIs. >> Yeah, great. So. here they pause at the top expert AI. level. And the reason why they pause is. because. their safety cases aren't good enough. for going beyond that level. Um so in. the sort of regulatory systems that they. set up over the course of these years, roughly speaking the way they would work. is when you're making a new AI and then. when you're trying to deploy the AI into.

something, you have to have some sort of. safety case explaining like. what your intentions are and like why. you think it's going to work the way. that you want it to work. And in. particular why the AI is going to like. do as it's told, for example, and why. nothing super terrible's going to happen. like AI takeover. It's relatively easy to make safety. cases like this when your AIs are still. not capable of automating everything. But the more powerful they get, the more. difficult it is to actually argue that. things are going to be fine because the. AIs are just more capable and they can.

they can get up to more stuff. And if. you if they're actually untrustworthy, the the possible downsides are bigger. So that's why they stop at this level is. that they they realize that if they keep. going then they might actually lose. control of everything. But at the. current level they're convinced by. safety cases that it's fine. But then. they don't want to go further. So they. stop there. And then what happens in 2040 is they've. made significant progress scientifically. including on alignment and they figured. out how to make AIs that are actually. aligned in a robust way. >> With humans?

>> With humans. So they can actually trust. those AIs and they can allow them to. become much smarter again. So, that's. why we call the whole thing AI 2040 cuz. in 2040 they sort of let off the brakes. and allow the AIs to become. significantly smarter than humans. >> I guess you know, this is a this is a. plan and this is a hope. >> Yes. >> But in reality, this is not what you. think probabilistically if you had to. >> That's right. It's important to. distinguish like this is what we. recommend. This is what we want to. happen from like this is what we.

actually think will happen by default. Now, we do think it's possible for this. to happen, but you know, that will. require a lot of people to sort of wake. up and pay more attention and advocate. for something like this to happen. So, our main scenario is mostly talking. about the policy choices made and the. broad scale effects on society. We. figured it would also be nice to. accompany this with a little mini. scenario that describes what it would. actually feel like to live through this. from an ordinary person's perspective. >> Okay. >> Um 2029, everyone's yelling at each.

other, the presidents are negotiating. something and they've paused AI, but you. still have access to the existing AIs, so it doesn't really feel that different. although it definitely is like something. exciting happening. 2031, they've. started progress again, the AIs are. really smart, more people have lost. their jobs, it's like really starting to. actually affect things, but I think. still most people have their jobs, but. their jobs are sort of transformed. So, like by 2031 it's like. most white collar jobs involve working. with AIs to a large extent or managing. teams of AIs or collaborating with them. somehow. >> And what was. >> Also, there are some things like robo. taxis that are basically just working.

Citizens dividend, you know, ideally. this would happen sooner. Like in our. scenario, they kind of do things at the. last minute. You know, so like a lot of these policy. things are like happening kind of like. just in time. Obviously, we would. recommend that you do them sooner and. and do a better job of them, too. But. so, 2033, you start getting your your. checks from your dividend. >> So, you're forecasting that there will. be a citizen's check. The your model. says it could be around 25,000 at the. start per person. >> And then it would grow as the economy. grows. >> But also as like as job displacement.

takes hold, they're going to need to to. grow that check and make sure you can. >> And that's why it's kind of the last. possible moment because if you waited to. implement this until like 2037, then. like everyone would have already lost. their jobs by the time that happens, right? >> People losing their jobs, especially if. it happens. quickly like like we see on this sort of. graph here, is going to cause lots of problems in. terms of civil unrest, social unrest, purpose, mental health, these kinds of. things theoretically. >> Yes. >> How do you think about that? >> Uh it's it's going to be rough and. hopefully we can navigate that well. We.

think that at a high level, people need. to have money. and also people need to have power. And. I think these are like somewhat. different things. It's like why are jobs. important? Well, there's a lot of. reasons why jobs are important, but I. think the main ones are. um well, it's how people get money so so. they can survive and get things that. they want by buying the things that they. want. So if people are going to be. losing their jobs, you need some other. way of people getting money. And then there's also the power thing, which is that right now people have. political power in part due to their. economic power. People can threaten to. go on strike, for example, or you know,

countries that are ruled by dictators. can't. just completely, you know, genocide an entire. subpopulation, or they can, but like. it's costly for them to do so because. then they'll have less money because. that subpopulation is contributing to. their economy and contributing tax. revenue and so forth. But if you end up. in a world where actually nobody's. contributing tax revenue revenue except. for the AI companies and the robot. companies, then you're you, the. government, are less incentivized to. care about what, you know, the common.

people think. So so. when people lose their jobs, they're not. just threatened with lack of loss of. income, they're also threatened with. loss of political power. And so we think that it's important to. like do things to push against that. >> What does that look like? How do you How. do people have power in such a world? >> Well, in democracies at least they still. have votes. >> Okay. >> So I think that it's very important for. there to be uh regulations on the use of. AI that help make. the public discourse more sane.

and more. um. actually giving the people what is in. their interest and what they want and. avoiding a sort of um opposite outcome. where. you know, the masses are easily. manipulated by AI-powered media, for. example. Or where everyone's talking all. day to their AI advisers, and the AI. advisers are like subtly steering them. away from voting for the candidate that. would. not be what the AI companies want. because the AI companies have this other.

candidate that they like better, and. they're like secretly biasing their AIs. to like steer people towards voting for. that candidate, right? So, so we want to. be in a situation where. um. people have AIs that are actually. trustworthy and that are truth-seeking. AIs, honest AIs, and that don't have any. sort of like political agendas put into. them by the AI companies or by the. government. You know, you want to avoid. a situation where the AI company where. where the government has issued some. sort of secret order that like. the AIs have to be such and such a way.

Yeah, the Department of War dispute. versus Anthropic is like a an. interesting sort of foreshadowing of. this, right? Where um Anthropic was giving. their AIs to the Department of War. Department of War wanted to use them. for certain things and was upset that. Anthropic's AIs were like. not supposed to be used for those. things. Uh the things in particular were. domestic surveillance and. uh. autonomous robots. There's going to be a lot more issues. like that coming up, and you want it to. be the case that like people know what. they're getting, and that if people are.

like spending hours a day talking to. their chatbot, that chatbot doesn't have. political biases put into it or a secret. agenda or things like that, and instead. has been trained to like give honest, true answers to things. And I think if. you can do that, it can improve the. discourse and help people to use their. votes to put even better regulations and. even better politicians in place, and so. forth. And you can sort of potentially. bootstrap this to having something where. people's power is even more secure than. it is today. >> A lot of this stuff we've we've covered. in part. So, you know, the wars and. drones and missiles, we're already.

seeing this around the world at the. moment, which is really, really. interesting. Um. and we've talked about robots. outnumbering humans as well, which is. part of this prediction. Some of the. ones down here I found to be really. curious, which is. people will be protected by AIs wherever. they go. >> Mm, yeah. In this scenario, they delay the creation of. superintelligence until 2040, and they in fact they pause in 2035, but. then they let it go after that. And then. they let the AIs become vastly. superintelligent. And we think that once the AIs are.

vastly superintelligent, the world will transform even more. radically than. what happens in the 2030s in this. scenario. So, in the 2030s in this. scenario, it's more like human level, you know, the AIs are not. they're they're doing the same sorts of. things that human experts would have. done, they're just doing it a little bit. better, a bit faster, and a lot cheaper. And there's a lot more of them. And the robots are still, you know, doing the same sorts of things that. human workers would have done. They're. just more of them, and they're cheaper. And because of exponential growth, uh.

you start with a world that looks not. that different from today in 2029, and. then by 2039, you end in a world that's. radically transformed, where everyone's. living in these like fancy new. apartments that were built by robots 2. years ago. There's like giant special. economic zones that are full of robots. and solar panels and factories producing. more robots and solar panels and. factories, and so forth. Most of the. economy is AIs and robots, and people. don't have jobs anymore. That sort of. transformation is what you get if you. pause at human level. But if you go beyond the. superintelligence, there's a whole 'nother transformation.

coming that's going to look more like. magic. Think about how the technology of. today. would look like magic to someone from. 500 years ago. >> Mhm. >> You know? And that's without even like a. qualitative improvement in intelligence, right? Like the humans of today aren't. like qualitatively smarter than the. humans from 500 years ago. It's just. that we've had more time to do research. and we have more like money and. resources to build, you know, prototypes and experiments and. run experiments and so forth. But if you. had a point where there were billions. and billions of AIs that were not only.

faster than humans, but like. qualitatively way, way, way better at. everything and in particular at doing. scientific research, we should expect. that some of the things that they. develop will seem like magic to us and. we'll just completely like we did not. think that was even possible, you know? People don't want to die. People don't. want to be hit by cars. People don't. want to be like attacked by a random. mass murderer. >> Cancer's gone? >> I mean, not just cancer, like. >> [snorts]. >> you know, all all a lot of the stuff. that happens in science fiction will. probably have happened by then. So, things like people scanning their brains. and uploading into into computers,

right? Or self-replicating robots. in the asteroid belt. uh creating more and more satellites to. uh produce more and more power to. produce more and more self-replicating. robots and so forth. >> Most people still live on Earth, but the. trend is to move to space? >> That's right. Yeah. So, like if. if you end up in the situation where the. entire. human economy. is just like a tiny drop in the bucket. that is the entire economy and it's just. like a huge amounts of robots and AIs.

that are. moving incredibly quickly, then what you. want is Earth to be. mostly left as something like a. preserve, you know? I think a lot of people are. worried about the environment being. destroyed, which it totally would be if it wasn't. protected. And uh. you know, there's a lot of people who. sort of like their lives as it is. and don't want to be uploaded or live in. some crazy new future thing. And it. seems to us like the reasonable solution. to these issues is. uh create new living spaces off the.

planet with some of that vast. economic wealth and activity that's. happening. for the people who want that sort of. thing. And then that way the Earth can. be preserved. >> Data center picture here of data centers. in the ocean. Uh I mean, there's three. images there of. different environments where humans. might live. >> Again, like our proposal was you. preserve like 99% of the Earth. uh mostly as is as historic or. environmental from as historic or. environmental reasons, but then like.

some parts of it you designate as. special economic zones where the robots. can go crazy and dig giant pit mines and. produce factories and so forth. Um. we were thinking it would be good to. build the data centers on the ocean. instead of um on land for a variety of. reasons, although later space would be. better and. I could see that being reasonable as. well. >> What about immortality in a world of AI? Um 20 Well, 30, 45, you say you've lived. a dozen lifetimes and are immortal. passing from life to life.

as if by reincarnation. I mean, there's a lot of billionaires at. the moment that are focused on. longevity. I mean, Brian Johnson's said. he's got this central rule, which is do. not die right now Yeah. Because we're in. the age of AI and it's conceivable that. with superintelligence we'll be able to. choose when we die. >> Yep. I think that's probably right. We. don't depict that happening in this part. because at this part they only have, you. know, human-level AIs, but that's one of. those things that seems quite plausible. that superintelligence could achieve um.

through a variety of means. >> What is your hope with all of this. stuff? And why did you do this? Why did you. make this 2040 plan A? >> In the like first week after we. published AI 2027, it it blew up a lot. bigger than we expected, by the way. Like after we published AI 2027, it it. blew up a lot bigger than we expected, by the way. Like we actually made. forecasts beforehand of like. how many views it would get and stuff. like that and it was like. 90th percentile outcome. So, like.

um very much not what we expected. Um. but in like the Twitter storm that. happened various people were like. all right, why are you giving us all. this like doom and gloom uh. predictions? Like how about a more. positive vision of like what you think. we should do instead? And I think that. that seed sort of like. implanted in us and then we were like, yeah, that's reasonable. Like we've sort. of depicted what we think the default. path looks like and why we think it's. pretty scary. Now maybe we should switch tacks and.

come up with some actual recommendations. and then depict that as well. >> Even though you don't believe they're. pro-probable. >> Yeah, I mean you can vote for a. political candidate even if you aren't. confident that they're going to win, you. know? And and you can say like here's. what I think we should do even if you. think that people are probably not going. to do it. You shouldn't say this if you think it's. completely unlikely. Like if you think. there's no chance, then like maybe you. shouldn't bother. But we think there's a. chance. Like in particular, for the. reasons that we described in the. scenario we think that people are going. to wake up to the. power of AI over the next few years.

>> Because of something happens? >> The companies are saying that they're. going to do this. >> Mhm. >> And [clears throat]. they are kind of on track and it just. sort of makes sense that like if they. get anywhere close. to this level of AI, then there's like. big issues and big problems and like we. need to like do something about this. And so I think that even if there's not. any like very dramatic warning shot or. something. I think that just naturally people are. going to start paying more attention to. this and reasoning through the. implications and trying to predict.

>> what's going to happen. >> And so naturally people are going to be. more interested in regulation of AI for. example. And in fact. there's actually like there's there's. actually more of this happening than we. predicted. >> More of what happening? >> Serious interest in reg- AI regulation. So at the time that we published AI 2047. the sort of like mainstream position of. the tech companies and in the government. was kind of like AI regulation bad idea. >> Free for all. >> Free for all. >> Yeah. >> In fact, there was even an attempt to um.

preemptively ban states from regulating. AI. >> Yeah. >> You remember that? Now it seems like the. conversation has changed a lot. Like now. that the US government just told. Anthropic they have to shut down. their AI because they were worried that. bad actors would use it for cyber. attacks, you know? The government. is like waking up and doing more stuff. than we expected already. And. we're actually hopeful that that trend. will just continue and that. before it's actually too late, there. will be very serious conversations. happening inside the government and.

outside the government and in the. broader society about all of these. issues and trying to uh. chart a course that is um avoids the. loss of control and concentration of. power risks that we mentioned. >> You um you've spent what must be almost. coming up to 15 years thinking about. this stuff. Um if this here was a button. and if you press that button, your plan. S would occur and it would shut down. every data center that is currently. training a frontier AI model uh for.

good. There would never be any other. >> Mhm. >> AI labs um working on these problems, would you press that button? >> I was I was about to slam it until you. said for good. >> Oh, okay. >> Like I think I think if it was a sort of. temporary shut down, I would totally. slam that button. Because we are not. ready to do this, you know? Like what. civilization is not ready to have these. companies. automate themselves and then get smarter. and smarter and then have the super. intelligent. Like no, there's a bunch of. reasons why that's really uh dangerous.

But I would be at least hesitant to. press this button. if it permanently foreclosed the. possibility of ever doing it again for. sure. >> But but if you think that plan D is. probable, which is this race we're on to. super intelligent. >> If I had a choice between D and S, I. think I would press it. >> Well, it's it comes down to what you. think, right? Cuz if you think that's. that is what's going to happen, plan B. And the only alternative. >> I didn't say this is what's going to. happen. >> Probabilistically. >> Yeah, yeah, yeah. Like like I'd be like. this is the most likely, maybe this is.

the second most likely, maybe this is. the third most likely. They are all. possible. >> So with your current perspective on. whatever one you think is going to. happen, would you press the button? I'm. giving you a an S, a definite S, or. whatever you think is going to happen. >> That's tough. >> [sighs]. >> What is the scope of the shutdown? So is. it. >> It's no one can train an AI model again. Ever again. >> That's real rough cuz like I said, there's loads of benefits that we could. get from AI if we do it right. Um.

>> I think I I've almost put you in the. position of Sam Altman. >> Yeah. [laughter]. >> To some degree. >> Yeah. Um let me Do you mind if I just take a. moment to think about this? >> think about it. Perfectly to think. >> Yeah. I think I would not press. the button, but I'm I feel very torn. about it. Um the reason why I think I would not. press the button is that. I still have substantial hope that we.

can get something much better than this, something more like this. And I think that. Basically, I think that if we don't. build powerful AI systems eventually, then. we're probably going to die as a. civilization. eventually, you know, like 100 years. from now, 200 years from now, something. like that. Like nuclear war, pandemic, you know. I I don't think human civilization right. now is like super super stable. Um.

and so. I think that. basically, what I was about to say was. the possible benefits for posterity and. for all the billions and billions of. people who could live in the future. outweigh the like. the current level of risk, but actually. >> I've heard that narrative before. Yeah, I don't know. Like. Yeah, like maybe maybe it's just like. nope. The people right now. are the people we should prioritize. People right now are in grave danger. They're going to be fine for at least.

the next couple of decades. So, never mind posterity. Prioritize the people right now. Um and people right now definitely don't. want. to do this lottery, I would say. Um. >> [sighs and gasps]. >> Yeah, you've really asked me a tough. question. So, would you press the button. if that was the button? Probably not, but I would feel very. torn. >> Okay. So, what I I always think about the. personas of like the audience that are. watching. And these are, you know,

they're they're very curious people, especially on the subject of AI as we've. seen, but they they want to know like. what it means for them. I think a lot of. them also want to know what they can do. >> Uh yes. Yeah, what can people do? Well, I think that if you either have. talent or passion, you can get directly. involved. There's lots of organizations. that are worried about these things and. that are trying to do something about. it, like political advocacy or technical. research or like building useful tools. that will hopefully help people be.

better and stuff. But if you don't want. to like make any major career changes or. or things like that, then. I would say just pay more attention to. these issues and talk about it more with. people. Do stuff like, you know, emailing your congressman or whatever. It doesn't change things that much, but. it does help. I think that especially. for this particular issue, the core. problem is that people aren't taking it. seriously yet. Like if the sorts of things that I was. just saying to you for the last hour or. two were just like. top of everybody's mind,

we wouldn't even be here. Like there. would there would already be much more. significant regulation in place, you. know? And not only would there be more. heavy regulation in place, but there. would have been better regulation in. place that's less, you know, less like a. cudgel and more like a scalpel and. that's like more sensitive to what's. actually bad and what's not so bad and. so forth. And there'd be more expert. people in the government and advising. the government and so forth. So just in. general like. the more people wake up to these.

concerns and to these projections, I think the more likely it is that we. can do good stuff before it's too late. >> What about how they should vote at the. polls? We've got an election coming up. in the United States in a couple of. years time, but there's elections. happening all over the world all the. time. >> You should ask your candidates what they. think about all this AI stuff. You. should try to get them to like have. opinions and then you should vote for. the candidates whose opinions are better. on this topic. This is the most. important thing happening. in our lifetimes, probably in all of. history in fact, and it's very important. that it go well. And so this is what all.

the all the leaders of all the countries. should be thinking about and making. plans for. >> Isn't it such a weird thing to be alive. at this moment in time? Like I was thinking about all the times. that I could have been born. And I guess. my ancestors probably thought the same, but I was thinking as you were speaking. I was like, I think it's when you. referred to it as like the final show. >> Yeah. >> What was the phraseology you used? >> I said the the climate it was the run-up. to the climax or something. >> Yeah. I mean what a what a crazy thing. to be born in the run-up to the climax. where everything you're describing here. is within my lifetime conceivably.

hopefully. >> Yeah. >> Um or maybe not hopefully. What a crazy time to be alive. >> Certainly. >> I noticed that when I meant asked you if. you had kids your demeanor changed quite. considerably. >> Well, it's yeah. >> It's like you dropped into a different. state. Obviously that's been central to the. rumination that you've been. experiencing. >> Well, it is a sad topic, right? Like. when when I had kids. like the reason to have kids is in large. part about the future, you know? Like it's not just like a cuddly thing.

to have with you in the moment. It's cuz. you have all these hopes and dreams. about how they'll grow up and how. they'll go to their own thing and be. their own person and stuff. And. because of what's happening with AI, I. think a lot of those dreams are in. jeopardy. >> Presumably you still would have had. kids? >> I've actually flip-flopped on this. occasionally. Yeah. Basically the top. line answer is I'm not sure. The. my first child was had we we had her. when we were um in 209 she was born in. 2019. Yeah. So this is before my. timeline shortened a lot. So at that at. this point I was interested in AI, I was. tracking the field, I was making.

forecasts, but I didn't like actually expect it to. happen soon. You know? And then this caused like. when I when I did start thinking like oh. my gosh, it's going to be happening like. real soon. Um like by 2030, you know? Um that caused. some reconsidering. And so. I basically told my wife like let's not. have any more kids. It's too uncertain, you know? But that turned out to be really hard. because. especially for my wife. Like we already. had one kid and like.

no siblings. Um so eventually I sort of gave in and. was like okay, well, you know what? We. already have one. It's going to be all right. Like. maybe maybe the future will be good and. even if it's not like. well, we're all in the same boat. together. >> It's quite chilling what you're saying. It's chilling because you know more than. me. And if you're at home saying to your. wife, "Listen, maybe we should pause on. having more children and building a. family because of what's going on with. AI.". >> To be clear, is it Yes, I mean yes, it's. very concerning. I am I am chilled.

Uh this is bad. This is what I've been. saying. I hope things go well. I think things. might go well. Um I think that there's a. lot we can do to like steer things in a. better direction. >> I mean one of those things as well I. have to say is just speaking about it. It's I think a lot of the progress we've. seen with governments waking up and. you know, we've seen certain things with. people booing certain people at certain. events. Yeah. Um is it is it downstream. from people like yourself actually. coming on shows like this and all the. other podcasts and. Yeah. telling us what's going on. Yeah.

Because else we're to be fair, we're. going to be gaslighted by the people. that have the biggest PR machines. >> Yeah. >> So, um I often I think it's probably. worth me saying I find myself kind of in. two minds cuz I'm an entrepreneur and. I'm an I'm an investor. I'm an investor. in probably more than 100 companies now. and well so many of those companies are. using AI. I invested in Grok, the. inference chip company. Invested in. SpaceX which now own another Grok and. they're doing AI. I use AI every day in. my life. I've been using it through this. conversation to understand different. things that you've said. So, that's one. side of me which is like business.

builder, entrepreneur who has seen the. benefits of AI in my own life and then. there's the other side of me. And it's. funny cuz I think sometimes people think. you have to pick a camp. But through all of my life, even when I. was a social media CEO and I was saying. by the way listen I'm building a social. media business but I think there's some. downsides to social media. Find myself. at the same moment where I'm like I. build with AI. I have AI investments. And at the same time as a civilian I'm. like. >> Yeah. I mean I think that is a tension. I. think that there's there's different. way ways you can draw the line. So, and. I know lots of people who draw the line.

in lots of different ways. So, like. there's some people who just like I'm. not going to use AI. I think this stuff. is bad um and on a bad trajectory so I'm. going to like boycott AI, right? I'm not. one of those people. I use AI a lot. We. all do at AI Futures Project. Um it's. helpful for a lot of our work. The opposite end of the spectrum is. you. people being like. well, it seems like it's on a trajectory. to happen so the thing to do to make it. go well is to like. get involved and accumulate power and. try to like steer it from the inside. >> Mhm. >> And so I'm going to go work at OpenAI or.

Anthropic and like try to like climb the. ranks and then like you know, be someone. who matters when the important decisions. are being made. And I know loads of. people like that. That was like what I. was doing when I was. That wasn't what I was doing exactly but. like. >> That was the path. >> That was like that was a I mean this In. some sense this is what the whole. narrative of the companies are, right? Like this is why they tell themselves. it's okay to do what they're doing is. that they're worried about the other. guys, you know? And so like all these. people are deciding like we're going to. like lean really hard into it. We're. going to like be there in the room when. the when decisions are being made, you.

know? So, there's a whole spectrum and. I'm sort of like somewhere in the. middle. Like I'm not at the at. companies, I'm not helping them. go faster. Instead, I'm talking to the broad public. and trying to advocate for what I think. is the. my current best guess as to the way out, you know, the way forward. Um but, I'm not like boycotting all the. AIs. I'm I'm not like, you know, uh trying to I'm not refusing to like. engage with it in that way. >> Do you think it's too late? >> No. I don't think it's too late. If I.

thought it was too late, I wouldn't be. here. >> Hm. Where would you [clears throat] be? >> With my family. >> What's your closing message to the. general public if you had to have a. closing statement to them? Maybe I would. say that like. >> you're going to hear a lot of things and. you already have been hearing a lot of. things about. AI and it's going to sound like science. fiction, but sometimes things which sound like. science fiction happen in reality. And in fact, many times historically. things which used to be science fiction. have then become reality. And people.

need to. stop thinking about what does or doesn't. sound like science fiction and just. start thinking about like the trends. and, you know, the actual trends that this. technology is on and. reading and forecasting how it's going. to go and then taking seriously the. possibility that it could go something. like this and then thinking about what. should be done about that. >> And where would you direct them to get. more information? You can go. >> to ai2047.com to read our previous. scenario. You can go to ai2040.com plan.

A to read our new proposal for what is. to be done. Um these things are not just. a sci-fi story. They also have lots of. like explainers and links to other. things. And so, they're kind of like a. nice jumping off point to to learn about. all of this stuff. Um. If you want, I could um after this is. over, like give a reading list of like. other papers and articles and. >> Please do. >> blogs to follow and so forth. >> And I'll link them all below in the. comment section. So, if you're listening. now, go ahead and take a look at the.

comment sec the description of this. episode and you'll see a bunch of links. which is Daniel's recommendations of. what you should read. You know, I think. it's it's just a really really great. moment in time to get educated on this. stuff. Um humans have a an inclination. because of cognitive dissonance where we. feel uncomfortable about something to. bury our heads in the sand and avoid it. >> Yeah. >> But actually, I think this is one such. time to do the very opposite. For many. reasons, to to inform yourself so you. know what actions to take, but also. because AI. you know, unavoidably is going to be a.

huge part of all of our lives and. careers. >> Yeah. Yeah, thank you. And that that's. the good way to. to say it. It's going to matter a lot. It's going to It's going to be. everywhere soon and um. we need to do something about it before. it's too late. >> What about AI Future Project? >> That's our organization. We spent a year. writing a 2047 after I left OpenAI and. then we spent another year writing a. 2040 Plan A. >> Daniel, thank you. >> Thank you. >> Thank you for all the work that you do. I can see how much you care about this. stuff and it's your care it's funny care. itself makes others feel care. And.

seeing how personal this is for you and. seeing how much you've dedicated your. life to this, but also hearing that you. you basically walked away from $2. million to be able to speak to the. public about this information. [clears throat] is incredibly admirable. and uh I I think voices like yours are. more important now than they've ever. been on this subject. So, please do keep. fighting the fight that you're fighting. and that's one of information, it is of. honesty, and it is uh of saying what. what is often the quiet part out loud. >> Thank you. >> doing really really smart research. I'll. link everything we've discussed today. below and I hope we can chat again.

sometime soon. >> Thank you. >> YouTube have this new crazy algorithm. where they know exactly what video you. would like to watch next based on AI and. all of your viewing behavior. And the. algorithm says that this video is the. perfect video for you. It's different. for everybody looking right now. Check. this video out. I bet you you might love. it.