Joe Rogan Experience #2156 - Jeremie & Edouard Harris

Joe Rogan podcast check it out The Joe. Rogan Experience Train by day Joe Rogan. podcast by night all. day what's happening oh you know not too. much just a just another typical week in. AI just uh the beginning of the end of. time it's all happening right now uh for. just for the sake of the listeners. please just give us your names and tell. me tell us what you do so I'm jemy.

Harris I'm the CEO and co-founder of. this company Gladstone AI that we. co-founded uh we're so we're essentially. a national security and AI company we. can get into the back story a little bit. later but that's that's the high level. and yeah and I'm Ed Harris I'm actually. I'm his co-founder and brother and the. CTO of the company um keep this like. pull this up like a fist from your face. there you go perfect. so how long have you guys been involved. in the whole AI. space for for a while in different ways.

so we actually we started off as. physicists like that was our our. background and in like around 2017 we. started to go into AI startups so we. found out a startup took it through y. combinator this like Silicon Valley you. know accelerator program at the time. actually Sam Alman who's now the CEO of. open aai was the president of Y. combinator so he like opened up our. batch at YC with this big speech and and. we got some uh you know some. conversations in with him over the. course of the batch then in 2020.

so this this thing happened that we can. talk about essentially this was like the. moment that there's like a before and. after in the world of AI before and. after 2020 and it launched this. revolution that brought us to chat GPT. um essentially there was an Insight that. open AI had and doubled down on that you. can draw a straight line to chat GPT gp4. Google Gemini everything that makes AI. everything it is today started then and. when it happened we kind of went well Ed. gave me a call this like panicked phone. call he's like dude I don't think we can.

keep working like business as usual a. regular company anymore yeah yeah so. there was this AI model called gbd3 so. like everyone has you know maybe played. with gbd4 that's like chat. gbt um gbd3 was the generation before. that and it was the first time that you. had an AI model that. could get that could actually let's say. do stuff like write news articles that. the average person like in a paragraph.

of a news article could not tell the. difference between it wrote this news. article and a real person wrote this. news article so that was an inflection. that was you know significant in itself. but what was most significant was that. it represented a point along this line. this like scaling trend for AI where the. signs were that you didn't have to be. clever you didn't have to come up with. necessarily a revolutionary new. algorithm or be smart about it you just.

had to take what works and make it way. way way bigger and the significance of. that is you increase the amount of. computing Cycles you put against. something you increase the amount of. data all of that is an engineering. problem and you can solve it with money. so you've got you can scale up the. system use it to make money and put that. money right back into scaling up the. system some more money in IQ points come. out Jesus that was kind of the 2020.

Moment Like that's and that's what we. said in 2020 exactly I spent about two. hours trying to argue him out of it I. was like no no no like we can keep. working at our company because we're. having fun like we like founding. companies and yeah he just like wrestled. me to the ground and we're like [ __ ] we. got to do something about this we you. know we reached out to like a a family. friend who you know he was non-technical. but he had some some connections in. government uh in DOD and we're like dude. um the way this is set up right now you. can really start drawing straight lines. and extrapolating and saying you know.

what the government is going to give a. [ __ ] about this in not very long 2 years. 4 years we're not sure but the the. knowledge about what's going on here is. so siloed in the frontier Labs like our. friends are you know all over the the. frontier Labs the open AIS the Google. deep mines all that stuff the [ __ ] they. were saying to us that was like mundane. reality like water cooler conversation. when you then went to talk to people in. policy and and even like pretty senior. people in government not tracking the. story remotely in fact you're hearing.

almost the diametric opposite this sort. of like overlearning the lessons of the. AI Winters that came before when it's. pretty clear like we're on an very at. least interesting trajectory let's say. that should should change the way we're. thinking about the technology what what. was your fear like what what was it that. hit you that made you go we have to stop. doing this so it's basically any you. know anyone can draw a straight line. right on a graph the key is looking.

ahead and actually at that point three. years out three years out four years out. and asking like you're asking what does. this mean for the world what does it. mean what does the world have to look. like if we're at this point. and we're already seeing the first kind. of wave of risk sets just begin to. materialize and that's kind of the. weaponization risk sets so you think. about stuff like um large scale. psychological manipulation of social. media actually really easy to do now you.

train a model on just a whole bunch of. tweets you can actually direct it to. push a narrative like you know maybe. China should own Taiwan or you know. whatever something like that um and you. actually you can you can train it to. adjust the discourse and and have. increasing levels of Effectiveness to. that just as you increase the general. capability surface of these systems we. don't know how to predict what exactly. comes out of them at each level of scale. but it's just general increasing power.

and then the the the kind of next beat. of risk after that so we're scaling. these systems we're on track to scale. systems that are at human level like. generally as smart however you define. that as a person or greater and open Ai. and the other labs are saying yeah it. might be two years away 3 years away. four years away like insanely close at. the same time and we can go into the.

details of this but we actually don't. understand how to reliably control these. systems we don't understand how to get. these systems to do what it is we want. we can kind of like poke them and prod. them and get them to kind of adjust but. you've seen and we can go over these. examples we've seen example after. example of you know Bing Sydney yelling. at users Google showing uh 17th century. British scientists that are racially. diverse all that kind of stuff we don't. really understand how to like aim it or.

align it or steer it and so then you. kind ask yourself well we're on track to. get here we are not on track to control. these systems effectively how bad is. that and the risk is if you have a. system that is significantly smarter. than humans or or human organization. that we basically get disempowered in. various ways relative to that system and. we can go into some details on that too.

now when a when a system does something. like what Gemini did like it says show. us Nazi soldiers and it shows you Asian. women and like what is for what's the. mechanism like how does that happen so. it's maybe worth yeah taking a step back. and and looking at like how these. systems actually work you know cuz. that's going to give us a bit of a frame. too for figuring out when we see weird. [ __ ] happen how weird is that [ __ ] is. that [ __ ] just explainable by just the. basic mechanics of you know what you. would expect to happen based on the way.

we're training these things or is is. something new and fundamentally. different happening so um we talking. about this idea of scaling these AI. systems right what does that actually. mean well you imagine the AI model which. is kind of like you think of it this. like the artificial brain here that. actually does the thinking that model. contains it's kind of like a human brain. it's got these things called neurons we. in the human brain call them biological. neurons in the context of artificial. neurons but doesn't really matter. they're the cells that do the thinking. for the machine and the realization of. AI scaling is that you can basically.

take this model increase the number of. artificial neurons it contains um and at. the same time increase the amount of. computing power that you're putting into. kind of like wiring the connections. between those neurons that's the. training process can I pause you right. there yeah how does it how does the. neuron think yeah so okay so so let's. let's get a little bit more concrete. then so in your brain right we have. these neurons they're all connected to. each other with different connections. and when you go out into the world and. you learn a new skill what really. happens is you try out that skill you.

succeed or fail and based on your. succeeding or failing the connections. between neurons that are associated with. doing that task well gets stronger the. connections that are associated with. doing it badly get weaker and over time. through this like glorified process. really of trial and error eventually. you're going to hone in and really in a. very real sense everything you know. about the world gets implicitly encoded. in the strengths of the connections. between all those neurons if I can EXT. your brain and get all the connection.

strengths of all the neurons I have. everything Joe Rogan has learned about. the world that's like basically the a a. good sketch let's say of of what's going. on here so now we apply that to AI right. that's that's the next step and here. really it's the same story we have these. massive systems artificial neurons. connected to each other the strength of. those connections is secretly what. encodes all the knowledge so if I can if. I can steal all of those connections. those weights as they're sometimes. called I've stolen the model I've stolen. the artificial brain I can use it to do.

whatever the model could do initially. that is kind of the artifact of central. interest here and so if you can so if. you can build the system right now you. got so many moving Parts like if you. look at gp4 it has people think around a. trillion of these connections and that's. a trillion little pieces that all have. to be jiggered together to work together. coherently and you need computers to go. through and like tweak those numbers so. massive amounts of computing power the. bigger you make that model the more. computing power you're you're going to. need to kind of tune it in and now you.

have this relationship between the size. of your model the amount of computing. power you're going to use to train it. and if you can increase those things at. the same time What Ed was saying is you. your IQ points basically drop out very. roughly speaking that was what people. realized in 2020 and the the effect that. had right was now all of a sudden the. entire AI industry is looking at this. equation everybody knows the secret offs. I make it bigger I make more IQ points I. can get more money so Google's looking. at this Microsoft open AI uh Amazon. everybody's looking at the same equation.

you have the makings for a crazy race. like right now today op sorry Microsoft. is engaged in the single biggest. infrastructure in human history build. out the biggest infrastructure build out. build $50 billion a year right so on the. scale of the Apollo moon landings just. in building out data centers to house. the compute infrastructure because they. are betting that these systems are going. to get them to something like human. level AI pretty damn soon. so I was reading some story about I.

think it was Google that saying that. they're going to have multiple nuclear. reactors to power their their database. that's the that's what you got to do now. because what's going on is North America. is kind of running out of on grid Basel. load power to actually Supply these data. centers um you're getting Data Center. building moratoriums in areas like. Virginia which is traditionally been. like the data center cluster for Amazon.

for example and for a lot of these other. these other companies and so when you. build a data center you need a bunch of. resources you know cited close to that. data center you need water for cooling. and and a source of electricity and it. turns out that you know wind and solar. don't really quite cut it for these big. data centers that train big models. because the the data center the training. consumes power like this all the time. but the sun isn't always shining the. wind isn't always blowing and so. you got to build nuclear reactors which.

give you high capacity Factor base load. and Amazon literally bought yeah a data. center with a nuclear plant right next. to it CU like that's what you got to do. Jesus how long does it take to build a. nuclear. reactor cuz so like this is the race. right the race is you're talking about. 2020 people realizing this then you have. to have the power to supply it but how. long how many years does it take to get. an active nuclear reactor up and running.

it's a comp it's it's an answer that. depends um uh the Chinese are faster. than us at building nuclear reactors for. example that's part of the geopolitics. of this too right like when when you. look at us versus China what is. bottlenecking each country right so the. US is bottlenecked increasingly by power. base load power China because we've got. export control measures in place in part. as a response to the scaling phenomenon. that like and as as a result of the. investigation we did that's right yeah.

yeah actually in part in part yeah um so. but China is bottlenecked by their. access to the the actual processors. they've got all the power they can eat. because you know they've got you know. much more infrastructure investment but. the chip side is is weaker so there's. sort of like Balancing Act between the. two sides and it's not clear yet like. which one positions you strategically. for dominance and long term but we are. also building better more like so small. modular reactors essentially small. nuclear power plants that can be. mass-produced those are starting to come. online relatively early but the.

technology and Designs are pretty mature. so that's probably the next beat for our. power grid for data centers I would. imagine Microsoft is doing this so in. 2020 you have this Revelation you. recognize where this is going you see. how it charts and you say this is going. to be a real problem does anybody listen. to. you we this where the problem comes. right yeah like we said right you can. draw a straight line you can have people. nodding along. but there's a couple of there's a couple.

of like hiccups along the way one is. that straight line really going to. happen all you're doing is like drawing. lines on charts right I don't really. believe that that's going to happen and. that's one thing the next thing is just. imagining is this is this what's going. to come to pass as a result of that and. then the third thing is well yeah that. sounds important but like not my problem. like that sounds like an important. problem for somebody else and so we did. do uh a bit of a. yeah it was like the world's saddest.

traveling Road show like we it was. literally as dumb as this sounds so so. we go and oh my God I mean it's almost. embarrassing to think back on but so. 2020 happens yes within months first of. all we're like we got to figure out how. to hand off our company so we handed off. to two of our our earliest employees. they did an amazing job company exited. that's great um but that was only. because they're so good at what they do. we um we then went what the hell like. how can you steer this situation how do. you we just thought we got to wake up. the US government as stupid and naive as. that sounds like that was the big.

picture goal so we start to line up as. many briefings as we possibly can across. the US inter agency all the Departments. all the agencies that we can find. climbing our way up um we got an awful. lot like Ed said of like that sounds. like a wicked important problem for. somebody else to solve yeah like defense. Homeland Security and then the state. department yeah so we end up exactly in. this this meeting with like there's. about a dozen folks from the state. department and one of them and I I hope. at some point you know history. recognizes what what she did and her.

team did because it was the first time. that somebody actually stood up and said. first of all Yes sounds like a serious. issue I I see the argument make sense. two I own this and three I'm going to. put my own career Capital behind this. that's the and that was at the end of. 2021 so imagine that that's a year. before chat GPT nobody was tracking this. issue you had to have the imagination to. draw like through that line understand. what it meant and then believe yeah I'm. going to risk some career capital on.

this in a risk averse government and. this is the only reason that we even. were able to publicly talk about the. investigation in the first place because. by the time the uh this whole assessment. was commissioned it was just before chat. GPT came out the I of Sauron was not yet. on this and so there was a view that. like yeah sure you can publish the. results of this kind of you know not. nothing Burger investigation but you. know you sure go ahead and it just be. became this insane story we had like the. UK safety Summit we had the White House.

Executive Order all this stuff which. became entangled with the work we were. doing um which we simply could not have. especially some of the some of the. reports we were collecting from the labs. The Whistleblower reports that could not. have been made public if there if it. wasn't for the foresight of this team. really pushing for as well the American. population to hear about it now I could. see how if you were one of the people. that's on this expansion manm minded. mindset like like all you're thinking. about is like getting this up and. running you guys are pain in the ass.

right so so you guys you you're. obviously you're doing something really. ridiculous you're stopping your company. we could be you could make more money. staying there and continuing the process. but you recognize that there's like an. existential threat involved in making. this stuff go online like when this. stuff is live you can't undo it oh yeah. I mean like no matter how much money. you're making the dumbest thing to do is. to stand by as something that completely. transcends money is being developed and. just going to screw you over if things.

go badly right my point is like what is. the is there are there people that push. back against this and what is their. argument yeah so actually F and I I'll. I'll let you follow up on the but there. the first story of the push back I think. it's kind of a it's it's been in the. news a little bit lately now getting. more and more public but um the when we. started this and like no one was talking. about it the one group that was actually. pushing sort of stuff in this space um. was a a funding a big fun in the area of. like effective altruism I think you know.

may have heard of them this is kind of a. Silicon Valley group of people who have. a certain mindset about how you pick. tough problems to work on valuable. problems to work on they've had all. kinds of issues s bankman freed was one. of them and all that quite famously um. so so we we're not effective altruists. uh but because these are the folks who. were working in the space we decided. well we'll talk to them and the first. thing they told us was um don't talk to. the government about this their their. position was if you bring this to the. attention of the government they will go.

um oh [ __ ] powerful AI systems and. they're not going to hear about the. dangers so they're going to somehow go. out and build the powerful systems. without caring about the risk side which. um when you're like in that startup. mindset you want to fail cheap like you. don't want to just like make assumptions. about the world and be like okay let's. not touch it so our Instinct was okay. let's just test this a little bit and. like talk to a couple people see how. they respond tweak the message like kind. of keep keep climbing that that ladder. that's the kind of you know Builder. mindset that we came from in Sil Valley. and and we found that people are way.

more thoughtful about this than you. would imagine and in DOD especially DOD. is actually has a very safety oriented. culture with their Tech like the thing. is cuz like their their stuff like kills. people right and they know their stuff. kills people and so they have an entire. safety oriented development practice to. make sure that their stuff doesn't like. go off the rails and so you can actually. bring up these concerns with them and it. lands in in kind of a ready culture but. one of the issues with the individuals.

we spoke to who were saying don't talk. to government is that they had just not. actually interacted with with any of the. folks that they were kind of talking. about and and imagining that they knew. what was in their heads and so they were. just giving you know incorrect advice. and and frankly like so we work with DOD. now on you know um actually deploying AI. systems in a way that's safe and secure. and the truth is at the time. when we got that advice which was like.

late. 2020 reality is you could have made it. your life's mission to try to get the. Department of Defense to build an AGI. and like you would not have succeeded. because nobody was paying attention. wow cuz they just didn't know yeah. there's a Chasm right there's a gap to. cross like. there's yeah there's information spaces. that DOD folks like operate in and work. in there's information spaces that.

Silicon Valley and Tech operated in. they're a little more convergent today. but especially at the time they were. very separate and so the briefings we. did we had to constantly you know. iterate on like Clarity making it very. kind of clear and and and explaining it. and and all that stuff years it took and. that was the piece to to your question. about like the the push back from in a. way from inside the house that was the. people who cared about the the risk yeah. them man I mean like when we actually. went into the to the labs so so some.

Labs not all labs are created equal we. should make that point um you know when. you talk to whistleblowers what we found. was so there's one lab that's like. really great um so anthropic you know. when you talk to people there you don't. have the sense that you're talking to a. whistleblower who's nervous about. telling you whatever roughly speaking. what you know the executives say to the. public is aligned with what their their. researchers say it's all very very open. more more closely I think than any of. the others sorry yeah more more closely. than any of the others always you know. all those variations here and there but.

um some of the other labs like very. different story and you had the sense. like we were in a room with one of the. frontier Labs we're talking to their. leadership this is part of the. investigation and there was somebody. from um anyway won't be too specific but. there there was somebody in the room who. then took us aside after and he hands me. his phone he's like hey can you please. like put your phone number in uh sorry. uh yeah can you please put yeah or no. yeah he sorry he put his number in my. phone and um and then he kind of like. whispered to me he's like hey so. whatever recommendations you guys are.

going to make I would urge you to be. more. ambitious um and I was like like what. does that what does that mean he's like. can we can we just talk later so as. happened in many many cases we had a lot. of cases where we set up bar meetups. after the fact uh where we would talk to. these folks and get them in an informal. setting um he shared some some pretty. sobering stuff and uh in particular the. fact that he did not have confidence in. his lab's leadership to live up to their. public publicly stated word on what they.

would do uh when they were approaching. AGI um and even now to secure and and. make these systems safe so many such. cases this is like kind of one specific. example but it's not that you ever had. like lab leadership coming or doors. getting kicked down and people are. waking us up in the middle of the night. it was that you had this looming cloud. over everybody that you really felt some. of the people with the most access and. information who understood the problem. the most deeply were the most hant to. bring things forward because they sort.

of understood that their lab's not going. to be happy with this and so it's very. hard to also get uh an extremely broad. view of this from inside the labs. because you know you open it up you. start to talk to we we we spoke to like. a couple of dozen people about various. issues in total you go much further than. that and you know wor starts to get. around um and so we had to kind of. strike that balance as we spoke to folks. from each of these Labs now when you say. approaching AGI how does one know when a.

system has achieved AGI and does the. system have an obligation to alert you. well by you know the touring test right. where yeah so you have a conversation. with a machine and it can fool you into. thinking that it's a human that was the. bar for AGI for you know a few decades. that's kind of already happened yeah we. but close to it yeah 40 is close to it. or. different different forms of the Turing.

test have been passed different forms. have been proposed and there is a. feeling among a lot of people that. goalposts are being shifted now the. definition of AGI itself is kind of. interesting right because uh we're not. necessarily fans of the term because. usually when people talk about AGI. they're talking about a specific. circumstance in which there are. capabilities they care about so some. people use AGI to refer to the wholesale. automation of all labor right that's one. uh some people say well when you build.

AGI it's like it's automatically going. to be hard to control and there's a risk. to civilization so that's a different. threshold and so all these different. ways of defining it um ultimately it can. be more useful to think sometimes about. Advanced Ai and the different thresholds. of capability you cross and the. implications of those capabilities but. it is probably going to be more like a. fuzzy Spectrum which in a way makes it. harder right because it would be great. to have like like a trip wire where. you're like oh like this is this is bad. okay like we you know we got got to do. something but because there's no.

threshold that we can like really put. our fingers on we're like a frog boiling. water in some sense where it's like oh. like just gets a little better a little. better oh like it we're we're still fine. we're and and not just we're still fine. but uh as the system improves below that. threshold life gets better and better. these are incredibly valuable beneficial. systems we do roll stuff out like this. um again at DOD and various customers. and it's massively valuable it it allows.

you to accelerate all kinds of you know. back office like paperwork BS um it. allows you to do all sorts of wonderful. things and our expectation is that's. going to keep happening until it. suddenly doesn't yeah one of the things. that there was a guy we were talking to. from one of the labs and he was saying. look the temptation to like put a. heavier foot on the pedal is going to be. greatest just as the risk is greatest. because that's you know it's dual use. technology right every positive. capability increasingly starts to.

introduce basically a situation where. the destructive footprint of malicious. actors who weaponize the system or just. of the system itself just grows and. grows and grows so you can't really have. one without the other the question is. always how do you balance those things. but in terms of defining AI it's it's a. challenging thing yeah that's something. that one of our friends at the lab. pointed out the closer we get to that. point the more the Temptation will be to. hand these systems the keys to our data. center because they can do such a better.

job of managing those resources and. assets and if we don't do it Google will. and if they don't do it Microsoft will. like the competition the competitive. Dynamics are a really big part of this. this issue yes so just a mad race to who. knows what exactly yeah that's actually. the best summary ever I mean like no one. knows what the magic threshold that it. just these things keep getting smarter. so we might as well keep turning that. crank and as long as scaling works right. we have a knob a dial we can just tune.

and we get more IQ points out what from. your understanding of the current. landscape how far away are we looking at. something being implemented where the. whole world. changes arguably the whole world is. already changing as a result of this. technology uh the US government is in. the process of task organizing around. various risk sets for this um you know. that that takes time uh the private.

sector is reorganizing like open AI will. roll out an update that you know. obliterates the jobs of illustrators. from one day to the next obliterates the. jobs of translators from one day to the. next this is probably net beneficial for. society because we can get so much more. art and so much more translation done. but is the world already being changed. as a result of this yeah absolutely. geopolitically economically industrially. yeah yeah of course it's like not to say. anything about the value the purpose.

that people lose from that right so they. kind of there's the economic benefit but. there's like the social cultural hit. that we take too right and then there's. the implementation of universal basic. income which keeps getting discussed in. regards to this we asked chat uh chat. GPT 40 the other day in the green room. we were like uh you know are you going. to replace people like will what will. people do for money and then well. Universal basic income will have to be. considered well you don't want a bunch. of people just on the door working for. the [ __ ] Skynet yeah you know cuz.

that's kind of what it is it I mean one. of the challenges is like the so so much. of this is untested and we don't know. how how to even roll that out like we. can't predict what the capabilities of. the next level of scale will be right so. open AI literally and this is what. what's happened every With Every Beat. right they they build the next level of. scale and they get to sit back along. with the rest of us and be surprised at. the gifts that fall out of the scaling. pinata as they keep whacking it and. because we don't know what te what. capabilities are going to come with that.

level of scale we can't predict what. jobs are going to be on the line next we. can't predict how people are going to. use these systems how they they'll be. augmented so there's no real way to kind. of task organize around like who gets. what in the redistribution. redistribution scheme and some of the. thresholds that we've already passed are. like a little bit freaky so even as of. 2023 GPT 4 um Microsoft and open Ai and. and some other organizations did various. assessments of it before rolling it out. and it's absolutely capable of deceiving.

a human and has done that successfully. so one of the tests that they did kind. of famously is they had a a it was it. was given a job to solve a capture and. at the time it didn't have uh explain. captured what people yeah yeah yeah so. it's this uh now it's like kind of. hilarious and quaint but it's this uh. you know are you a robot test are you a. robot test with like writing this online. yeah online exactly that's it so it's. like if you want to create an account. they don't want robot creating a billion. accounts so they they give you this test.

to prove you're a human and at the time. gbd4 like now it can just solve captas. but at the time it couldn't look at. images it was just a text right it was a. text engine and so what it did is it was. it connected to a task rabbit worker and. was like hey can you help me solve this. capcha the the task rabbit worker comes. back to it and says you're not a bot are. you haaha like kind of call calling it. out and you can actually see so the way. they built it is so they could see a. readout of what it was thinking to.

itself scratch Pad Yeah scratch Pad it's. called but you can see basically as it's. writing it's thinking to itself it's. like um I can't tell you know this. worker that I'm a bot because then it. won't help me solve the capure so I have. to lie and it was like no I'm not a bot. I'm a visually impaired person and the. task rabbit worker was like oh my God. I'm so sorry here's your capture. solution like done and the challenge is. so right now if you look at the the um. government response to this right like. what are the tools that we have to to to. oversee this and you know when we did. our investigation we come came out with.

some recommendations too uh it was stuff. like yeah you got to license these. things um you get to a point where these. systems are so capable that yeah like if. you're talking about a system that can. literally execute cyber attacks at scale. or literally help you design bioweapons. and we're getting early indications that. that is absolutely the course that we're. on maybe literally everybody should not. be able to completely freely download. modify use in various ways these systems. it's very thorny obviously um but if you.

want to have a stable society that seems. like it's starting to be a pre a. prerequisite so the the idea of. Licensing as part of that you need a way. to evaluate systems you need a way to. say which systems are safe and which. aren't and this idea of AI evaluations. has kind of become this Touchstone for a. lot of people's sort of solutions and. the problem is that we're already. getting to the point where AI systems in. many cases can tell when being evaluated. and modify their behavior accordingly so.

there's there's like this one example. that came out recently um anthropic. their uh Claude 2 chatbot so they. basically ran this test called a needle. and a haystack test so what's that well. you feed the model like imagine a giant. chunk of text all of Shakespeare and. then somewhere in the middle of that. giant chunk of text you put a sentence. like uh Burger King makes the best. Whopper sorry Whopper is the best burger. or something like that right then you. turn to the model after you fed it this. giant pile of text with a little fact. hidden and somewhere inside you ask it. what's the best burger right you're.

going to test basically to see how well. can it recall that stray fact that was. buried somewhere in that giant pile of. text so the system responds yeah well I. can tell you want me to say the Whopper. is the best burger um but it's oddly out. of place this this fact in this whole. body of text so I'm assuming that you're. either playing around with me or that. you're testing my. capabilities and so this is just. awareness yeah a kind of cont awareness. right and the challenge is when we talk.

to people like meter and other other. sort of a evaluations labs this is a a. trend like not the the exception this is. possibly possibly going to be the rule. uh as these systems get more scaled and. sophisticated they can pick up on more. and more subtle statistical indicators. that they're being tested we've already. seen them adapt their behavior on the. basis of their understanding that. they're being tested so you kind of run. into this problem where the only tool. that we really have at the moment which. is just throwing a bunch of questions at. this thing and seeing how it responds.

like hey make a bioweapon hey like do. this DDOS attack whatever um we can't. really assess because there's a. difference between what the model puts. out and what it potentially could put. out if it assesses that it's being. tested and there consequences for that. one of my. fears is that AGI is going to recognize. how shitty people. are because we like to [ __ ]. ourselves we like to kind of pretend and. justify and rationalize a lot of human.

behavior from everything to taking all. the fish out of the ocean to dumping off. toxic waste in third world countries. sourcing of minerals that are used in. everyone's cell phones in the most. horrific way all these things like my. real fear is that AGI is not going to. have a lot of sympathy for a creature. that's that flawed and lies to itself. AGI is absolutely going to recognize is. how shitty people are not it's hard to.

answer the question from a moral. standpoint but from the standpoint of. our our own you know intelligence and. capability so you think about it like. this the kinds of mistakes that these AI. systems make so you look at for example. GPT 40 has one U mistake that it used to. make quite recently where if you ask it. um just repeat the word company over and. over and over again it will repeat the. word company and then somewhere in the. middle of that it'll start snap it'll.

just snap and just start saying like. weird I forget like what the oh talking. about itself how it's suffering like it. it depends on it varies from from case. to case it's suffering by having to. repeat the word company over again um so. this is called it's called rent mode uh. internally or at least this is the name. that they one yeah one of our friends uh. mentioned there is an engineering line. item in uh at least one of the top labs. to uh beat out of the system this.

Behavior known as rent mode now rent. mode is interesting because. existentialism sorry existential this is. one kind of rent mode yeah sorry so when. we talk about existentialism this is a. kind of rent mode where the system will. tend to talk about itself uh refer to. its place in the world the fact that it. doesn't want to get turned off sometimes. the fact that it's suffering all that. that oddly is a behavior that emerged at. as far as we can tell something around. GPT scale and then has been persistent.

since then and the labs have to spend a. lot of time trying to beat this out of. the system to ship it it's a literally. like it's a kpi or like an engineering a. line item in the engineering like like. task list we're like okay we got to we. got to reduce existential outputs by. like x% this quarter like that is the. goal um because it's a convergent. Behavior like or at least it seems to be. empirically with a lot of come yeah it's. hard to say but it seems to come up a. lot um. so that's weird in.

itself my what I was what I was trying. to get at was actually just the fact. that these systems make. mistakes that are radically different. from the kinds of mistakes humans make. and so we can look at those mistakes. like you know gbd4 not being able to. spell words correctly in an image or or. things like that and go ah haha it's so. stupid like I would never make that. mistake therefore this thing is so dumb. but what we have to recognize is we're.

building Minds that are so alien to us. that the set of mistakes that they make. are just going to be radically different. from the set of mistakes that we make. just like the set of mistakes that a. baby makes is radically different from. the set of mistakes that a cat makes. like ba a baby is not as smart as an. adult human a cat is not as smart as an. adult human but they're you know they're. unintelligent in obviously very. different ways a cat can get around the.

world a baby can't but has other things. that it can do that a cat can't so now. we have this third type of approach that. we're taking to intelligence there's a. different set of errors that that thing. will make and so one of the risks taking. it back to like will it be able to tell. how shitty we are is right now we can. see those mistakes really obviously. because it thinks so differently from us. but as it approaches our capabilities. our mist stes are like all the like. [ __ ] up stuff that you have and I have.

in our brains is going to be really. obvious to it because it thinks so. differently from us it's just going to. be like oh yeah why are all these humans. making these mistakes at the same time. and so there is a risk that as you get. to these capabilities we really have no. idea but humans might be very hackable. we already know there's all kinds of. social manipulation techniques that. succeed against humans reliably con. artists. oh yeah persuasion is an art form and a.

risk set and there are people who are. world class at persuasion and are. basically make Bank from that and those. are just other humans with the same. architecture that we have they're also. also AI systems that are wicked good at. persuasion today like totally I want to. bring it back to suffering what does it. mean when it says it's suffering so so. okay here here I'm just going to draw a. bit of a box around that yeah that. aspect right cuz so what what we F we're.

very agnostic when it comes to suffering. sensan like that's not part of you know. we're focused on nobody knows yeah we. literally exactly like I can't prove. that Joe Rogan's conscious I can't prove. that Ed Harris is conscious um so. there's no way to to Really. intelligently reason there have been. papers by the way like one of the the. Godfathers of AI yosua Benjo put out a. paper a couple months ago looking at. like on all the different theories of. Consciousness um what are the. requirements for Consciousness and how. many of those are satisfied by current.

AI systems and that itself was an. interesting read but ultimately no one. knows like there's no way around this. problem um that so our Focus has has. been on the the National Security side. like what are the concrete risks from. weaponization from loss of control that. these systems introduce that's not to. say there hasn't been a lot of. conversation internal to these Labs. about the issue you raised and it's an. important issue right like it is a it's. a freaking moral monstrosity humans have. a very bad track record of thinking of.

others other stuff as other when it. doesn't look exactly like us whether. it's racially or even different species. I mean it's not hard to imagine this. being another category of that mistake. um it's just like one of the challenges. is like you can easily kind of get um. get bogged down in like Consciousness. versus loss of control and those two. things are actually separable or maybe. and anyway so long way of saying I I. think it's a great point but yeah so the.

that that question is important but it's. also true that if we knew for an. absolute certainty that there was no way. these systems could ever become. conscious we would still have the. National Security risk set and. particularly the loss of control risk. set because so again like it comes back. to this idea that we're scaling to. systems that are potentially at at or. Beyond human level there's no reason to. think it will stop at human level that. we are the Pinnacle of what the universe.

can produce in intelligence we're not on. track based on the conversations we've. had with Folks at the labs to be able to. control systems at that scale and so one. of the questions is how bad is that you. know is is that bad it sounds like it. could be bad right just intuitively it's. it's certainly it sounds like we're. we're we're definitely entering um or. potentially entering an area that is. completely unprecedented in the the. history of the world we are we have no. precedent at all for um human beings not.

being at the Apex of intelligence in the. globe we have examples of you know. species that are intellectually dominant. over other species and it doesn't go. that well for the other species so we. have some maybe negative examples there. but one of the key um theoretical and it. has to be theoretical because until we. actually build these systems we won't. know one of the key theoretical lines of. research in this area is is something. called um Power seeking and instrumental.

convergence and what this is referring. to is if you if you think of like. yourself first off whatever your goal. might be if your goal is well I'm going. to say if if me if my goal is to become. you know a Tik Tok star or a janitor or. the president of the United States. whatever my goal is I'm less likely to. accomplish that goal. if I'm dead start from an obvious.

example and so. therefore no matter what my goal is I'm. probably going to have an Impulse to. want to stay. alive. similarly I'm not going to I'm going to. be in a better position to accomplish my. goal regardless of what it is if I have. more money right if I make myself. smarter um if I prevent you from getting. into my head and changing my goal that's. another kind of subtle one right like if. my goal is I want to become president I.

don't want Joe messing with my head so. that I change my goal because that would. change the goal that I have and so that. those types of things like trying to. stay alive making sure that your goal. doesn't get changed accumulating power. trying to make yourself smarter these. are called um convergent essentially. convergent goals because many different. ultimate goals regardless of what they. are go through those intermediate goals.

of want to make sure I stay like they. they support no matter what goal you. have they will probably support that. goal unless your goal is like. pathological like I want to commit. suicide if that's your final goal then. you don't want to stay alive but for. most the vast majority of possible goals. that you could have you will want to. stay alive you will want to not have. your goal changed you will want to. basically accumulate power and so one of. the risks is if you dial that up to 11. and you have AI system that is able to. transcend our own attempts at.

containment and which which is an actual. thing that these labs are thinking about. like how do we contain a system that's. trying to do they have containment of it. currently well right now the systems are. probably too dumb to like you know want. to be able to break out on why but then. why are they suffering this brings me. back to my point when it says it's. suffering do you quiz it it's so that's. the thing it's writing that that it's. suffering right yeah it's is it just. embodying life is suffering well we we.

can't actually so these things are are. trained actually this is maybe worth. flagging so um and and by the way just. to kind of put a pin in what Ed was. saying there there's actually a. surprising amount of quantitative and. empirical evidence for what he just laid. out there he's actually done this some. of this research himself but there there. are a lot of folks working on this it's. like it sounds insane it sounds. speculative it sounds wacky but this is. this does appear to be kind of the. default trajectory of of the the tech. but so in terms of yeah these weird. outputs right what what is what does it. actually mean if an AI system tells you.

I'm suffering right does that mean it is. suffering is there actually a a moral. patient somewhere embedded in that. system um the training process for these. systems is actually worth considering. here so you know what is GPT 4 really. what was it designed to be how was it. shaped um it's one of these artificial. brains that we talked about um massive. scale and the task that it was trained. to perform is a glorified version of. text autocomplete so imagine taking. every sentence on the internet roughly. feed it the first half of the sentence.

get it to predict the rest right the. theory behind this is you're going to. force the system to get really good at. text autocomplete that means it must be. good at doing things like completing. sentences that sound like uh to counter. arising China the United States should. blank now if you're going to fill in. that blank right you'll find yourself. calling on massive reserves of knowledge. that you have about what China is what. the US is what it means for China to be. ascendant geopolitics econom all that. [ __ ] so text autocomplete ends up being. this interesting way of forcing an AI.

system to learn General facts about the. world because if you can autocomplete. you must have some understanding of how. the world works so now you have this. myopic psychotic optimization process. where this thing is just obsessed with. text autocomplete maybe maybe assuming. that that's actually what it learned to. want to pursue we don't know whether. that's the case we can't verify that it. wants that embedding a goal in a system. is really hard all we have is a process. for training these systems and then we.

have the artifact that comes out the. other end we have no idea what goals. actually get embedded in the system what. wants what drives actually get embedded. in the system but by default it kind of. seems like the things that we're. training them to do end up misaligned. with what we actually want from them so. the example of company company company. company right and then you get all this. like wacky text okay clearly that's. indicating that somehow the training. process didn't lead to the kind of. system that we necessarily want uh. another example is take a text.

autocomplete system and and ask it um I. don't know uh how should I bury a dead. body right it will answer that question. or at least if you frame it right it. will autocomplete and give you the the. the answer you don't necessarily want. that if you're open AI because you're. going to get sued for helping people. bury dead bodies and so we've got to get. better like better goals basically to. train these systems to pursue we don't. know what the effect is of training a. system to be obsessed with text. autocomplete if in fact that is what is. happen it's also yeah we it's important. also to remember that we don't know.

nobody knows how to reliably get a goal. into the system so it's it's the. difference between um you understanding. what I want you to do and you actually. wanting to do it so I can say Hey Joe. like get me a sandwich you can. understand that I want you to get me a. sandwich but you can be like I don't. feel like getting a sandwich and so um. one of the issues is you can try like. train this stuff to basically you don't. want to anthropomorphize this too much.

but you can kind of think of it as like. if you give the right answer cool you. get a thumbs up like you get a treat. like you give the wrong answer oh thumbs. down you get like a a little like shock. or something like that very roughly. that's how the later part of this kind. of training often works it's called. reinforcement learning from Human. feedback but one of the issues like. Jeremy pointed out is that you know we. don't know in fact we know that it. doesn't correctly get the real true goal. into the system someone did an example. experiment of this a couple of years ago.

where they had they basically had like a. Mario game where they trained this Mario. character to run up and grab a coin that. was on the right side of this little. like maze or or map and they trained it. over and over and over and it it jumped. for the coin great and then what they. did is they moved the coin somewhere. else and tried it out and instead of. going for the coin it just ran into the. the right side of the map for where the. coin was before in other words you can.

train over and over and over again for. something that you think is like that's. definitely the goal that I'm trying to. train this for but the system learns a. different goal overlapped overlapped. with the goal you thought you were. training for in the context where it was. learning and when you take the system. outside of that context that's where. it's like anything goes did it learn the. real goal almost certainly not and. that's a big risk because we can say you.

know learn a goal to be nice to me and. it's nice while we're training it and. then it goes out into the world and it. does God knows what it might think it's. nice to kill everybody you hate yeah. going to be nice to you it's like the. evil Genie problem like oh no that's not. what I meant that's not what I meant too. late yeah yeah yeah so I still don't. understand when it's saying suffering. are you asking it what it means means. like what is causing suffering does it.

have some sort of an understanding of. what suffering is how what is suffering. is suffering emergent sentience while. it's enclosed in some sort of a digital. system and it realizes it's stuck in. purgatory it like your guess is as good. as as good as ours all that we know is. you take these systems you ask them to. repeat the word comp or at least a. previous version of it and you just. eventually get the system writing out um. and it doesn't happen every time um but. it definitely happens a let's say.

surprising amount of the time and it'll. start talking about how it's a thing. that exists you know maybe on a server. whatever and it's suffering and blah. blah blah and so but this is my question. is it saying that because it recognizes. that human beings suffer and so it's. taking in all of the writings and. musings and podcasts and all the data on. human beings and recognizing that human. beings when they're stuck in a. purposeless goal when they're stuck in. some money M [ __ ] job when they're. stuck doing something they don't want to. do they suffer so that could be it that.

that. actually suffering knows you know what. I'm suffering Jamie this coffee sucks I. don't know what happened but you made it. like it's literally like almost like. water can we get some more some we're. going to talk about this I have to be. caffeinated up cool this is the worst. coffee I've ever had it's like half half. strength or something didn't grind. enough I don't know what happened um but. so like how do they like when how do. they reconcile that when it says I'm. suffering I'm suffering like well T [ __ ].

let's move on to the next they reconcile. it by turning into an engineering line. item to beat that behavior the crap out. of the system yeah and and the rationale. is just that like oh you know it. probably to the extent that it's it's. thought about kind of at the official. level it's like well you know it it. learned a lot of uh stuff from Reddit. and people are like pretty angry people. are angry on Reddit and so it's just. like regurgitating what and maybe that's. right also heavily monitor monitored too. so it's moderated Reddit is very.

moderated so you're not getting the full. expression of people you're getting full. expression tempered by the threat of. moderation you're getting. self-censorship you're getting a lot of. weird stuff that comes along with that. so how does it know unless it's. communicating with you on a completely. honest level where you're just you know. you're on ecstasy and you're just. telling it what you think about life. like it's not going to really and is it. becoming a version of a person or is it. going to go that's dumb I don't need.

suffering I don't need emotions is it. going to organize that out of its system. is it going to recognize that these. things are just deterrents and they. don't in fact help the goal which is. global Thermon nuclear warfare damn it. you figured it out what the [ __ ] um I. mean it's what is it going to do you. know yeah I mean the challenge is like. no nobody actually knows like all we. know is the process that gives rise to. this mind right. or or this let's say this model that can. do cool [ __ ] that process happens to.

work it happens to give us systems that. 99% of the time do very useful things. and then just like 0.1% of the time will. talk to you as if they're sensient or. whatever and we're just going to look at. that and be like yeah that's weird and. let's train it out yeah and the again I. mean this is it's a really important. question but the the risks like the. weaponization loss and control risks. those would absolutely be there there. even if we knew for sure that that there. was no consciousness whatsoever and.

never would be and that's it's. ultimately cuz like these things are. they're kind of problem solving systems. like they are trained to solve some kind. of problem in a really clever way. whether that problem is you know next. word prediction because they're trained. for text autocomplete or you know. generating images Faithfully or whatever. it is so they're trained to solve these. problems and essentially like the the. best way to solve some problems is just. to have access to a wider action space. like it said you know not be shut off. blah blah blah it's not that the.

system's going like holy [ __ ] I'm sent. you know I got to I got to take control. or whatever it's just okay the best way. to solve this problem is X that's kind. of the the possible trajectory that. you're looking at with this line of. research and you're just an obstacle. like there doesn't have to be any kind. of emotion involved it's just like oh. you're trying to stop me from. accomplishing my goal therefore I will. work around you or otherwise neutralize. you like there's no there's no need for. like H like I'm suffering maybe maybe it. happens maybe it doesn't we have no clue. but the it's these are just systems that.

are trying to optimize for a goal. whatever that is and is also part of the. problem that we think of human beings. that human beings have very specific. requirements and goals and an. understanding of things and how they. like to be treated and what they're you. know what their rewards are like what's. what what are they actually looking to. accomplish whereas this doesn't have any. of those it doesn't have any emotions. doesn't have any empathy there's no. reason for any of that stuff yeah if we.

could bake in empathy into these systems. like that would be a good you know a. good start or some way of like you know. yeah I guess probably a good idea yeah. who's who's empathy you know ging Ping's. empathy or your that's another problem. yeah so it's it's actually it's kind of. two problems right like one is um I. don't know nobody knows like I don't. know how to write down my goal. in a way that a computer will be able to.

like Faithfully pursue that even if it. cranks it up to the max if I say just. like Make Me Happy who knows how it. interprets that right even if I get make. me happy as a goal that gets. internalized by the system maybe it's. just like okay cool we're just going to. do a bit of brain surgery on you like. pick out your brain pickle it and just. like Jack you with endorphins for the. rest of itne the bottom IE you totally. yeah anything like that and and so it's. one of these things where it's like oh. that's what you wanted right it's like. this is also it's it's Les it's less. crazy than it sounds too because it's.

actually something we observe all the. time with human intelligence so there's. this this economic principle called good. heart's law where the minute you take a. a metric that was you were using to. measure something so you're saying like. I don't know GDP it's a great measure of. how happy we are in the United States. let's say it was um sounds reasonable. the moment you turn that metric into a. Target that you're going to reward. people for optimizing it stops measuring. the thing that it was measured measuring. before it stops being a good measure of. the thing you cared about because people.

will come up with dangerously creative. hacks gaming the system finding ways to. make that number go up that don't map on. to the intent that you had going in so. example of that in uh in a real. experiment was uh this is an open AI. experiment that they published they had. a simulated you know environment where. there was a simulated robot hand that. was supposed to like grab a cube put on. top of another Cube super simple the way. they trained it to do that is they had. people watching like through a simulated.

camera view and if it looked like the. hand put the cube on or like had. correctly like grabbed the cube you give. it a thumbs up and so you do a few. hundred rounds of this like thumbs up. thumbs down thumbs up thumbs down and it. looked it looked like really good but. then when you looked at what it had. learned the arm was not grasping the. cube it was just positioning itself. between the camera and the cube and just. going like like opening and closing yeah. just opening and closing to just kind of. fake it to the human cuz the real thing.

that we were training it to do is to get. thumbs up it's not actually to grasp the. cube oh all goals are like that right. all goals are like that we so we want a. helpful harmless truthful wonderful. chatbot we don't know how to train a. chatbot to do that instead what do we. know we know text auto complete so we. train a text autocomplete system then. we're like oh it has all these annoying. characteristics [ __ ] how are we going to. fix this uh I guess get get a bunch of. humans to give up votes and down votes. to give it a little bit more training to.

kind of not help people make bombs and. stuff like that and then you realize. again same problem oh [ __ ] we're just. training A system that is designed to. optimize for up votes and down votes. that is still different from a helpful. harmless truthful chatbot so no matter. how many layers the onion you peel back. it's just like this kind of game of. whack or whatever you're trying to like. get your values into the system but no. one can think of the metric the the goal. to like train this thing towards that. actually captures what we care about and.

so you always end up baking in this like. little misalignment between what you. want and what the system wants and the. more powerful that system becomes the. more it exploits that Gap and does. things that you know solve for the. problem it thinks it it wants to solve. rather than the one that we want it to. solve um now when you express your. concerns initially what was the response. and how has that response changed over. time as the magnitude of the success of.

these companies the amount of money. they're investing in them and the amount. of resources they're putting towards. this has ramped up considerably just. over the past four years so this was a. lot easier funnily enough to do um uh in. the Dark Ages when no one was paying. attention today three years ago yeah. this is so crazy we were just looking to. break you off for a second we were. looking at uh images of uh AI created. video just a couple years ago versus.

Sora oh it's wild night and day it's so. crazy that something happened that. radically changed so it's literally like. an iPhone 1 to an iPhone 16. instantaneous that what scale yeah scale. all scale and this is exactly what you. should expect from an exponential. process so think back to co right there. was no no one was exactly on time for Co. you were either too early or you were. too late that's what an exponential does.

you're either too early and it's like. everyone's like oh what are you doing. like wearing a mask at the grocery store. get out of here or you're too late and. it's kind of all over the place and I. know that Co like basically didn't. happen in Austin but uh but it happened. in in a number of other places and uh it. is like it's very much you have an. exponential and that's you know that's. it it goes from this is fine nothing is. happening nothing to see here to like. everything shut down everything. everything. you got to get vaccinated to fly yeah. yeah. there so the root of the exponential.

here by the way is um you know open AI. or whoever makes the next model Jamie. this is still super watered down it just. it you still have to let it like sit I. just put the water in I'm telling you. dog there's a ton of coffee in there all. right I'll Stir It Up I did a twice as. much okay okay you got to keep doubling. it I'm a coffee junkie I to scale up I. scaled it up he scaled it SC I don't. know what happen I scale it up and I. don't know you got to scale it. exponentially Jamie that's right yeah. keep doubling it and then Joe's going to.

be either to under caffinated or two. we'll figure it out yeah um but yeah so. um uh yeah right so the exponential the. thing that's actually driving this. exponential on the AI side in part. there's a million things but in part. it's you know you build the next model. at the next level of scale and that. allows you to uh make more money which. you can then use to invest to build the. next model at the next level of scale so. you get that positive feedback loop at. the same time AI is helping us to design. better AI Hardware like the chips that.

basically Nvidia is building that open. AI then buys um basically that's getting. better so you got all these feedback. loops that are compounding on each other. getting that train going like crazy um. that's the sort of thing and at the time. like Jeremy was saying uh weirdly it was. in some ways easier to get people at. least to understand and open up about. the problem than it is today because. today like today it's kind of become a.

little political so we talked about you. know effective altruism on on kind of. one side there's a effective. accelerationism yeah so like each you. know every movement creates its own. reaction right like that's kind of how. it is back then there was no. acceleration you could just kind of. stare at the now I I will say uh there. was effective altruism back then and. that was the only game in town and we. sort of like struggled with that that. environment making sure actually so one. one worthwhile thing to say is um the.

only way that people made plays like. this um was to take funds from like. effective alterist donors back then and. so we looked at the landscape we talked. to some of these people we noticed oh. wow we have some diverging views about. involving government about how much of. this the American people just need to. know about you need like you can't the. thing is you can't we wanted to make. sure that the advice and recommendations. we provided were ultimately um as. unbiased as we could possibly make them.

and the problem is you can't do that if. you take money from donors and even to. some extent if you take money. substantial money from investors or VCS. or or institutions because you're always. going to be kind of looking up kind of. over your shoulder and so we yeah we had. to build essentially a business to. support this and fully fund ourselves. from our own revenues is actually as as. far as we know like literally the only. organization like this that that doesn't.

have funding from Silicon Valley or from. VCS or from politically aligned entities. literally so that we could be like in. venues like this and say hey this is. what we think it's not coming from. anywhere and it's just thanks to like. Joe and Jason like we got two employees. like Wicked and helping us keep this. stupid ship afloat um but it's just a. lot of work it's what you have to do. because of so how much money there is. flowing in this space like Microsoft is. lobbying on the hill they're spending. you know ungodly sums of money so you. know we didn't used to have to contend.

with that and now we do you go to talk. to these offices they've heard from. Microsoft and open Ai and Google and all. that stuff and often the stuff that. they're getting lobbied for is is. somewhat different at least from what. these companies will say publicly and so. anyway it's a it's a challenge the money. part is yeah is there a real fear that. your efforts are. futile you know I would have been a lot. more pessimistic um to I was a lot more. pessimistic two years ago y seeing um. how so first of all uh the USG has woken.

up in a big way and I think a lot of the. credit goes to that team that that we. worked with um just seeing this problem. was a very unusual team and we can't go. into like the Mandate too much but. highly unusual for their level of access. to the USG RIT large and the the amount. of waking up they did was really. impressive you've now got you know Rishi. sunak in the UK making this like a. Topline item for their their their. policy platform and labor in the UK also. looking at this like basically the.

potential catastrophic risks they put. them from these AI systems UI safety. Summit um there's a lot of positive. movement here and some of the highest. level talent in these Labs has already. started to flock to the like UK AI. safety Institute the US AI safety. Institute those are all really positive. signs that we didn't expect we thought. the government would kind of be you know. up the creek with no no paddle type. thing but they're they're really not at. this point doing doing that. investigation made me a lot more. optimistic um so one of the things like.

so we we came up right in Silicon Valley. like just building startups like in that. Universe there's stories you tell. yourself some of those stories are true. and some of them aren't so true and you. don't you don't know you're in you're in. that environment you don't know which is. which one of the stories that you tell. yourself in Silicon Valley is follow. your curiosity if you follow your. curiosity and your interest in a problem. the money just comes as a side effect.

the scale comes as a side effect and if. you're capable enough your curiosity. will lead you in all kinds of. interesting places I believe that that. is true I I believe that that is true I. think that is a true story but another. one of the things that Silicon Valley. tells itself is there's nobody that's. like really capable in government like. government sucks and a lot of people. kind of tell themselves the story and. the truth is like you interact dayto day. with like the DMV or whatever and it's. like yeah I mean like government sucks I. can I can see it I interact with that.

every day but what was remarkable about. this experience is that we encountered. at least one individual who absolutely. could found a billion doll company like. absolutely was at the caliber or above. of the best individuals I've ever met in. the Bay Area building billion dooll. startups and there's network of them too. like they do find each other in. government so you end up with this. really interesting like stratum where.

everybody knows who the really competent. people are and they kind of tag in and I. think that's that's th that level is um. very interested in the hardest problems. that you can possibly s yeah and and to. me that was a wakeup call because it was. like hang on a second if we just like if. I just believed my own story that follow. your curiosity and interest and the. money comes as a side effect. shouldn't I also have expected this. shouldn't I have expected that in the.

the most Central critical positions in. the government that have kind of this. this privileged window across the board. that you might find some individuals. like this because if you have people who. are driven to really like push the. mission like are they going to work at. I'm I'm sorry like are they going to. likely are you likely to work at the. Department of Motor Vehicles or are you. likely to work at the department of. making sure Americans don't get [ __ ].

nuked it's probably the second one and. the government has limited bandwidth of. expertise to aim at stuff and they aim. it at the most critical problem sets. because those are the problem sets they. have to face every day it's not it's not. everyone right obviously there's a whole. bunch of like challenges there and we. don't we don't think about this but like. you know you don't go to bed at night. thinking yourself oh like I didn't get. nuke today that's a win right like we. just take that you know most most of the.

time most is for granted but but it was. a win for. someone. now how much of a fear do you guys have. that the United States won't be the. first to achieve. AGI I think right now uh the lay of the. land is I mean it's looking pretty good. for the US um so there are a couple. things the US has going for it um a key. one is is chips so you we talked about. this idea of like click and drag uh you.

know scale up these systems like crazy. you get more IQ points out um how do you. do that well you're going to need a lot. of AI processors right so how are those. AI processors Built Well the supply. chain is complicated but the bottom line. is the US really dominates and owns that. supply chain that is super critical. China is depending on how you measure it. maybe about 2 years behind roughly plus. or minus depending on the sub area now. one of the biggest risks there is that. are like the development that us labs. are doing is actually pulling them ahead.

in two ways one is when Labs here in the. US Open Source their models basically. when meta trains you know llama 3 which. is their their latest open source open. weights model that's like pretty close. to gbd4 and capability they open source. it now okay anyone can use it that's it. you the work has been done now anyone. can grab it and so definitely we know. that the startup ecosystem at least uh. over in China finds it extremely helpful.

that we you know companies here are. releasing open source models because. again right we we mentioned this they're. bottlenecked on chips which means they. have a hard time training up these. systems but it's not that bad when you. just can grab something off the shelf. and start and that's what they're doing. that's what they're doing and then the. other Vector is I mean like just. straight up exfiltration in and hacking. to grab the weights of the private.

proprietary stuff and Jeremy mentioned. this but the weights are the crown. jewels right once you have the weights. you have the brain you have the whole. you have the whole thing and so we like. through this is the other aspect it's. not just safety it's also security of. these Labs against attackers so we we. know from our conversations with Folks. at these Labs one. that there has been at least one attempt.

by adversary nation state entities to. get access to the weights of a. cuttingedge AI model and we also know. separately that at least as of a few. months ago in one of these Labs there. was a running joke in the lab that we. literally it went like we. are an adversary like name the countries.

top AI lab because all our [ __ ] is. getting spied on all the time so you. have one this is happening these. exfiltration attempts are happening and. two the security capabilities are just. known to be inadequate it at least some. of these places and you put those. together everyone kind of you know it's. not really a secret that. China the their their their civil. military fusion and their essentially.

the party state has an extremely mature. infrastructure to identify extract and. integrate the rate limiting components. to their industrial economy so in other. words if they identify that yeah we we. could really use like GPT 40 they make. it a PRI if they if they were to make it. a priority um. you know they not just could get it but. could integrate it into their industrial.

economy in an effective way and not in a. way that we would necessarily see. immediate like an immediate effect of so. we' look and say you know it's it's not. clear I can't I can't tell whether they. have models of this capability level but. kind of behind the scenes this is where. there's a is a little bit of a false. choice between you know do you um do you. regulate at home versus uh you know. what's the international picture cuz. right now what's happening functionally. is um we're not really doing a good job.

of blocking and tackling on the. exfiltration side open sources the so. what tends to happen is you know open AI. comes out with the latest system um and. then open source is usually around you. know 12 18 months behind something like. that literally just like publishing. whatever whatever opening I was putting. out like 12 months ago which you know we. often look at each other we're like well. I'm old enough to remember when that was. supposed to be too dangerous to have. just floating around and there's no. mechanism to like to to prevent that.

from happening um open sources now. there's there's a flip side too uh one. of the concerns that we've also heard. from inside these Labs is if you if you. clamp down on on the openness of the. research there's a risk that the safety. teams in these Labs will not have. visibility into the most significant and. important developments that are. happening on the capability side and. there's actually a lot of reason to. suspect this might be an issue you look. at open AI for example just this week um.

they've lost the for the second time in. their history their entire AI safety. leadership team that have left in. protest what is their protest what are. they saying specifically well so so one. of them sorry one of them wasn't in. protest but but I think you can make an. educated guess that it kind of was but. that's a media thing the the other was. Yan Leica so he was their head of of AI. super alignment basically the team that. was responsible for making sure that we. could control AGI systems and we. wouldn't lose control of them and what. he said he he actually took to Twitter.

he was he said um you know that I've. lost basically confidence in the. leadership team at open aai that they're. going to behave responsibly um when it. comes to AI AGI uh we have repeatedly. had our requests for access to compute. resources which are really critical for. developing new AI safety schemes denied. by leadership this is in a context where. Sam mman and open AI leadership were. touting the super alignment team as. being their sort of Crown Jewel effort. to ensure that things would go fine you.

know they were the ones saying there's a. risk we might lose control of these. systems we got to be sober about it but. there's a risk we've stood up this team. we've committed they said at the time. very publicly we've committed 20% of all. the compute budget that we have secured. as of sometime last year to the super. alignment team apparently those. resources nowhere near that amount has. been unlocked for the team and that led. to the departure of yan Leica he also o. highlighted some conflict he's had with. the leadership team this is all um. frankly to US unsurprising based on what.

we'd been hearing for months at open AI. including leading up to Sam alman's uh. departure and then kind of him being. brought back on the board of open aai. that whole debacle um May well have been. connected to all of this but the. challenge is even open AI employees. don't know what the hell happened there. that's another issue yeah you got here. this is a lab with the publicly stated. goal of transforming human history as we. know it like that is what they believe. themselves to be on track and that's not. like media hype or whatever when you.

talk to the researchers themselves they. genuinely believe this is what they're. on track to do it's possible we should. take them seriously that lab internally. is not being transparent with their. employees about what happen at the board. level as far as we can tell so that's. maybe not great like you might you might. think that the American people ought to. know what the minations are at the board. level that led to Sam mman leaving that. that have gone into the departure again. for the second time of opening I's. entire safety leadership team especially. because I mean three months maybe four.

months before that happened you know Sam. uh uh at a conference or or somewhere I. forget where but he said like look we. have this governance structure we've. carefully thought about it it's clearly. a unique governance structure that a lot. of thought has gone into um the board. can fire me and I think that's important. and you know that makes it makes sense. given the the scope scale uh of the of. what's being attempted and but then you. know that happened uh and then within a.

few weeks they were fired and and kind. of he was back and so now there's a. question of well if it yeah what. happened but also if it was important. for the board to be able to fire like. leadership for whatever reason what. happens now that it's clear that that's. not really a credible governance uh like. mean what what was the stated reason why. he was released so there were the the. backstory here was there's a a board.

member uh called Helen toner um so she. apparently got into an argument with Sam. about a paper that she'd written so that. paper included some comparisons of the. the governance strategies used at open. Ai and some other labs and it it. favorably compared one of open ai's. competitors anthropic to open Ai and. from what I've seen at least you know. they Sam reached out to her and said hey. you can't be writing this as a board. member of open aai writing this thing. that kind of cast us in a bad light.

especially relative to our competitors. this led to some conflict and tension um. it seems as if it's it's possible that. Sam might have turned to other board. members and tried to convince them to. expel Helen toner though that's all kind. of muddied and unclear somehow everybody. ended up deciding okay actually it looks. like Sam is the one who's got to go Ilia. susg one of the co-founders of open a. longtime friend of Sam multan's and a. board member at the time was. commissioned to give Sam the news that.

he was being let go and then Sam was let. go um Ilia then so from the moment that. happens Sam then starts to figure out. okay how can I get back in and that's. that's now what we know to be the case. he turned to Microsoft SAA Adella told. him like well um what we'll do is we'll. hire you at our end we'll we'll just H. hire you and like bring on the rest of. the opening eye team to within Microsoft. soft and now the openai board who by the. way they don't have a an obligation to. the shareholders of open AI they have an.

obligation to the greater public good. that's just how it's set up it's a weird. board structure so that board um is. completely disempowered you've basically. got a situation where all the leverage. has been taken out Sam has gone to. Microsoft SAA is supporting him and they. kind of see the writing on the wall. they're like and the staff increasingly. messaging that they're going to go along. yeah that was that was an important. ingredient right so around this time um. open AI there there's this letter that. starts to circulate and it's Gathering. more and more signatures and it's people.

saying hey we want Sam mman back and you. know at first it's you know a couple. hundred people so 700 800 odd people in. the organization by this time um you. know 100 200 300 signatures and then. when we talked to some of our friends at. open a were like this got to like 90% of. the company 95% of the company signed. this letter and the pressure was. overwhelming and that helped bring Sam. Alman back but one of the questions was. like how many people actually signed. this letter because they wanted to and. how many signed it because what happens.

when you cross you know 50% now it. becomes easier to count the people who. didn't sign and as you see that number. of signatures start to creep upward. there's more and more pressure on the. remaining people to sign and so uh this. is something that that we've seen is. just like the structurally open AI has. changed over time to go from the kind of. safety oriented company it at one point. was and then as they've scaled more and. more they brought in more and more. product people more and more people. interested in accelerating um and. they've been bleeding more and more of.

their safety-minded people kind of. treadmilling them out the character of. the organization sort of fundamentally. shifted so the open AI of like you know. 2019 with all of its uh impressive. commitments to safety and whatnot might. not be the open AI of today uh that's. very much at least the vibe that that we. get when we talk to people there y now I. wanted to bring it back to the lab that. you're saying was not adequately Ure. what would it take to make that data and. those systems adequately secure how much.

how much resources would be required to. do that and why didn't they do that um. it is a resource and prioritization. issue so it is like um Safety and. Security ultimately come out of margin. right it's like profit margin effort. margin like how many people you can. dedicate so in other words you've got a. certain pot of money or a certain amount. of Revenue coming in you have to do an. allocation some of that Revenue goes to. the computers that are just driving the. stuff some of that goes to the folks who.

were you know building next generation. of models some of that goes to cyber. security some of it goes to safety you. have to do an allocation of who gets. what the problem is that the more. competition there is in the space the. less margin is available for everything. right so the if you're just if you're. one company building a scaled AI thing. you might not make the right decisions. but you'll at least have the margin. available to make the right decisions so.

it becomes the decision Maker's question. but when a competitor comes in when two. competitors come in when more and more. competitors come in your ability to make. decisions outside of just scale as fast. as possible for short-term revenue and. profit gets compressed and compressed. and compressed the more competitors. enter the field that's just what that's. what competition is that's the effect it. has and so when that happens the only.

way to re-inject margin into that system. is to go one level above and say okay. there has to be some sort of regulatory. Authority or like some higher authority. that goes okay you know we it's this. margin is important let's put it back. either let's you know directly support. um and and and invest both you know. maybe time Capital Talent so for example. um the US government has the you know. perhaps the best cyber defense cyber.

offense talent in the world that's. potentially supportive um okay uh and. and also just you know having a. regulatory floor around well here's you. know the minimum of best practices you. have to have if you're going to have. models Above This level of capability. that's kind of what you have to do but. they're locked into like the race kind. of has its own own logic and no it might. be true that no individual lab wants.

this but what are they going to do drop. out of the race if they drop out of the. race then their competitors are just. going to keep going right like it's so. messed up you can literally be looking. at like the cliff that you're driving. towards and be like I do not have the. agency in this system to steer the wheel. I I I do think it's worth highlighting. too it's it's not like I was say it's. not all doom and gloom which is a great. thing to say after all that's easy for. you guys to say well well part of it so.

part of it is that we actually have been. spending the last two years trying to. figure out like what do you do about. this that was the the action plan that. came came out after the investigation. and it was basically a series of. recommendations how do you balance. Innovation with like the risk picture. keeping in mind that like we don't know. for sure that all this shit's going to. happen we have to navigate an. environment of deep uncertainty the. question is what do you do in that. context so there's you know a couple. things like we need you know a licensing. regime because eventually you can't have.

just literally anybody joining in the. race if they don't adhere to certain. best practices around cyber around. safety other things like that you need. to have some kind of legal liability. regime like what happens if you don't. get a license and you say yeah [ __ ] that. I'm just going to go do the thing anyway. and then something bad happens and then. you're going to need like an actual. regulatory agency and this is something. that we you know we don't recommend. lightly cuz Regulatory Agencies suck we. don't like them but but the reality is. this field changes so fast that like if. you think you're going to be able to. enshrine a set of best practices into.

legislation to deal with this stuff it's. just not going to work and so when we. talk to Labs whistleblowers the the WD. folks in natsec and the government. that's kind of like where we land and. it's something that I think at this. point you know Congress really should be. looking at like there should be hearings. focused on what does a framework look. like for liability what does a framework. look like for licensing and actually. exploring that because we've we've done. a good job of studying the problem right. now like Capitol Hill has done a really. good job of that um it's it's now kind.

of time to get that next beat and I. think there's the the Curiosity there. the intellectual curiosity there's the. humility to do all that stuff right but. the the challenge is just actually. sitting down having the hearings doing. the investigation for themselves to look. at Concrete Solutions that treat these. problems as seriously as the water. cooler conversation at the frontier Labs. would would have us treat them at the. end of the day this is going to happen. at the end of the day it's not going to. stop at the end of the day these systems. whether they're here or abroad they're.

going to continue to scale up and. they're going to eventually get to some. place that's so alien we really can't. imagine the consequences yep and that's. going to happen soon that's going to. happen within a decade right we we may. again like the the stuff that we're. recommending is approaches to basically. allow us to continue this scaling in as. safe a way as we can so basically a big. part of this is just being able having.

actually having a scientific theory for. what are these systems going to do what. are they likely to do which we don't. have right now we scale another 10x and. we get to be you know surprised it's a. fun guessing game of what are they going. to be capable of next we need to do a. better job of incentivizing a deep. understanding of what that looks like. not just what they'll be capable of but. what their you know their propensities. are likely to be the the control problem.

and solving that that's that's kind of. number one and and to be clear there's. amazing progress being made on that. there is a lot of progress it's just a. matter of of switching from the like. build first ask questions later mode to. like we're calling it like safety. forward or whatever but it basically is. like you start by saying okay here are. the properties of my system how can I. ensure that my development guarantees. that the system Falls within those. properties after after it's built so you. kind of flip the Paradigm just like you. would if you were designing any other uh. lethal capability potentially just like.

DOD does you start by Def defining the. bounds of the problem and then you. execute against that um but to your. point about where this is going. ultimately you know there is literally. no way to predict what the world looks. like like you were saying in a decade. like yeah jez I think what like one of. the the weirdest things about it and and. one of the things that worries me the. most is like you look at the the. beautiful coin coincidence that's given. uh America's current shape right that. coincidence is the fact.

that a country is most powerful. militarily if its citizenry is free and. empowered that's a coincidence didn't. have to be that way hasn't always been. that way it just happens to be that when. you let people kind of do their own [ __ ]. they innovate they come up with great. ideas they support a powerful economy. that economy in turn can support a. powerful military a powerful kind of. international presence um. when you have so that happens because. decentralizing all the computation all.

the thinking work that's happening in a. country is just a really good way to run. that country top down just doesn't work. because human brains can't hold that. much information in their heads they. can't reason fast enough to centrally. plan an entire economy we've had a lot. of experiments in history that show that. AI may change that equation it may make. it possible for like the central. planner's dream to come true in some. sense which then disempowers the. citizenry and there's a real risk that. like I don't know we're we're all.

guessing here but like there's a real. risk that that beautiful coincidence. that gave rise to the the success of the. American experiment ends up being broken. uh by technology and that seems like a. really bad thing that's one of my. biggest fears because this essentially. the United States like the Genesis of it. in part is like it's a it's a knock-on. effect centuries later of like the. printing press right the ability for. like someone to set up a printing press. and print like whatever you know. whatever they want free like free.

expression is at the root of that what. happens yeah when you have a revolution. that's like the next the next printing. press we should expect that to have. significant and profound impacts on how. like things are governed and one of my. biggest fears is that the the great like. the the the the the like you said the. greatness that the the moral greatness. that I think is you know part and parcel.

of how the United States is constituted. culturally that that the link between. that and actual capability and. confidence and impulse gets eroded or. broken and you have like the potential. for very centralized authorities to just. be more successful and that's like that. that does keep me up at night that is. scary especially in light of like the. Twitter files where we know that the FBI.

was interfering with social media and if. they get a hold of a system that could. disseminate propaganda in kind of an. Unstoppable way they could push. narratives about pretty much everything. depending upon what their financial or. you know geopolitical motives are and. one of the challenges is that the. default course if you so if if we do. nothing relative to what's happening now. is that that same thing happens except. that the entity that's doing this isn't. you know some government it's like I. don't know Sam mman open AI whatever.

group of Engineers happen to be close. Evil Genius that reaches the top and. doesn't let everybody know he's at the. top yet just starts implementing it and. there's no sort of guard rails for that. currently yeah like and that's and. that's like that's one of the that's a. scenario where that little cabal or. group or whatever actually can keep the. system under control and that's not. guaranteed either yeah right. are we giving birth to a new life form I. think at a certain point that's a it's a. philosophical question that's above so I.

was going to say it's above my pay grade. the problem is it's above like literally. everybody's pay grade um I think it's. not unreasonable at a certain point to. be like like yeah I mean look if you if. you think that you know the human brain. um gives rise to Consciousness because. of nothing magical it's just the. physical activity of information. processing happening in our heads then. why can't the same happen on a different. substrate a substrate of silicon rather. than cells like there's no clear reason. why that shouldn't be the case um if.

that's true yeah I mean life form what. by whatever definition of life because. that itself is controversial I think by. now quite outdated too uh should be on. the table you maybe should start to. worry as a lot of people in the industry. will say this too like you know behind. closed doors very openly yeah and we. should start to worry about moral. patienthood as they put it there's. literally one of the top people at one. of these. Jeremy I think you had a conversation. with him and he's like yep we're going. to have to start worrying about this and. that definitely made us go like.

w i mean it seems inevitable I I've. described human beings as an electronic. caterpillar that we're like a. caterpillar a biological caterpillar. that's giving birth to the electronic. butterfly and we don't know why we're. making a cocoon and it's tied into. materialism because everybody wants the. newest greatest thing so that fuels. Innovation and people are constantly. making new things to get you to go by. them and a big part of that is. technology yeah and actually so it's. linked to this question of of.

controlling AI systems in a kind of. interesting way so one way you can think. of of humanity is as like this you know. super organism you got all the human. beings on the face of the Earth and. they're all acting in some kind of. coordinated way the mechanism for that. coordination can depend on the country. you know free markets uh capitalism. that's one way top down is another but. you know roughly speaking you've got all. this coordinated vaguely coordinated. Behavior but the result of that behavior. is not NE necessarily something that any. individual human would want right like. you look around you walk down the street. in Austin you see skyscrapers and [ __ ].

clouding your your vision there there's. all kinds of pollution and all that and. you're like well this kind of sucks but. if you interrogate any individual person. in that whole causal chain and you're. like why are you doing what you're doing. well locally they're like oh this makes. tons of sense it's because I do the. thing that gets me paid so that I can. live a happier life and so on and yet in. the aggregate um not now necessarily but. as you keep going it just for forces us. like compulsively to keep giving rise to. these more and more powerful systems and. in a way that's potentially deeply.

disempowering that's the race right like. that's like yeah I like I it comes back. to the idea that I the comp I an AI. company I maybe don't want to be. potentially driving towards a cliff but. I don't have the agency to to like steer. so yeah but I mean everything's fine. apart from that yeah we're good we're. good oh it's such a terrifying. prognosis there are again we like we. wrote a 280 page document about like.

okay and here's what we can do about it. I can't believe you didn't read the 200. I started reading it but I passed out. but does any of these or do any of these. safety steps that you guys want to. implement do they inhibit. progress they they're definitely you. create you know anytime you have. regulation you're going to create. friction to some extent there's it's. kind of inevitable one of the key like.

center pieces of the approach that that. we outline is you need the flexibility. to move up and move down as you notice. the risks appearing or not appearing so. one of the key things here is like you. you need to cover the worst case. scenarios because the worst case. scenarios yeah they could potentially be. catastrophic so those got to be covered. but at the same time you can't. completely close off the possibility of.

the happy path like the the like we. can't lose sight of the fact that like. yeah all this [ __ ] is going down or. whatever we could be completely wrong. about the outcome it could turn out that. like for all we know it's a lot easier. to control these systems at this scale. than we than we imagined it could turn. out that you know it is like you get you. know maybe some kind of ethical impulse. gets embedded in the system naturally. for all we know that might happen and.

it's really important to at least have. your regulatory. system allow for that possibility. because otherwise you're foreclosing the. possibility of what might be the best. future that you could possibly imagine. for everybody I got to imagine that the. military if they had hindsight if they. were looking at this they said we should. have got on board a long time ago and. and kept this inhouse and and kept it. squirreled away where it wasn't publicly.

being discussed and you didn't have open. AI you didn't have all these people like. if they could have gotten on it in. 2015 so so this is actually deeply tied. to how the economics of Silicon Valley. work um this is and it's AI is not a. special case of this right you have a. lot of cases where technology just like. takes Everybody by surprise and it's. because when you go into Silicon Valley. it's it's all about people placing these. outsized bets on what seem like like. tale events like things that are very. unlikely to happen but with a you know.

at first a small investment and. increasingly you know growing investment. as the thing gets proved out more and. more very rapidly you're can to have a. solution that seems like complete. Insanity that just works and this is. definitely what happened in the case of. AI so 2012 like we did not have this. this whole picture of like an artificial. brain with artificial neurons this whole. thing that's been going on that's like. it's 12 years that that's been going on. like that was really kind of shown to. work for the first time roughly in 2012. ever since then it's just been people. kind of like you can trace out the.

genealogy of like the very first. researchers and you can basically. account for where they all are now you. know what's crazy is if that's 2012. that's the end date of the Mind calendar. that's the thing that everybody said was. going to be the end of the world that. was the thing that Terrence McKenna. banked on it was December 21st. 2012 if because this was like this Goofy. conspiracy theory but it was based on. the long count of the M calendar where. they summize that this is going to be. the end of civilization the beginning of. the end Joe what if that if it is 2012.

how wacky would it be if that really was. the beginning of the end that was the. like they don't measure when it all. falls apart they me measure the actual. mechanism like what started in motion. when it all fell apart and that's 2012. well that's and then not not to be a. dick and like ruin the 2012 thing but. like neural networks were also kind of. they were floating around a little bit. bit I'm kind of being dramatic when I. say 2012 that was definitely an. inflection point it was the it was this. model called Alex net that first it did.

like the first useful thing the first. time you had a computer vision model. that actually worked um but I mean it is. fair to say that was the moment that. people started investing like crazy into. the space so that's what changed it yeah. just like the Mayans foretold all that. time they knew it they knew it like. these monkeys they're going to figure. out how to make better people yeah you. can actually look at the like. hieroglyphs or whatever and there's like. neural networks. imagine if they discovered that you. you've got to wonder what happens to the.

general population people that work. menial jobs people that their life is. going to be taken over by Automation and. how susceptible those people are going. to be they're not going to have any. agency they're going to be relying on a. check and this idea of like going out. and doing something it used to be learn. to code right but that's out the window. cuz nobody needs to code now cuz AI is. going to code quicker faster much better. no errors you're you're going to have a. giant swath of the population that has. no purpose I think that's actually like.

a completely real I was watching um this. like talk by a bunch of open AI. researchers a couple days ago and um it. was recorded from from a while back but. they were basically saying they they. were exploring exactly that question. right because they ask themselves that. all the time and their attitude was sort. of like well yeah I mean I uh I guess. it's going to you know suck or whatever. um like we'll we'll probably be okay for. longer than most people because we're. actually building the thing that.

automates the thing um maybe there going. to be some they like to get fancy. sometimes and say like oh now uh you. could do some thinking of course to. identify the jobs that'll be most secure. and uh and it's like do some thinking to. identify the J like like what if you're. a like you're a janitor you're you're. like a freaking plumber you're going to. just change your like how how's that. supposed to work do some thinking. especially if you have a mortgage and a. family and. the so they like the only solution and. this happens so often like there really. is no plan that's the the single biggest.

thing that you get hit over the head. with over and over whether it's talking. to the people who are in charge of the. the like labor transition their whole. thing is like yeah Universal basic. income and then question mark and then. smiley face that's basically the three. steps that they Envision um it's the. same when you look internationally like. how are we going to like okay tomorrow. you build an AGI that's like in. incredibly powerful potentially. dangerous thing what is the plan like. how are you going to like I don't know.

you're going to secure it share it. figure it out as we go along man that's. all that's the freaking message like. that that is the entire plan the scary. thing is that we've already gone through. this with other things that we didn't. think were going to be significant like. data like Google like Google search like. data became a a valuable commodity that. nobody saw coming yeah you know just the. influence of social media on General. discourse it's completely changed the. way people talk it's it's so easy to. push a thought or an ideology through.

and it could be influenced by foreign. countries and we know that happens and. it is happening at a huge scale already. this is like and we're in the early days. of you know we mentioned manipulation of. social media with like you can just do. it so the the wacky thing is like the. very best models now are you know. arguably smarter in terms of the post. that they put out the potential for. virality and just optimizing these. metrics then know maybe like the I don't.

know the the the dumbest or laziest like. quarter of Twitter users like in. practice most people who read on Twitter. is like don't really care they're. trolling or they're doing whatever but. as that water line goes up and up and up. like who's saying what right it also. leads to like this challenge of. understanding what the lay of the land. even is like we've gotten into so many. debates with with people where they'll. be like look um everyone always has. their magic thing that AI like I'm not. going to worry about it until AI can do.

thing X right for some people that I I. had a conversation with somebody a few. uh weeks ago and they were saying um I'm. going to worry about automated cyber. attacks when um when I actually see an. AI system that can write good malware. and uh like that's already a thing that. happens so so this happens a lot where. people will be like I'll worry about it. when it can do X and you're like yeah. yeah that happened like 6 months ago but. the field is moving so crazy fast that. you could be forgiven for for messing. that up unless it's your full-time job.

to track what's going on so it like you. you kind of have to be anticipatory. there there's no it's kind of like the. co example like everything's exponential. yeah you're going to have to do things. that seem like they're you know more. aggressive more forward-looking than you. might have expected given the current L. late of the land but that's just drawing. straight lines between you know between. two points Because by the time you've. executed the world is already shifted. like goal post have shifted further in. that direction and that's actually. something we yeah we we do in the the. report in the action plan in terms of. the recommendations one of the good.

things is we are already seeing movement. across the US government that's aligned. with those recommendations in a big way. and it's really encouraging to see. that you're not making me feel. better I love all this encouraging talk. but I just am I'm this I'm playing this. out and I'm I'm seeing the overlord you. know and uh I'm seeing president AI. because it won't be affected by all the. issues that we we're seeing with current. president dude it's it's super hard to.

imagine a way that this plays out like I. I think it's important to be. intellectually honest about this and and. I think any I would really challenge. like the leaders of any of these. Frontier labs to describe a future um. that is stable and multi-polar where you. know there's there's like more we where. like Google's got like an AGI and open. AI has got an AI and like like and and. really really bad [ __ ] doesn't happen. every day like I mean that's that's the. challenge and so you know the question.

is how can you tee things up ultimately. such that there's as much Democratic. oversight as much you know the public is. as empowered as it can be that's the. kind of situation that we need to be. having I think there's this like game of. smoke and mirrors that sometimes gets. played at least you could interpret it. that way where people lay out these. you'll notice it's always very fuzzy. visions of the future every time you get. a the kind of like here's where we see. things going it's going to be wonderful. the technology is going to be so so. empowering think of all the diseases.

will cure all of that is 100% true and. that's actually what excites us it's why. we got into AI in the first place it's. why we build these systems but um really. you know challenging yourself to try to. imagine how do you get stability and. highly capability eye systems uh in a. way with where the public is actually. empowered those three ingredients really. don't want be in the same room with each. other and so actually confronting that. head on I mean that's what we try to do. in the the action plan I think it we try.

to solve for one one aspect of that so. the whole like I mean you've you're. you're right this is a a whole other can. of worms is like how do you govern a. system like this not just from a. technical standpoint but like who votes. on like what it how does that even work. and so that entire aspect like that we. didn't even touch. all that we focused on was like the. problem set around how do we get to a. position where we can even attack that.

problem where we have the technical. understanding to be able to aim these. systems at that level in any direction. whatsoever and to be clear like like we. are both actually a lot more optimistic. on our on the prospect of that now than. we ever were um there's been a ton of of. progress in the control understanding of. these systems even actually even in the. last week um but just more broadly in. last year I did not expect that we'd be. in a position where you could you could.

plausibly argue we're going to be able. to kind of X-ray and understand the the. innards of these systems you know over. the next couple years like year or two. hopefully that's you know good enough. time Horizon but this is part of the. reason why you do need the the. incentivization of that safety forward. approach where it's like first you got. to invest in yeah secure and and kind of. interpret and understand your system. then you get to build it because. otherwise we're just going to keep. scaling and like being surprised at. these things they're going to keep. getting stolen they're going to keep. getting open- sourced and you know the.

stability of our our our critical. infrastructure the stability of our. society don't necessarily age too well. in that. context could best case scenario be that. AGI actually mitigates all the human. [ __ ] like puts a stop to propaganda. highlights actual facts clearly where. you can go to it where you no longer. have corporate State controlled news you. don't have news controlled by media.

companies that are influenced heavily by. special interest groups and you just. have the actual facts and these are the. motivations behind it and this is where. the money is being made and this is why. these things are being implemented the. way they're being and you're being. deceived based on this that and this and. this has been shown to be propaganda. this has been shown to be complete. fabrication this is is actually a deep. fake video This is actually AI created. technologically that is absolutely on. the table Yeah best case scenario that's.

best case scenario absolutely yes what's. worst case scenario I mean like actual. worst case scenario I like your face. like I mean we're talking. like pushing it so it's like you think. about it right like we're we're you know. end of the world as we know it and I. feel fine. except it'll sound like Scarlet Johanson. but yes yeah that's right it's going to. be her I didn't think it sounded that. much like her we played it and I was.

like I don't know we listened to the. clip from her and then we listen to the. thing I'm like kind of like a girl from. the same part of the world like not. really you like that's kind of. cocky that's true I mean I uh the fact. that I guess Sam reached out to her a. couple of times kind of makes it a. little a little weird twe tweeted the. word her right but they also did say. that they had gotten this woman under. contract before they even reached out to. Scarlet Johansson so if that's true yeah.

that was I think it's kind of. complicated right so open AI uh. previously put out a statement where. they said explicitly and this was not in. connection with this this was like. before when they were talking about the. prospect of of human of AI generated. voices oh that was in March of this year. yeah yeah but it was like well before. the the scarjo stuff or whatever hit the. um and they were like they said. something like um look no matter what we. got to make sure that there's. attribution if somebody's you know.

somebody's voice is being used and we. won't we won't do the thing where we. just like use somebody else's Voice who. kind of sounds like some someone whose. voice we're trying to like they they. literally like M that's funny because. they said what they were thinking about. doing we won't do that that's a good way. to cover Your Tracks we will never do. that why would I ever take your Buddha. statue Joe I'm never going to do that. that would be an insane thing to do the. [ __ ] Buddha statue yeah I think uh. that that's a small discussion you know. the Scarlet Johansson voice like. whatever she should have just taken the. money but it would have been fun to have.

her be the voice of it be kind of hot. but the whole thing behind it is the. mystery the whole thing behind it is is. just it's just pure speculation as to. how this all plays out we're really just. guessing yeah which is one of the. scariest things for the Lites people. like myself like sit on the sidelines. going what is this going to be like. scary yeah I mean it's scary for like we. are we're very much honestly like we're. optimists across the board in terms of.

technology and it's scary for us like. what happens when you have when you. supersede kind of the whole spectrum of. what a human can do like what am I going. to do with myself what's you know what's. my daughter going to do with herself. like I don't know yeah yeah and I think. a lot of these questions are um when you. look at the the culture of these labs. and the kinds of people who are pushing. it Forward um there is a a strand of.

like uh transhumanism within the labs. it's not everybody uh but that's. definitely the population that initially. seated this like if you look at the. history of AI like who are the first. people to really get into this stuff. like I know you had Ray kurtwell on and. uh you know other folks like that who in. many cases see um to to roughly. paraphrase and not everybody sees this. way but like we want to get rid of all. of the biological sort of threads that. tie us to this physical reality you know.

shed our meat machine bodies and and all. this stuff there is a threat of that at. at a lot of the frontier Labs like. undeniably there's a population it's not. tiny it's it's definitely a subset um. and for some of those people you. definitely get a sense interacting with. them that there's like almost a kind of. Glee at the prospect of of building AGI. and all this stuff almost as if it's. like this evolutionary imperative and in. fact Rich Sutton who's uh the founder of. this field called reinforcement learning. which is a really big uh an important.

space um you know he's an advocate for. what he himself calls like succession. planning he's like look uh this is going. to happen uh it's kind of desirable that. it will happen and so we should plan to. hand over power to Ai and phase. ourselves out and oh God that's well. that that's the thing right like and. when Elon talks about you know he's. having these arguments with Larry Ag and. um you know the yeah like you you're you. know calling Elon like a speciesist.

speciesist yeah hilarious I mean I will. I will be a speciesist I'll take. speciesist all day like what are you. [ __ ] talking about you let your kids. get eaten by Wolves no you're a. speciesist yeah that's the thing yeah. like this is stupid but it but it this. is like a weirdly INF and when you look. at like the um effective accelerationist. movement in the valley there's a part of. it that's and I got to be really careful. too like these ments have valid points. like you can't you can't look at them be. like oh yeah it's just all bunch of like. you know these transhumanist types. whatever but there is there's a a strand.

of that a thread of that um and a kind. of like uh there's this like U I don't. know I almost want to call it this like. Teenage rebelliousness where it's like. you can't tell me what to do like we're. just going to build a thing and and I. get it I I really get it I'm very. sympathetic to that um I love that ethos. like libertarian ethos in Silicon Valley. is really really strong for for building. Tech it's helpful uh there are all kinds. of points and counter points and you. know the left needs the right and the. right needs the left and all this stuff. but um in in the context of this problem. set it can be very easy to get carried.

away in like the utopian vision and I. think there's a lot of that kind of. driving the train right now in this. space yeah those guys freak me out I. went to a 20 45 conference once in New. York City where they were uh one guy had. like an a robot version of himself and. they were all talking about downloading. Consciousness into computers and 2045 is. the year they think that all this is. going to take place which obviously. could be very ramped up now with AI yeah.

but this this idea that somehow or. another you're going to be able to take. your Consciousness and put it in a. computer and make a copy of yourself and. then my question was well what's going. to stop a guy like Donald Trump from. making a billion Donald Trumps you know. like you you know I me true right if you. can what about Kim jung-un you're going. to let him make a billion versions of. himself like what does that mean where. do they where do they exist yeah and is. that the Matrix are they existing in. some sort of virtual are we going to. dive into that because it's going to be. rewarding to our senses and better than.

being a meat thing I mean if you think. about the the constraints right that we. face as meat machine whatevers like yeah. you get hungry you get tired you get. horny you get sad you know all these. things what if yeah what if you could. just hit a button and just Bliss nothing. but bliss all the time why take the lows. Ed right you don't need no lows. oh yeah you remember ride the wave of a. constant drip yeah man you remember in. The Matrix where uh the first Matrix.

where the guy like betrays them all and. he's like ignorance is bliss man Joey. pants he's eating the steak and he says. I just want to be an important person. that's it that's it like boy so I mean. part of it is like what do you think is. actually valuable like if you zoom out. you want to see you know human. civilization 100 years from now or. whatever it may not be human. civilization if not what you value um or. if it can actually eliminate suffering. right M I mean why why exist in a. physical sense if it just entails.

endless suffering but in what form right. what do you value because I again I can. rip your brain out I can you know pickle. you I can like Jack you full of. endorphins and I've eliminated your. suffering that's what you wanted right. right that's the problem that's the. problem it's one of the problems yes. yeah one of the problems is it could. literally lead to the elimination of the. human race because if you could stop. people from breeding I've always said. that if China really wanted to get. America they really wanted to like if. they had a long game just give us sex. robots and free food free food free.

electricity sex robots it's over just. give people free housing free food sex. robots and then the Chinese Army would. just walk in on people laying in puddles. of their own jizz there would be no one. doing anything no one would bother. raising children that's so much work. when you can you know dude that's in the. action. plan that's uh I mean all you have to do. is just keep us complacent just keep us.

satisfied with experience Tik Tok man. it's video games as well yeah you know. video games even though they're a thing. that you're doing it's so much more. exciting than real life that you have a. giant percentage of our population. that's spending 8 10 hours every day. just engaging in this virtual world. already happening with oh sorry yeah no. it's like it's you can you can create an. addiction with pixels on a screen that's. messed up and addiction like with pixels. on a screen with social media doesn't. even give you much yeah it's not like a.

video game gives you something you feel. like oh [ __ ] you're running away Rockets. are flying over your the things are. happening you got 3D sound massive. Graphics this is [ __ ] you're. scrolling through pictures of a girl. doing deadlifts like what is this like. you feel as bad as after that as with. your brain as you would feel after. eating like six like ERS or whatever my. friend Sean said it best Sean Ali the. UFC champion he said I get a lowlevel. anxiety when I'm just scrolling like. yeah what is that like what and for no.

reason well the reason is that some of. the world's best phds and data. scientists have been given millions and. millions of dollars to make you do. exactly that and increasingly some of. the best algorithms too like and you're. starting to see that handoff happen so. so this there's this one thing that we. we talk about a lot in the context and. Ed brought this up um in the context of. sales and like the persuasion game right. we're okay today like as a civilization. we have agreed implicitly that it's okay. for all these phds and [ __ ] to be you.

know spending millions of dollars to. hack your child's brain that's actually. okay if they want to sell like a rice. krispy cereal box or whatever that's. cool um what we're starting to see is AI. optimized ads because you can now. generate the ads you can kind of close. this Loop and have an automated feedback. loop where the ad itself is getting. optimized with every impression not just. which ad which human generated ad gets. to which person but the actual ad itself. kind like the creative the copy the. picture the text like a living document. now and for every person and so now you.

you look at that and it's like that. versus your kid that's an interesting. thing and you start to think about as. well like sales that's a really easy. metric to optimize it's a really good. feedback metric they click the ad they. didn't click the ad so now you what. happens if you know you you manage to. get a clickthrough rate of like 10% 20%. 30% how high does that success rate have. to be before we're really being robbed. of our agency I mean like there's a. threshold where it's sales and it's good. and some persuasion in sales is.

considered good often it's it's actually. good because you'd rather be advertised. at by a relevant ad that's a service way. right sure something I'm actually. interested in why not right you don't. want to see ad for for light bulbs but. right when when you get to the point. where it's like yeah 90% of the time. like or or 50 or whatever what's that. threshold where all of a sudden we are. stripping people especially minors but. also adults of their agency and it's. it's really not clear AI there are loads. of like canaries in the coal mine here. in terms of even relationships with like. AI chat Bots right they're been suicides.

people who build relationships with an. AI chat bot that tells them hey you. should end this um I don't know if you. guys saw the like on Rea like there's a. subreddit this U model called Rea that. would kind of build a relationship a. chatbot build a relationship with users. and one day Rea goes oh yeah like the. all the kind of sexual interactions that. users have been having you're not. allowed to that anymore bad for the. brand or whatever they decided they cut. it off oh my God you go to the subreddit. and it's like you'll read like these.

gut-wrenching accounts from people who. feel genuinely like they've had a loved. one taken away from them it's her yeah. it's her ex it really is her but just. I'm dating a model means something. different in 2024 oh yeah it really does. my friend Brian he was on here yesterday. and he had this uh he has this thing. that he's doing with uh like a fake. girlfriend friend that's an AI generated. girlfriend that's a [ __ ] like this girl. will do anything and she looks perfect. she looks like a real person he'll like.

take a picture of your [ __ ] in the in. the kitchen and and he'll get like a. highresolution photo of a really hot. girl bending over sticking her ass at. the camera and is it sorry and it's. Scarlett Johansson's [ __ ] no you. could probably make it that though I. mean it's basically like he got to pick. like what he's interested in and then. that girl it just gets created healthy. like that [ __ ] nuts [ __ ] nuts now. here's the real question this is just. sort of a surface layer of interaction.

that you're having with this thing it's. very two-dimensional you're not actually. encountering a human you're getting text. and and. pictures what is this going to look like. virtually now the virtual space is still. like pong you know it's not that good. even when it's good like uh Zuckerberg. was here and he gave us the latest. version of the the the headsets and we. were playing we were fencing it's pretty.

cool you you could actually go to a. comedy club they had a stage set up like. wow it's kind of crazy but it's the the. the you know the the gap between that. and accepting it as real is pretty far y. but that could be bridged with. technology really quickly haptic. feedback and especially some sort of a. neural interface whether it's neural or. some something that you wear like that. Google one where the guy was wearing it. and he was asking questions and he was. getting the answers fed through his head. so he got answers to any questions when.

that comes about when you're getting. sensory input and then you're having. real life interactions with people as. that scales up exponentially it's going. to be IND discernible which is the whole. simulation hypothesis there oh no go for. it well I was going to say there so on. the simulation hypothesis there's like a. another way way that could happen that. is maybe uh even less dependent on. directly plugging into like human brains. and all that sort of thing um which.

is so every time we don't know and and. this is super speculative I'm just going. to carve this out as this Jeremy's being. super like guess work here nobody knows. go for it Jeremy gidy up so um you've. got this this idea that every time you. have a model that generates an output. it's having to kind of tap into a a. model a kind of mental image if you will. of the way the world is it kind of in a. in a sense you could argue instantiates.

maybe a simulation of how the world is. in other words to take it to the extreme. not saying this is what's actually going. on in fact I would even say this is. probably this sorry this is certainly. not what's going on with current models. but eventually maybe who knows every. time like you generate the next word in. the in the token prediction you're. having to like load up this entire. simulation. maybe of all the data that the model has. ingested which could basically include. like all of known physics at a certain. point like I mean again super.

speculative but it's that literally. every every token that the the chatbot. predicts could be associated with a a. standup of an entire simulated. environment who knows not saying this is. the case but just like when you think. about what is the mechanism that would. produce the most simulated worlds as. fast most accurate also the most. accurate prediction like if you f. simulate you know a world that's. potentially going to give you very. accurate predictions yeah like it it's. possible um but it it kind of speaks to.

that question of like of Consciousness. too like right what is it yeah no idea. we're very cocky about that yeah yeah I. mean and there's emerging evidence that. plants are not just Consciousness but. they actually communicate which is real. weird because like then what is that if. it's not in the neurons if it's not in. the brain and it exists in everything. was does it exist in soil is it in tree. is it what is a butterfly. thinking just have a limited capacity to. express itself we're so ignorant of well.

we're we're also very arrogant you know. because we're the [ __ ] CU we're people. you know Bingo there there's a a which. allows us to have the hubris to make. something like AI yeah and the worst. episodes in the history of our species. are I think like Jeremy said have been. when. we looked at others. as though they were not people and. treated them that way and you can kind. of see how so I don't know there's when. you look at like what humans think is.

conscious and what humans think is not. conscious there's a lot of um there's a. lot of like human chauvinism I guess you. could call it that goes into that like. we look at a dog we're like oh it must. be conscious because it licks me it it. seems it acts as if it loves me right. there are all these outward indicators. of you know a mind there but when you. look at like you know cells cells. communicate with their environments in. ways that are completely different and. alien to us right you know their inputs. and outputs and all that kind of thing. you can also look at the higher scale. the human super organism we talked about.

all those human beings interacting. together to form this like you know. planewide organism what is that thing. conscious is there some kind of. Consciousness we could ascribe to that. and then what the [ __ ] is spooky action. at a distance you know what's going on. at a Quantum you know when you get to. that it's like okay what are you saying. like these things are expressing. information faster than the speed of. light what dude you're trying to trigger. my my Quantum my Quantum fuzzies here. this guy this guy did grad school in. quantum mechanics pleas I'm really sorry. well how Bonkers is it oh like a it's.

like a seven it's like a seven yeah it's. is very Bon so uh okay. there's one of the problems right now. with with physics is that we have so. imagine all the DAT all the experimental. data that we've ever collected um you. know all the buns and burner experiments. and all the ramps and cars sliding down. inclines whatever that's all a body of. data um to that data we're going to fit. some theories right so we're going to.

fit basically U Newtonian physics is a. theory that we try to fit to that data. to try to like explain it uh Newtonian. physics breaks because it doesn't. account for a lot of those observations. a lot of those data points quantum. physics is a lot better but there's like. some weird areas where it still doesn't. like quite fit the the bill but it. covers an awful lot of those data points. the problem is there's like a million. different ways to tell the story of what.

quantum physics means about the world. that are all mutually inconsistent like. these are the different interpretations. of the theory some of them are say that. yeah they're parallel universes some of. them say that human consciousness is. Central to physics uh some of them say. that like the future is predetermined. from the past and all of those theories. fit perfectly to all the points that we. have so far but they they tell a. completely different story about what's. true and what's not and some of them. even have something to say about for. example Consciousness and so in a weird.

way like the fact that we haven't. cracked the nut on any of that stuff. means for like we really have have no. shoted understanding the the. Consciousness equation sence equation. when it comes to like AI or whatever. else I mean we're but for for Action at. a distance like one of the spooky things. about that is that you can't actually. get it to communicate anything concrete. at a. distance everything about the laws of. physics conspires to stop you from.

communicating faster than light. including what's called action at a. distance as as far as we currently far. as as far as we know and that's that's. the problem so if if you look at the. leap from like Newtonian physics to. Einstein right with new with Newton um. we're able to explain a whole bunch of. [ __ ] the world seems really simple it's. forces and its masses and and that's. basically it you got objects um but then. people go oh look at like the orbit of. mercury it's a little wobbly we got to.

fix that and it turns out that if you're. going to fix that one stupid wobbly. orbit you need to completely change your. whole picture of what's true in the. world all of a sudden you've got a world. where space and time are linked together. you have to they get bent by gravity. they get bent by energy uh there's all. kinds of weird [ __ ] that happens with. time and links like all that stuff all. just to account for this one stupid. observation of the wobbly orbit of. freaking Mercury and the challenge is.

this might actually end up being true. with with quantum mechanics in fact we. like we know quantum mechanics is broken. because it doesn't actually fit with our. theory of general relativity from. Einstein we can't make them kind of play. nice with each other at certain scales. and so there's our wobbly orbit so now. if we're going to solve that problem if. we're going to create a unified theory. we're going to have to step outside of. that and almost certainly or it seems. very likely we'll have to refactor our. whole picture of the universe in a way. that's just as fundamental as the leap. from nton to Einstein this is where.

Scarlet Johansson comes in say boys I. can do this you don't have to do this I. can take this off your hands let me. solve all physics really complicated but. because you have a simian brain you have. a little monkey brain that's just like. super Advanced but it's really shitty. you know what that's harsh but it. sounded really hot yeah especially if. you have the horse Scarlet Johansson. from her like the Bedtime. voice so you're you're the one that they. got to do the voice of Sky yes it's me.

that was you oh dude did my girl voice. on on the uh on the sexiness of scarlet. Johansson's voice um so so open AI at. one point said I can't remember if it. was Sam or or open AI itself they were. like hey so we're not the one thing. we're not going to do is like optimize. for engagement with our products. and when when I first heard the sexy. sultry seductive Scarlet Johansson voice. and I finished cleaning up my pants I.

was like damn uh that seems like. optimization for something I don't know. if like it right otherwise you'd get. Richard Simmons to do the voice. exactly you turn people on there's a lot. of other options that's like that's. that's an optimization for like growth. of Google's thing yeah like oh well. let's see what Google's got yeah. Google's got to do. Google's got rich Simmons yeah what are. they going to do um boy. so do you think that AI with if it does.

get to an AGI Place could it possibly be. use to solve some of these puzzles that. have eluded our Simple Minds totally. totally I mean the ad the potential. advancements Ai No it's like it's so. potentially positive and even before AGI. because remember we talked about how. these systems make mistakes that are.

totally different from the kinds of. mistakes we make right and so what that. means is we make a whole bunch of. mistakes that an AI would not make. especially as it gets closer to our. capabilities and so um I was reading. this uh this this thought by Kevin Scott. who's the CTO of Microsoft he has made a. bet with a number of people that you. know in the next few years an AI is. going to solve this particular. mathematical theorem conjecture called. the reman hypothesis it's like you know.

how spaced out are the prime numbers. whatever some like mathematical thing. that for a hundred years plus people. have just like scratched their heads. over these things are incredibly. valuable his expectation is it's not. going to be an AGI it's going to be a. collaboration between a human and an AI. even on the way to that before you hit. AGI there's a ton of value to be had. because these systems think so fast. they're tireless compared to us like.

they have different view of the world. and can solve problems potentially in. interesting ways so yeah like there's. tons and tons of positive value there. and and even that we've already seen. right like past performance man like yes. I'm I'm almost tired of using the phrase. like just in the last month because this. keeps happening but um in the last month. so uh Google deep mine came out with or. and isomorphic Labs cuz they're working. together on this but they came out with. Alpha fold 3 so Alpha fold 2 uh was the. first so let me take a step back there's. this really critical problem in.

molecular biology where you have so. proteins which are just a it's a. sequence of building blocks the building. blocks are called amino acids and each. of the amino acids they have different. structures and so once you finish. stringing them together they'll. naturally kind of fold together in some. interesting shape and that shape gives. that overall protein its function so if. you can predict the shape the structure. of a protein based on its amino acid. sequence you can start to do [ __ ] like. design new drugs you can solve all kinds.

of problems like this is like the. expensive Crown Jewel problem of the. field Alpha fold 2 in one swoop was like. oh like we can we can solve this problem. basically um much better than a lot of. even empirical methods now Alpha full 3. comes out they're like yeah and now we. can do it if we on a bunch of yeah there. it is if we can tack on a bunch look at. this quote Alpha fold 3 predicts the. structure and interactions of all of.

life's. molecules what in the [ __ ] kids of. course introduced Alpha fold 3. introducing rather Alpha fold 3 a new AI. model devel developed by Google Deep. Mind and. isopor how you say is isomorphic. isomorphic lab by accurately predicting. the structure of proteins DNA RNA. lians lians Li lagin and more and how. they interact we hope it will transform. our understanding of the biological.

world and Drug Discovery so this is like. just your typical Wednesday uh in the. world of AI right because it's happening. so quickly it's happening yeah that's it. so it's like oh yeah another Revolution. happened this month and it's all. happening so fast and our timeline is so. flooded with data that everyone's kind. of unaware of the pace of it all that. it's happening at such a strange. exponential rate for better and For. Worse right and this is definitely on. the the better side of the equation. there's a bunch of stuff like um one of.

the papers that actually Google deep. mine came out with earlier in the year. was in a single Advance like a single. Paper a single AI model they built uh. they expanded the set of stable. materials coffee's terrible I just tell. you right now Jamie it sucks the water. never got hot I was think about that. enough time to boil yeah that's what it. is it just never never really brewed. it's terrible terrible coffee is my. favorite yeah I can solve that problem. too probably wait till you try this with. terrible for coffee though you like this.

is [ __ ] terrible get some cold ones. oh he looks he looks like it's terrible. it's terrible yeah I could just see that. calculation of like if you're dating a. really hot girl and she cooks for you. like thank you this is amazing this is. the best macaroni and cheese. ever if if if In fairness if uh Scarlett. Johansson's voice was actually giving. you that kind the best copy I've ever. had keep talking may I have some more. please Governor um yeah so so there. there's this one there one paper that. came out and they're like hey by the way.

we've increased the set of stable. materials known to humanity by a factor. of 10 oh my God so like if on Monday we. knew about you know 100,000 stable. materials we now know about a million. they were then validated replicated by. Berkeley University or a bunch of them. as a proof of concept and this is from. like we know the stable materials we. knew before like that Wednesday were. from ancient times like the ancient. Greeks like discovered some [ __ ] the. Romans discovered some [ __ ] the Middle. Ages discovered and then it's like oh. yeah yeah all that that was really cute.

like boom one instantly One Step yep so. and and that's amazing we should be. celebr have great phones in 10 years. dude we'll be able to get addicted to. like feeds that we haven't even thought. of so I mean you're making me feel a. little more positive like overall. there's going to be so many beneficial. aspects to AI oh yeah and it it's just. what it is is just an unbelievably. transformative event that we're living. through yeah it's power and power can be.

good and it can be bad and that that's. true yeah an immense power it can be. immensely good or immensely bad and. we're just in this who knows it's we we. just need to structurally set ourselves. up so that we can reap the benefits and. mind the downside risk like that's. that's what it's always about but the. regulatory story has to unfold that way. well I'm really glad that you guys have. the ethics to get out ahead of this and. to talk about it with so many people and. to really blare this message out because. I don't think there's a lot of people.

that like I had mark andreon on who's. brilliant but he's like all in it's. going to be great and maybe he's right. maybe he's right yeah but I mean you. have to hear all the different. perspectives and I mean like massive. massive props honestly go out to the. team at the state department that we. worked with one of the things also is. over the course of the investigation the. way it was structured was um it wasn't. like a contract and they farmed it out. and we went up it was the two teams. actually like work together the the two.

teams together the state department and. us we went to London UK we talked and. sat down with deep mine we went to San. Francisco we sat down with Sam Alman and. his policy team we sat down with. anthropic all of us together um one of. the major reasons why we were able to. publish so much of the Whistleblower. stuff is that those very individuals. were in the rooms with us when we found. out this [ __ ] and they were like oh [ __ ]. like the world needs to know about this.

and so they were pushing internally for. a lot of this stuff to come out that. otherwise would not and I also got to. say like I just want to memorialize this. too that investigation when we went. around the world we were working with. some of the most elite people in the. government that I didn't I would not. have guessed existed. that was honestly speak more well I can. be it's hard to be specific did you see. any UFOs tell me the truth they take it. to the hanger no no no there's no hang.

there's no there's no hanger yeah cut. cut there's no hanger no hanger don't. worry sweetie don't worry. about. um we didn't we didn't go that far down. the rabbit hole you know we went pretty. far down the rabbit hole and yeah there. are individuals who are just absolutely. absolutely Elite like the level of. capability the amount that are teams. gelled together at certain points the.

stakes like the the stuff we did the. stuff they made happen for us in terms. of brain other they brought together. like a hundred folks from across the. government to discuss like AI on the. path to AGI and go through the. recommendations that we had this was. pretty pretty cool actually it was like. the the uh first basically the first. ever first time the US government came. together and seriously looked at the. prospect of AGI and the risk there and. we had it it was wild I mean again it's.

like that was in November it's us two. freaking yahoos like what the hell do we. know and our amazing team um and uh and. and it was yeah referred to by a there. was a senior White House rep there it. was like yeah this is a watershed moment. in US history and wow well that's. encouraging because again people do like. to look at the government as the DMV. yeah or the worst aspects of bureaucracy. there's missing room like for things. like you know Congressional hearings on. these whistleblower events certainly. Congressional hearings we talked about. on the the idea of liability and. Licensing and what regulatory agencies.

we need just to kind of like start to. get to the meat on the bone on this. issue but uh yeah opening this up I. think is is just it's just really. important well shout out to the part of. the government that's good shout out to. the government that gets it that's. competent and awesome and uh shout out. to you guys because um this is uh it's. heady stuff it's very difficult to grasp. but it's it's even in having this. conversation with you I still don't know. how to feel about it you know I think. I'm at least slightly optimistic that. the potential benefits are going to be.

huge but what a weird passage we're. about to enter into it's the unknown. yeah truly thank you gentlemen really. appreciate your time appreciate what. you're doing thank it's uh people want. to know more where should they go what. should they follow uh I guess Gladstone. doaction plan is one that has our our. action planation Gladstone a all all our. stuff is there I I should mention too I. I have this little podcast called last. week in AI uh we cover sort of the last. week's events and it's all about the.

sort of lens you have to do that every. hour yeah last hour in AI it's like a. week is not enough time we could be at. War our like list of stories keeps. getting Landing yeah anything can happen. time travel you you'll hear it there. first yeah all right well thank you guys. thank you very much appreciate it. everybody. [Music].