Michael Littman: Reinforcement Learning and the Future of AI | Lex Fridman Podcast #144

the following is a conversation with. michael littman a computer science. professor at brown university. doing research on and teaching machine. learning. reinforcement learning and artificial. intelligence. he enjoys being silly and lighthearted. in conversation. so this was definitely a fun one quick. mention of each sponsor. followed by some thoughts related to the. episode thank you to. simply safe a home security company i. use to monitor. and protect my apartment expressvpn the.

vpn i've used for many years to protect. my privacy and the internet. masterclass online courses that i enjoy. from some of the most amazing humans in. history. and better help online therapy with a. licensed professional. please check out the sponsors in the. description to get a discount and. to support this podcast as a side note. let me say that i may experiment with. doing some solo episodes in the coming. months. or two the three ideas i have floating. in my head.

currently is to use one a particular. moment in history. two a particular movie or three a book. to uh drive a conversation about a set. of uh related concepts. for example i could use 2001 a space. odyssey or x machina. to talk about agi for one two three. hours. or i could do an episode on the yes. rise and fall of hitler and stalin. each in a separate episode using.

relevant books and historical moments. for reference i find the format of a. solo episode. very uncomfortable and challenging but. that just tells me. that it's something i definitely need to. do and learn from the experience. of course i hope you come along for the. ride also. since we have all this momentum built up. on announcements. i'm giving a few lectures on machine. learning at mit this january. in general if you have ideas for the. episodes. for the lectures or for just short.

videos on youtube. let me know in the comments that i. still definitely read despite my better. judgment. and the wise sage device of the great. joe rogan if you enjoy this thing. subscribe on youtube. review it with five stars on apple. podcast follow on spotify. support on patreon or connect with me on. twitter. lex friedman and now here's my. conversation. with michael littman i saw a video of.

you talking to. charles this bell about westworld the tv. series. you guys were doing a kind of thing. where you're watching new things. together but let's. rewind back is there a sci-fi. movie or book or shows. that you that was profound that had an. impact on you philosophically or just. like. specifically something you enjoyed. nerding out about. yeah interesting i think a lot of us. have been inspired by robots in movies.

the one that i really like is uh. there's a movie called robot and frank. which i think is really interesting. because it's very near-term. future where uh robots are being. deployed as. uh helpers in people's homes and it was. it was. and we don't know how to make robots. like that at this point but it seemed. very plausible it seemed very. realistic or imaginable and i thought. that was really cool because. they did they're awkward they do funny. things it raised some interesting issues. but. it seemed like something that would.

ultimately be helpful and good if we. could do it right. yeah he was an older cranky gentleman. right he was an older cranky. uh jewel thief yeah it's kind of funny. little. thing which is you know he's a dual. thief and so he. pulls the robot into his life which is. like which is something you could. imagine. taking a home robotics thing. and pulling into whatever quirky thing. that's involved in your this is. meaningful to you exactly so yeah and i. think i think from that perspective i. mean not all of us are jewel thieves and.

so when we bring our robots into. it for yourself uh explains a lot about. this apartment actually. but no the idea that that people should. have the ability to. you know make this technology their own. that that it becomes part of their lives. and and i think that's. it's hard for us as technologists to. make that kind of technology it's easier. to mold people into what we need them to. be. and um just that opposite vision i think. is really inspiring. and then there's a anthropomorphization. where we project.

certain things on them because i think. the robot was kind of dumb. but i have a bunch of roombas that play. with and they you immediately project. stuff onto them much greater level of. intelligence we'll probably do that with. each other too. much much greater degree of compass. that's right one of the things we're. learning from ai is. where we are smart and where we are not. smart yeah. you also enjoy as people can see. and i enjoyed myself uh watching you. sing. and even dance a little bit a little bit.

a little bit a little bit of dancing. a little bit of dancing that's not quite. my thing as a as a method of education. or just in life you know in general. so easy question what's the. definitive objectively speaking top. three songs of all time. maybe something that you know uh. to walk that back a little bit maybe. something that. others might be surprised by the three. three songs that you kind of enjoy. that is a great question that i cannot.

answer but instead let me tell you a. story so. pick a question you do want it that's. right i've been watching the. presidential debates and vice president. debates and turns out yeah it's really. you can just answer any question you. want. so so it's a related question. [Laughter]. yeah well said i really like pop music. i've enjoyed pop music ever since i was. very young so 60s music 70s music. 80s music this is all awesome and then i. had kids and i think i stopped listening. to music and i was starting to realize. that the. like my musical taste had sort of frozen.

out and so i decided. in 2011 i think to start listening to. the top 10. billboard songs each week so i'd be on. the on the treadmill and i would listen. to that week's top 10 songs. so i could find out what was popular now. and what i discovered. is that i have no musical taste. whatsoever i like what i'm familiar. with and so yeah the first time i'd hear. a song it's the first week that was on. the charts i'd be like. and then the second week i was into it a. little bit and the third week. i was loving it and by the fourth week.

is like just part of me and so. i'm afraid that i can't tell you the. most my favorite song of all time. because it's whatever i heard most. recently yeah. that's interesting people have told me. that um there's an art to listening to. music as well. you can start to if you listen to a song. just carefully like. explicitly just force yourself to really. listen you start to. uh i did this when i was part of jazz. band and fusion band in college is. there's they you you start to hear the.

layers. of the instruments you start to hear the. individual instruments and you start to. uh. you can listen to classical music or to. orchestra this way you can listen to. jazz this way i mean. uh it's funny to imagine you now to. walk in that forward to listening to pop. hits now as like a scholar. listening to like cardi b or something. like that or justin timberlake is he. no not temple like bieber i guess. they've both. been in the top 10 since i've been. listening they're still still up there. oh my god i'm so clueless.

if you haven't heard justin timberlake's. top 10 in the last few years. there was one song that he did where the. music video was set. at essentially nurips oh wow oh. the one with the robotics yeah yeah yeah. yeah yeah yeah he's like at an academic. conference and he and he's doing it he. was. presenting it was sort of a cross. between the apple. like steve jobs kind of talk and nurips. um so i you know it's always fun when ai. shows up in pop culture i wonder if he. consulted somebody for that.

that's very that's really interesting so. maybe on that topic i've seen your. um your celebrity multiple dimensions. but one of them is you've done. cameos in different places i've seen you. in a turbo tax commercial as like. i guess the the brilliant einstein. character. and the the point is that turbo tax. doesn't need. somebody like you doesn't need a. brilliant. very few things need someone like me but. yes they were specifically. emphasizing the idea that you don't need.

to be a like a computer expert to be. able to use their software. how did you end up in that world i think. it's an interesting story so i was. teaching my class it was an intro. computer science class for. non-concentrators non-majors. and sometimes when people would visit. campus they would check in to say hey we. want to see what a class is like can we. sit on your class. so a person came to my class. who was the daughter of the brother. of the hus.

husband of the best friend of my wife. anyway basically a family friend came to. campus to. to check out brown and asked to come to. my class and. and came with her dad her dad is uh. who i've known from various kinds of. family events and so forth but he also. does. advertising and he said that he was. recruiting. scientists for this this this ad this. this turbotax. set of ads and he said we wrote the ad.

with the idea that we get like. the most brilliant researchers um but. they all said no. so can you help us find the like b. level scientists i'm like sure. that's that's who i hang out with so. that should be fine. so i put together a list and i did what. some people call the dick cheney so i. included myself on the list. of possible candidates uh you know with. a little blurb about each one and why i. thought. it would make sense for them to to do it. and they reached out to a handful of.

them but then they ultimately they. youtube stalked me a little bit and they. thought. oh i think he could do this and um they. said okay we're gonna offer you the. commercial. i'm like what so um it was it was such. an interesting experience because it's. it's they have another world the people. who do. like nationwide kind of ad campaigns and. and television shows and movies and so. forth it's quite. a a remarkable system that they have. going because like a set. yeah so i went to uh it was just.

somebody's house that they rented in new. jersey. um but it in the in the commercial it's. just me and this other woman. in reality there were 50 people in that. room and another. i don't know half a dozen kind of spread. out around the house in various ways. there were people whose job it was to. control the sun. they were in the backyard on ladders. putting. filters up to try to make sure that the. sun didn't glare off the window in a way. that would wreck the shot. so there was like six people out there. doing that there was three people out.

there giving. snacks the craft table there was another. three people giving. healthy snacks because that was a. separate craft table there was one. person whose job it was. to keep me from getting lost and. the i think the reason for all this is. because so many people are in one place. at one time they have to be time. efficient they have to get it done. this the morning they were going to do. my commercial in the afternoon they were. going to do a commercial of a. mathematics professor from princeton. they had to get it done no you know no. wasted time or energy and so there's.

just a fleet of people. all working as an organism and it was. fascinating i was just the whole time. just looking around like. this is so neat like one person whose. job it was to take the camera off of the. camera man. so that someone else whose job it was to. remove the film canister because every. couple's takes they had to replace the. film because you know. film gets used up it was just i don't. know i was. geeking out the whole time it was so fun. how many takes did it take it looked the. opposite like there was. more than two people there it was very. relaxing right yeah the super.

i mean the person who i was in the scene. with um is a professional. she's a you know uh she's an actor. improv comedian okay in your community. and when i got there they had given me a. script as such as it was and then i got. there and they said. we're gonna do this as improv i'm like i. don't know how to improv like this is. not. i don't know what this i don't know what. you're telling me to do here. don't worry she knows okay okay we'll. see how this goes. i get i guess i got pulled into the. story because like where the heck did. you come from.

i guess in the scene like how did you. show up in this random person's house. i don't know yeah well i mean the. reality of it is i stood outside in the. blazing sun there was someone whose job. it was to keep an umbrella over me. because i started to schvitz i started. to sweat. and so i would wreck the shot because my. face was all shiny with sweat so there. was one person who would dab me off. had an umbrella um but yeah like the. reality of it like. why is this strange stalkery person. hanging around outside somebody's house. yeah we're not we're not sure when you. have to look in we'll have to wait for. the book. but are you uh so you make you make like.

you said youtube you make videos. yourself. you make awesome parody sort of uh. parody songs that kind of focus in on. particular aspects of computer science. how much those seem really natural. how much production value goes into that. do you also have a team of. 50 people videos almost all the videos. except for the ones that people would. have actually seen. were just me i write the lyrics i sing. the song i i generally. find a um like a backing track online.

because i'm unlike you can't really play. an instrument and then i do. in some cases i'll do visuals using just. like powerpoint. lots and lots of powerpoint to make it. sort of like an animation. the the most produced one is the one. that people might have seen which is the. overfitting video that i did with. charles isbell um. and that was produced by the georgia. tech and udacity people because we were. doing a class together it was kind of i. usually do parody songs. kind of to cap off a class at the end of.

a class so that one. you're wearing so it's a this the. thriller yeah you're wearing the michael. jackson the red. leather jacket the interesting thing. with podcasting that you're also. uh into is that. i really enjoy is that there's not a. team of people. it's kind of more because you know the. the there's something that happens. when there's more people involved than. just one person.

that just the way you start acting i. don't know. there's a censorship you're not given. especially for like slow thinkers like. me you're not. and i think most of us are if we're. trying to actually think. we're a little bit slow and and careful. it it kind of large teams get in the way. of that. and i don't know what to do with ice. like that's the to me. like if you know this it's very popular. to criticize quote unquote mainstream. media.

i but there is legitimacy to criticizing. them the same i. love listening to npr for example but. every. it's clear that there's a team behind it. there's a commercial there's constant. commercial breaks there's this kind of. like rush of like. uh okay i have to interrupt you now. because we have to go to commercial just. this whole. it creates it destroys the possibility. of nuanced conversation. yeah exactly evian uh which. charles uh isabel who i i talked to.

yesterday told me that. evian is naive backwards which the fact. that his mind thinks this way is just. uh it's quite brilliant anyway there's a. freedom to this podcast he's dr awkward. which by the way is a palindrome that's. a palindrome that i happen to know. for from other parts of my life and i. just you just throw it out. well you know use it against charles dr. awkward. so what uh what was the most challenging. parody song to make. was it the thriller one hmm no that was.

really fun i. wrote the lyrics really quickly um and. then i gave it over to the product. production team they recruited a. a cappella group to to sing that went it. went really smoothly it's great having a. team because then you can just focus on. the part that you really love which in. my case is writing the lyrics. uh for me the most challenging one not. challenging in a bad way but challenging. in a really fun way. was i did one of this one of the parody. songs i did. is is about the halting problem in. computer science the the fact that. you can't create a program that can tell.

for any other arbitrary program whether. it actually going to get stuck in. infinite loop or whether it's going to. eventually stop. and so i i did it to an 80s song. because that's i hadn't started my new. thing of learning current songs. and it was billy joel's the piano man. nice which is a great song great song. yeah yeah. and sing me a song you get the piano man. yeah yeah so the lyrics are great.

because first of all it rhymes uh not. all songs rhyme i did i've done. rolling stone songs which turn out to. have no rhyme scheme whatsoever they're. just. sort of yelling and having a good time. which makes it not fun from a parody. perspective because like you can say. anything. but this you know the lines rhymed and. there was a lot of internal rhymes as. well. and so figuring out how to sing with. internal rhymes. a proof of the halting problem was. really challenging and. it was i really enjoyed that process. what about uh. last question on this topic what about.

the dancing in the thriller video how. many takes that take. so i wasn't planning to dance they they. had me in the studio and they gave me. the jacket and it's like well you can't. if you have the jacket and the glove. like there's not much you can do yeah. so i um i think i just danced around. and then they said why don't you dance a. little bit we there was a scene with me. and charles dancing together. they did not use it in the video but we. recorded it um yeah yeah no it was. it was pretty funny and charles who has. this.

beautiful wonderful voice doesn't really. sing he's not really a singer and so. that was why i designed the song with. him doing a spoken section and me doing. things very like barry white yeah it's a. smooth baritone. yeah yeah it's great that was awesome so. one of the other things charles said is. that you know. everyone knows you as like a super nice. guy super passionate about. teaching and so on uh what he said. i don't know if it's true that despite. the fact that you're.

you are cold like okay. i will admit this finally for the first. time that was that was me. it's the johnny cash song the man in. reno just to watch him die. uh that you actually do have uh some. strong opinions on some topics. so if this in fact is true what. uh strong opinions would you say you. have is there ideas. you think maybe an artificial. intelligence machine learning. maybe in life that you believe is true.

that others might. you know some number of people might. disagree with you on. so i try very hard to see things from. multiple perspectives. there's there's this great calvin and. harp's calvin and hobb's cartoon where. cal do you know okay so calvin's dad is. always kind of a bit of a foil and he. he was he talked to calvin and just. calvin had done something wrong. the dad talks him into like seeing it. from another perspective and calvin like. this breaks calvin because he's like oh. my gosh now i can see the opposite sides.

of things and so the. it's it becomes like a cubist cartoon. where there is no front and back. everything's just exposed. and it really freaks him out and finally. he settles back down it's like oh good. no i can make that go away. but like i'm that i'm that i live in. that world where i'm trying to see. everything from every perspective all. the time so there are some things that. i've formed opinions about that i. would be harder i think to disavow me of. one is um the super intelligence. argument and the existential. threat of ai is one where i feel pretty.

confident. in my feeling about that one like i'm. willing to hear other arguments but like. i am not particularly moved by the idea. that. if we're not careful we will. accidentally create a super intelligence. that will destroy. human life let's talk about that let's. get you in trouble and record your video. it's like bill gates uh i think he said. like. some quote about the internet that. that's just gonna be a small thing it's. not gonna really go anywhere. and i think uh steve ballmer said uh.

i don't know why i'm sticking on. microsoft uh that's something. that like smartphones are useless. there's no reason why microsoft should. get into smartphones that kind of. so let's get let's talk about agi as agi. is destroying the world we'll look back. at this video and see. no uh i think it's really interesting to. actually talk about because nobody. really. knows the future so you have to use your. best intuition it's very. difficult to predict it but you have. spoken about agi. and the existential risks around it and.

sort of. based on your intuition that we're. quite far away from that being a serious. concern relative to the other concepts. we have. can you maybe uh unpack that a little. bit yeah sure so. so as as i understand it that. uh for example i read boston's book and. a bunch of other. reading material about this sort of. general way of thinking about the world. and i think. the story goes something like this that. we will at. some point create computers that.

are smart enough that they can help. design. the next version of themselves which. itself will be smarter than the previous. version of themselves and eventually. bootstrapped up to being smarter than. us at which point we are essentially at. the mercy of this sort of. more powerful intellect which in. principle. uh we don't have any control over what. its goals are and so if its goals. are at all out of sync with our goals.

like the ex for example the continued. existence of humanity. we won't be able to stop it it'll be way. more powerful. than us and we will be toast so. there's some i don't know very smart. people who have signed on to that story. and it's a. it's a compelling story i once. now i can really get myself in trouble i. once wrote an op-ed about this. specifically responding to some quotes. from elon musk who has been. you know on this very podcast uh more.

than once. and well the e-e-a-i's summoning the. demon that you get. i think he said but then he came to. providence rhode island which is where i. live. and said uh to the governors of all the. states. uh you know you're worried about. entirely the wrong thing you need to be. worried about ai you need to be very. very worried about ai so uh and peop. journalists kind of reacted to that and. they wanted to get people's people's. take and. i was like okay my my my belief. is that one of the things that makes.

elon musk so successful and so. remarkable as an individual. is that he believes in the power of. ideas he believes that you can. have you can if you know if you have a. really good idea for getting into space. you can get into space if you have a. really good idea for a company or for. how to change the way that people drive. you just have to do it and. and it can happen it's really natural to. apply that same idea to ai you see. these systems that are doing some pretty. remarkable computational. tricks uh demonstrations and then to.

take that idea and just push it. all the way to the limit and think okay. where does this go where is this going. to take us next. and if you're a deep believer in the. power of ideas. then it's really natural to believe that. those ideas could. be taken to the extreme and kill us. so i think you know his strength is also. his undoing because. that doesn't mean it's true like it. doesn't mean that that has to happen. but it's natural for him to think that. so. another way to phrase the way he thinks. and.

i find it very difficult to argue with. that. line of thinking uh so sam harris is. another person. from neuroscience perspective that. things like that is. saying well is there something. fundamental. in the physics of the universe that. prevents this from eventually happening. and this nebosh from things in the same. way they're kind of zooming out. yeah okay we humans now uh are existing. in this. like time scale of minutes and days and.

so our. intuition is in this time scale of. minutes hours and days. but if you look at the span of human. history. is there any reason we you. can't see this in in 100 years and. like is there is there something. fundamental about the laws of physics. that prevent this. and if it doesn't then it eventually. will happen or will. we will destroy ourselves in some other. way it's very difficult. i find to actually argue against that. yeah.

me too and not sound like. not sound like you're just like rolling. your eyes uh i'm like i have. like science fiction we don't have to. think about it but even even. worse than that which is like i don't. know kids but like i gotta pick up my. kids now like this okay i see there's. more pressing shortcuts yeah there's. more pressing short-term things that. like. uh stop over this existential crisis. where much much shorter things like. now especially this year there's cova so. like any kind of discussion like that is.

like there's there's p you know there's. pressing things. uh today it's it's and then so the sam. harris argument well like. any day the exponential singularity. can can occur it's very difficult to. argue against i mean i don't know but. part of his story is also. he's he's not going to put a date on it. it could be in a thousand years it could. be in 100 years it could be in two years. it's just that as long as we keep making. this kind of progress. it's ultimately has to become a concern. i i kind of am on board with that but.

the thing that the the piece that i feel. like is missing from that. that way of extrapolating from the. moment that we're in. is that i believe that in the process of. actually developing technology that can. really get around in the world and. really process and and. and do things in the world in a. sophisticated way we're going to learn a. lot about. what that means which that we don't know. now because we don't know how to do this. right now. if you believe that you can just turn on. a deep learning network and eventually. give it enough compute and it'll. eventually get there well sure that.

seems really scary because we won't we. won't be in the loop at all we want we. won't be helping to design or or target. these kinds of systems but i don't i. don't see. that that feels like it is against the. laws of physics because these systems. need help right they need. they need to surpass the the. the difficulty the wall of complexity. that happens in arranging something in. the form that. that will happen in yeah like i believe. in evolution like i believe that the. that that there's an argument right so.

there's another argument just to look at. it from a different. perspective that people say well i don't. believe in evolution how could evolution. it's it's sort of like a random set of. parts. assemble themselves into a 747 and that. could just never happen. yeah so it's like okay that's maybe hard. to argue against but clearly. 747s do get assembled they get assembled. by us basically the idea being that. there's a process by which we will get. to the par the point of making. technology that has that kind of. awareness and.

in that process we're going to learn a. lot about that process and we'll have. more. ability to control it or to shape it or. to build it in our own image. it's not something that is going to. spring into existence like that 747. and we're just gonna have to contend. with it completely unprepared. it's very possible that in the context. of the long arc of human history it will. in fact spring into existence. but that springing might take like if. you look at nuclear weapons.

like even 20 years is a springing. in in the context of human history and. it's very possible just like with. nuclear weapons that we could have. i don't know what percentage you want to. put at it but the the possibility. could have knocked ourselves out yeah. the possibility of human beings. destroying themselves in the 20th. century. with nuclear weapons i don't know you. can if you really think through it. you could really put it close to like i. don't know 30 40 percent. given like the certain moments of crisis. that happen.

so like i think one. like fear in the shadows that's not. being acknowledged. is it's not so much the ai will run away. is. is that as it's running away we won't. have enough time to uh think through how. to stop it. right fast takeoff or foom yeah i mean. my. much bigger concern i wonder what you. think about it which is. we won't know it's happening.

so i kind of that argument i think that. there is an. agi situation already happening with. social media. that our minds our collective. intelligence of human civilization is. already being controlled by an algorithm. and like we're we're already super. like the the level of a collective. intelligence thanks to wikipedia people. should donate to wikipedia. to feed the agi man if we had a super. intelligence that. that was in line with wikipedia's values.

that it's a lot better than a lot of. other things i can imagine i've i trust. wikipedia more than i trust facebook or. youtube. as far as trying to do the right thing. from a rational perspective. yeah now that's not where you were going. i understand that but it it it does. strike me that there's sort of. smarter and less smart ways of of. exposing ourselves to each other on the. internet yeah the interesting thing is. that wikipedia. and social media have very different. forces you're right i mean wikipedia if. if agi was wikipedia it'd be just like. this.

cranky overly competent editor. of uh articles uh you know there's. there's something to that but the social. media aspect is is is not. so the vision of agis is as a separate. system. that's super intelligent that's super. intelligent that's one key little thing. i mean there's the paper clip argument. that's super dumb. but super powerful systems but with. social media you have. a relatively like algorithms we may talk. about today. very simple algorithms that when.

uh something charles talks a lot about. which is interactive ai when they start. like. having at scale like tiny little. interactions with human beings. they can start controlling these human. beings so a single algorithm. can control the minds of human beings. slowly to what we might not. realize it could start wars it could. start it can change the way we. think about things it feels like in the. long arc of history. if i were to sort of zoom out from all.

the outrage and all the tension on. social media. that it's progressing us towards uh. better and better things. it feels like chaos and toxic and all. that kind of stuff but it's chaos and. toxic. yeah but it feels like actually the. chaos and toxic is similar to the kind. of debates we had. from the founding of this country you. know there was a civil war that happened. over that over that period and. ultimately it was all about. this tension of like something doesn't.

feel right about. our implementation of the core values we. hold as human beings and they're. constantly struggling with this. and that results in people calling each. other. uh like just just being shitty to each. other on twitter. but i ultimately the algorithm is. managing all that and it feels like. there's a possible future in which that. algorithm. controls us to into the direction of. self-destruction. whatever that looks like yeah so so all.

right i do believe in the power of. social media to. screw us up royally i do believe in the. power of social media to benefit us too. i do think that we're in a. yeah it's sort of almost got dropped on. top of us and now we're trying to as a. culture figure out how to cope with it. there's a sense in which i don't know. there's there's some arguments that say. that for example. i guess college-age students now late. college-age students now people who are. in middle school when when social media. started to really take off.

maybe maybe really damaged like me this. may have really hurt their development. in a way that we don't. have all the implications of quite yet. that's the generation who. if and i hate to make it somebody else's. responsibility but like they're the ones. who can fix it they're the ones who can. who can figure out. how do we keep the good of this kind of. technology without. letting it eat us alive and. if they're successful we move on to the.

next phase the next level of the game. if they're not successful then yeah then. we're going to wreck each other we're. going to. destroy society so you're going to in. your old age sit on the porch and watch. the world burn. because the tick tock generation that uh. i believe well so my this is my kids age. right and that's certainly my daughter's. age and she's very tapped in. to social stuff but she's also she's. trying to find that balance right of. participating in it and then getting the. positives of it but without letting it. eat her alive um and i think sometimes.

she ventures. hopes just to watch this sometimes i. think she ventures a little too far and. is. in and is consumed by it and other times. she gets a little distance. um and if you know if there's enough. people like her out there they're gonna. they're gonna navigate this this choppy. waters that's that's an. interesting uh skill actually to develop. i talked to my dad about it. you know i've uh now somehow. this podcast in particular but other. reasons.

has received a little bit of attention. and with that apparently in this world. even though. i don't shut up about love and i'm just. all about kindness. i i have now a little mini army of. trolls. oh it's kind of hilarious actually but. it also doesn't feel good. but it's a skill to learn. to not look at that like to moderate. actually how much you look at that. the discussion i have with my dad is. similar to uh it doesn't have to be. about trolls it could be about checking.

email. which is like if you're anticipating you. know there's uh my dad. runs a large institute at drexel. university and. there could be stressful like emails. you're waiting like there's drama of. some kind. and so like there's a temptation to. check the email if you send an email you. cut it. and that pulls you in into it doesn't. feel good. and it's a skill that he actually. complains that he hasn't learned i mean. he. grew up without it so he hasn't learned. the skill of how to.

shut off the internet and walk away and. i think young people. while they're also being quote-unquote. damaged by like. uh you know being bullied online all. those stories which are very. like horrific you basically can't escape. your bullies. these days when you're growing up but at. the same time they're also learning that. skill of how to. be able to shut off uh the like. disconnect with it be able to laugh at. it not take it too seriously. it's fascinating like we're all trying. to figure this out just like you said.

it's. been dropped on us and we're trying to. figure it out yeah i think that's really. interesting and i. i guess i've become a believer in the. human design. which i feel like i don't completely. understand like how do you make. something as robust as us like we're. so flawed in so many ways and yet and. yet. you know we dominate the planet and we. do seem to manage to get ourselves out. of scrapes. eventually not necessarily the most. elegant possible way but somehow we get.

we get to the next step and i don't know. how i'd make a. machine do that i i i. generally speaking like if i train one. of my reinforcement learning agents to. play a video game and it works really. hard on that first stage over and over. and over again and it makes it through. it succeeds on that first level. and then the new level comes and it's. just like okay i'm back to the drawing. board and somehow humanity we keep. leveling up. and then somehow managing to put. together the skills necessary to. achieve success some semblance of.

success in that next level too. and you know i hope we can keep doing. that. you mentioned reinforcement learning so. you've have uh. a couple years in the field no quite. you know quite a few quite a long career. in artificial intelligence broadly but. reinforcement learning specifically. can you maybe give a hint about your. sense. of the history of the field and in some. ways has changed with the.

advent of deep learning but has a long. roots like how is it. weaved in and out of your own life how. have you seen the community change or. maybe the ideas that it's playing with. change. i've had the privilege the pleasure of. being. of having almost a front row seat to a. lot of this stuff and it's been really. really fun and interesting so uh when i. was in college in the 80s. early 80s uh the neural net. thing was starting to happen and i was.

taking a lot of psychology classes a lot. of computer science classes. as a college student and i thought you. know something that can play tic-tac-toe. and just like learn to get better at it. that ought to be a really easy thing so. i spent almost. almost all of my what would have been. vacations during college. like hacking on my home computer trying. to teach it how to play tic-tac-toe and. programming language. basic oh yeah that's that's i was i. that's my first language that's my. native language. is that when you first fell in love with. computer science just like programming. basic on that.

uh what was the computer do you remember. i had i had a trs-80. model one before they were called model. ones because there was nothing else uh. i got my computer in 1979. uh instead so i was i was i would have. been bar mitzvahed. but instead of having a big party that. my parents threw on my behalf. they just got me a computer because. that's what i really really really. wanted i saw him in the in the. in the mall in radio shack and i thought. what how are they doing that i would try.

to stump them i would give them math. problems like. one plus and then in parentheses two. plus one yeah and i would always get it. right i'm like. how do you know so much message like. i've had to go to algebra class for the. last few years to learn this stuff and. you just seem to know. so i was i was i was smitten and i got a. computer and i think ages. 13 to 15. i have no memory of those years i think. i just was in my room with the computer. listening to billy joel communing. possibly listening to the radio.

listening to billy joel. that was the one album i had uh on vinyl. at that time. and um and then i got it on cassette. tape and that was really helpful. because then i could play it i didn't. have to go down to my parents wi-fi or. hi-fi. sorry uh and at age 15 i remember kind. of walking out and like okay. i'm ready to talk to people again like. i've learned what i need to learn here. and um so yeah so so that was that was. my home computer and so i went to. college and i was like oh i'm totally. going to study computer science. i opted the college i chose specifically.

had a computer science major. the one that i really wanted the college. i really wanted to go to didn't so. bye-bye to them which college did you go. through so i went to yale. uh princeton would have been way more. convenient and it was just beautiful. campus and it was close enough to home. and i was really excited about princeton. and i visited. i said so computer science major like. well we have computer engineering i'm. like oh i don't like that word. engineering. i like if you're science i really i want. to do like you're saying hardware and. software they're like yeah like i just. want to do software i.

i couldn't care less about hardware you. grew up in philadelphia i grew up. outside philly yeah yeah okay. uh so the you know local schools were. like penn and drexel. and uh temple like everyone in my family. went to temple at least at one point in. their lives except for me. so yeah philly philly family yale had a. computer science department and that's. when you. it's kind of interesting you said 80s. and you're all that works that's when. you know that which is a hot new thing. or a hot. thing period uh so what is that in. college when you first learned about. neural networks yeah.

yeah was she learned like it was in a. psychology class not in a cs wow. yeah was it psychology or cognitive. science or like do you remember like. what context it was yeah yeah yeah so so. i was a. i've always been a bit of a cognitive. psychology groupie. so like i studied computer science but i. like i like to hang around where the. cognitive. scientists are because i don't know. brains man they're like. they're wacky cool and they have a. bigger picture view of things they're a. little less. engineery i would say they're more.

they're more interested in the. nature of cognition and intelligence and. perception it's called like the vision. system work. they're asking always bigger questions. now with. the deep learning community there i. think more there's a lot of. intersections but i do find in. that the neuroscience folks actually. and uh cognitive psychology cognitive. science folks. are starting to learn how to program how. to use your own artificial neural. networks. and they are actually approaching. problems in like totally new interesting.

ways. it's fun to watch that grad students. from those departments. like approach the problem of machine. learning right they come in with a. different perspective yeah they don't. care about like your. imagine that data set or whatever they. they want like to understand the. the like the basic mechanisms. at the at the neuronal level and the. functional level of intelligence it's. kind of. it's kind of cool to see them work but. yeah okay so. you always you're always a group you. have cognitive psychology.

yeah yeah and so uh so it was in a class. by richard garrick he was kind of my. my favorite uh psych professor in. college and i took uh like three. different classes with him. and yeah so that we they were talking. specifically the class i think was kind. of a. there was a big paper that was written. by stephen pinker and. uh prince i don't i'm blanking on. prince's first name but prince and. pinker and prince. they wrote kind of a they were at that. time. kind of like ah i'm blanking on the.

names of the current people. um the cognitive scientists who are. complaining a lot about deep networks. oh uh gary gary marcus sorry marcus. and who else i mean there's a few but. gary gary's the most feisty. sure gary's very feisty and with this. with his co-author they they you know. they're kind of. doing these kind of takedowns where they. say okay well yeah it does all these. amazing amazing things but. here's a shortcoming here's a. shortcoming here's your shortcoming and. so the pinker prince paper. is kind of like the that generation's.

version of. marcus and davis right where they're. they're trained as cognitive scientists. but they're looking skeptically at the. results in the. in the artificial intelligence neural. net kind of world and saying. yeah it can do this and this and this. but like it can't do that and it can't. do that and it can't do that. maybe in principle or maybe just in. practice at this point but but the fact. of the matter is. you're you've narrowed your focus too. far. to be impressed you know you're. impressed with the things within that.

circle. but you need to broaden that circle a. little bit you need to look at a wider. set of problems. and so um so we have so i was in this. seminar in college. that was basically a close reading of. the pinker prince paper. which was like really thick there was a. lot going on in there. and um and and it talked about. the reinforcement learning idea a little. bit i'm like oh that sounds really cool. because behavior is what is really. interesting to me about. psychology anyway so making programs. that i mean programs are things that. behave.

people are things that behave like i. want to make learning that learns to. behave. in which way was reinforcement learning. presented is this uh talking about. human and animal behavior or are we. talking about actual mathematical. constructs ah that's. right so that's a good question right so. this is i think. it wasn't actually talked about as. behavior in the paper that i was reading. i think that it just talked about. learning and to me learning is about. learning to behave but really. neural nets at that point were about. learning like supervised learning so. learning to produce outputs from inputs.

so i kind of tried to invent. reinforcement learning. i uh when i graduated i joined a. research group at. bellcore which had spun out of bell abs. recently at that time because of the. divestiture of the. of long distance and local phone service. in the 1980s 1984. and i was in a group uh with dave ackley. who. was the first author of the boltzmann. machine paper so the very first neural. net paper that could handle. xor right so xor sort of killed neural.

nets the very first the zero with the. first winter. yeah um the the perceptron's paper. and hinton along with his student dave. ackley and and i think there was other. authors as well. showed that no no with both machines we. can actually learn. non-linear concepts and so everything's. back on the table again and that kind of. started that second wave of neural. networks. so dave ackley was he became my mentor. at bellcore and we. talked a lot about learning and life and.

computation and how all these things fit. together. now dave and i have a podcast together. so um so i get to. kind of enjoy that sort of. his his perspective uh once again even. even all these years later. and so i said so i said i was really. interested in learning but. in the concept of behavior and he's like. oh well that's reinforcement learning. here and he gave me rich sutton's 1984. td paper. so i read that paper i honestly didn't. get all of it. but i got the idea i got that they were.

using that he was using ideas. that i was familiar with in the context. of neural nets and and. like sort of backprop uh but with this. idea of making predictions over time i'm. like this is so interesting but i don't. really get all the details i said. to dave and dave said oh well why don't. we have him come and give a talk. and i was like wait what you can do that. like these are real people. i thought they were just words i thought. it was just like ideas that somehow. magically seeped into paper he's like no. i. i i know rich like we'll just have him.

come down and and he'll give a talk. and so i was you know my mind was blown. and uh so rich came and he gave a talk. at bellcore. and he talked about what he was super. excited which was they had just figured. out at the time. uh q learning so uh watkins had visited. the rich sutton's lab at umass or. it's andy barto's lab that rich was a. part of. and um he was really excited about this. because it resolved a whole bunch of.

problems that he didn't know how to. resolve in the. in the earlier paper and so uh. for people who don't know td temporal. difference these are all just algorithms. for reinforcement learning. right and td separate difference in. particular is about making predictions. over time. and you can try to use it for making. decisions right because if you can. predict how good a future action. and action outcomes will be in the. future you can choose one that has. better and or. but the theory didn't really support. changing your behavior like the. predictions had to be of a consistent.

process if you really. wanted it to work and one of the things. that was really cool about. q-learning algorithm for reinforcement. learning is it was off policy which. meant that you could actually be. learning about the environment and what. the value of different actions would be. while actually figuring out how to. behave. optimally yeah so that was a revelation. yeah and the proof of that is kind of. interesting. i mean that's really surprising to me. when i first read that and then enriched. rich sutton's book on the matter it's. it's kind of. beautiful that a single equation can.

capture an equation one line of code and. like you can learn anything. yeah like enough time so equation and. code you're right like. you can the code that. you can arguably at least if you like. squint your eyes. can say this is all of intelligence. is that you can implement that in a. single wall i think i started with lisp. which is uh. shout out to lisp uh like a single line. of code. key piece of code maybe a couple that.

you could do that it's kind of magical. it's uh feels too good to be true. well and it sort of is yeah it's kind of. kind of. it seems to require an awful lot of. extra stuff supporting it but. yeah but nonetheless the ideas the the. idea is really good and as far as we. know it is it is. a very reasonable way of trying to. create adaptive behavior. behavior that gets better at something. over time. did you find the idea of optimal uh at. all compelling that.

you could prove that it's optimal so. like one part of computer science. that it makes people feel warm and fuzzy. inside. is when you can prove something like. that a sorting algorithm worst case. runs and and log n and it makes. everybody feel so good. even though in reality it doesn't really. matter what the worst case is what. matters is like. does this thing actually work in. practice on this particular actual set. of data that i. that i enjoy did you so here's that. here's a place where i have maybe a. strong opinion uh-oh which is like.

you're right of course but no no like so. so the what makes worst case so great. right if you have a worst case analysis. so great is that you get modularity. you can take that thing and plug it into. another thing and still. have some understanding of what's going. to happen when you click them together. right if it just works well in practice. in other words with respect to some. distribution that you care about. when you go plug it into another thing. that distribution can shift. it can change and your thing may not. work well anymore and you want it to.

and you wish it does and you hope that. it will but it might not and then. ah so you're so so you're saying you. don't like. machine learning. but we have some positive theoretical. results for these things. you know you can come back at me with. yeah but they're really weak and yeah. they're really weak and and you can even. say that. you know sorting algorithms like if you. do the optimal sorting algorithm it's. not really the one that you want. and that might be true as well but but.

it is the modularity is a really. powerful statement. really as an engineer you can then. assemble different things you can count. on them to be. i mean it's interesting it's it's a. balance. like with everything else in life you. don't want to get too obsessed i mean. this is what computer scientists do. which they. potentially get obsessed they over. optimize things. or they start by optimizing them they. over optimize yeah so it's. it's easy to like get really granular.

about this thing. but like the step from an n squared. to an n log n sorting algorithm is a big. leap. for most real-world systems no matter. what the actual. behavior of the system is that's a big. leap and the same can probably be. said for other kind of first. leaps that you would take on a. particular problem like it's the. picking the low hanging fruit or. whatever the equivalent of. doing the not the dumbest thing but the.

next to the dumbest thing. is picking the most delicious reachable. fruit yeah. most delicious reachable fruit i don't. know why that's not a saying and. yeah okay so uh so you. then this is the 80s and this kind of. idea starts to percolate of uh. yeah at that point i got to re i got to. meet rich sutton so everything was sort. of downhill from there and that was that. was really the pinnacle of everything. um but then i you know then i felt like. i was kind of on the inside so then as. interesting results were happening i.

could like check in with with. rich or with jerry tessaro who had a. huge impact on. uh kind of early thinking in in temporal. difference learning and reinforcement. learning and showed that you could do. you could solve problems that we didn't. know how to solve any other way. and so that was really cool so as good. things were happening i would hear about. it from. either the people who were doing it or. the people who were talking to the. people who are doing it. and so i was able to track things pretty. well through through the 90s. so what uh wasn't most of the excitement.

on reinforcement learning in the 90s. era with what is it td gamma like. what's the role of these kind of little. like. fun game playing things and. breakthroughs about uh. get you know exciting the community was. that like. what were your because uh you've also. built across or. we're part of building a crossword a. puzzle. uh solver program yeah solving program. uh called proverb so.

so you were interested in this as as a. problem like in forming. using games to understand how to build. uh intelligent systems so like what did. you think about tt gamble like what did. you think about that whole thing in the. 90s yeah i mean i found the td gammon. result really just remarkable so i had. known about some of jerry's stuff before. he did td gammon he did a system. just more vanilla well not not entirely. vanilla but a more classical backproppy. kind of.

uh network for playing back ammon where. he was training it on. expert moves so it was kind of. supervised but the way that it worked. was not. to mimic the actions but to learn. internally an evaluation function so to. learn. well if the expert chose this over this. that must mean that the expert values. this. more than this and so let me adjust my. weights to make it so that. the network evaluates this as being. better than this. so it could learn from from human. preferences. it could learn its own preferences and.

then when he took the step from that to. actually. doing it as a full-on reinforcement. learning problem where you didn't need. a trainer you could just let it play. that was that was remarkable right and. so i think. as as humans often do as we've done. in the recent past as well people. extrapolate it's like oh well if you can. do that which is obviously very hard. then obviously you could do all these. other problems that we. that we want to solve that we know are.

also really hard. and it turned out very few of them ended. up being practical. partly because i think neural nets. certainly at the time were struggling to. be. consistent and reliable and so training. them in a reinforcement learning setting. was a bit of a mess. i had i don't know generation after. generation of. like master students who wanted to do. value function approximation basically. learn reinforcement learning. with neural nets and.

over and over and over again we were. failing we couldn't get. the good results that jerry tessaro got. i now believe that jerry is a neural net. whisperer. he has a particular ability to get. neural networks to do things. that other people would find impossible. and it's not. the technology it's the technology and. jerry together. yeah and which i think speaks to. the role of the human expert in the. process of machine learning. right it's so easy we're so drawn to the.

idea that that it's the technology that. is that. is where the power is coming from that i. think we lose sight of the fact that. sometimes you need a really good. just like i mean no one would think hey. here's this great piece of software. here's like i don't know gnu emacs or. whatever. um doesn't that prove that computers are. super powerful and. basically going to take over the world. it's like no stallman is a hell of a. hacker right so he was able to make. the code do these amazing things he. couldn't have done it without the. computer but the computer couldn't have. done it without him. and so i think people discount the role.

of people like jerry who. who um who have just a particular. particular set of skills on that topic. by the way. as a small side note i tweeted emacs is. greater than vim yesterday. and deleted deleted the tweet 10 minutes. later when i realized. you're you were honest i started a war. yeah i was like oh. i was just kidding i i was just being. um walk so people still feel.

passionately. about that particular piece of uh i. don't get that because emacs is clearly. so much better. i i don't understand but you know why do. i say that because i cause. like i spent a block of time in the 80s. um making my fingers know the emacs. keys and now like that's part of the. thought process for me like i need to. express and if you take that if you take. my emacs key bindings away. i become little. i can't express myself i'm the same way.

with the i don't know if you know what. what it is but it's a kinesis keyboard. which is uh this. butt shaped keyboard yes i've seen them. yeah and. they're very uh i don't know sexy. elegant yeah they're. just beautiful yeah they're they're. gorgeous uh way too expensive. but uh the the problem with them similar. with emacs. is when once you learn to use it. it's harder to use other things it's. hard to use other things there's this. absurd thing where i have like. small elegant lightweight beautiful.

little laptops. and i'm sitting there in a coffee shop. with a giant kinesis keyboard. and a sexy little laptop it's absurd but. it you know like i used to feel bad. about it but at the same time you just. kind of have to. sometimes it's back to the billy joel. thing you just have to throw that billy. joe record and. throw taylor swift and justin bieber to. the wind. so see but i like them now because i. cause again i have no musical taste like. like now that i've heard justin bieber. enough i'm like i really like his songs. and.

taylor swift not only do i like her. songs but my daughter's convinced that. she's a genius and so now i basically. have i'm signed on to that so. so yeah that that speaks to the back to. the robustness of the human brain that. speaks to the neuroplasticity that you. can just. you can you can just like a mouse teach. yourself to a problem. dog teach yourself to enjoy taylor swift. i'll try it out. i don't know i try you know what it. has to do with just like acclimation. right just like you said a couple weeks. yeah that's an interesting experiment.

i'll actually try that like i'll listen. that wasn't the intent of the experiment. just like social media it wasn't. intended as an experiment. to see what we can take as a society but. it turned out that way. i don't think i'll be the same person on. the other side of the week listening to. taylor swift but. let's try it it's more compartmental. don't be so worried like it's. like i get that you can be worried but. don't be so worried because we. compartmentalize really well and so. it won't bleed into other parts of your. life you won't start i don't know. wearing red lipstick or whatever like. it's it's fine it's changed fashion and. everything.

but you know what the the thing you have. to watch out for is you'll walk into a. coffee shop once we can do that again. and recognize the song and you'll be no. you won't know that you're singing along. until everybody in the coffee shop is. looking at you and then you're like. that wasn't me yeah that's the. you know people are afraid of agi i'm. afraid of the taylor uh. the tail taylor swift takeover yeah and. i mean people should know that td gammon. was i get would you call it do you like. the terminology of self. play by any chance so like systems that.

learn. by playing themselves just i don't know. if it's the best word. but uh so what's what's the problem with. that term. okay so it's like the big bang like. it's it's like talking to serious. physicists do you like the term big bang. and. when when it was early i feel like it's. the early days of self-play i don't know. maybe it was just previously but. i think it's been used by only a small. group of people. uh and so like i think we're still. deciding is this.

ridiculously silly name a good name for. the. cons potentially one of the most. important concepts in artificial. intelligence. okay it depends how broadly you apply. the term so i used the term in my 1996. phd dissertation. wow the actual terms of yeah because. because. tessaro's paper was something like um. training up an expert. backgammon player through self-play so i. think it was in the title of his paper. okay if not in the title it was. definitely a term that he used. there's another term that we got from. that work is rollout so i don't know if.

you do you ever hear the term rollout. that's a backgammon term that has now. applied. generally in computers well at least in. ai. because of td gammon yeah that's. fascinating so how is health play being. used now and like why is it does it does. it feel like a more general powerful. concept. sort of the idea of well the machine's. just going to teach itself to be smart. yeah so that's that's where maybe you. can correct me. but that's where you know the. continuation of the spirit and actually. like. literally the exact algorithms of td.

gammon are applied. by deep mind and open ai to learn games. that are a little bit more complex. that when i was learning artificial. intelligence go was presented to me. with artificial intelligence the modern. approach i don't know if they explicitly. pointed to go. in those books as like unsolvable kind. of thing. like implying that these approaches hit. their limit. in this with these particular kind of. games so something. i don't remember if the book said it or. not but something in my head.

or was the professors instilled in me. the idea like this is the limits of. artificial intelligence. of the field like it instilled in me the. idea that. if we can create a system that can solve. the game of go. we've achieved agi that was kind of i. didn't explicitly like. say this but it that was the feeling and. so from i was one of the people that it. seemed. magical when a learning system was able. to. to beat a uh a human.

world champion at the game of go and. even more so. from that that was alphago even more so. with alphago zero then. kind of renamed and advanced into alpha. zero. beating a world champion or world-class. player. without any supervisors learning on. expert games. we're doing only through by playing. itself. so that is. i don't know what to make of it i think. it would be interesting to hear.

what your opinions are on just how. exciting. surprising profound. interesting or boring the. breakthrough performance of alpha zero. was. okay so alphago knocked my socks off. that was that was so remarkable which. aspect of it. that they they got it to work that they. actually were able to. leverage a whole bunch of different. ideas integrate them into one.

giant system just the software. engineering aspect of it is mind-blowing. i don't. i i've never been a part of a program as. complicated as the program that they. built for that. and um and just the you know like like. jerry tessaro is a neural net whisperer. like you know david silver is a kind of. neural net whisperer too he was able to. coax these networks. and these new way out their. architectures to do. these you know solve these problems that. um as you said. you know when we were learning from uh. ai.

no one had an idea how to make it work. it was it was remarkable. that um these you know these these. techniques that were so good at playing. chess and they could beat the world. champion in chess. couldn't beat you know your typical go. playing teenager and go. so the fact that that you know in a very. short number of years we kind of ramped. up to. uh trouncing people and go just. blew me away so you're kind of focusing. on the engineering. aspect which is also very surprising i.

mean there's something. different about large well-funded. companies i mean there's a compute. aspect to it too. sure like that of course. i mean that's similar to deep blue right. with uh with ibm. like there's something important to be. learned and remembered about. a large company taking the ideas that. are. already out there and investing a few. million dollars into it. or or more and. so you're kind of saying the engineering. is kind of fascinating both on the.

with alphago is probably just gathering. all the data. right of the expert games like. organizing everything. actually doing distributed supervised. learning. and to me. see the engineering i kind of took for. granted. to me philosophically being able to. persist in the. in the face of like long odds because it. feels like. for me i'll be one of the skeptical.

people in the room thinking that you can. learn your way to. to beat go like it sounded like. especially with david silver it sounded. like david was not. confident at all it's like it was like. not it's funny how confidence works. yeah it's like you're not like cocky. about it. like but right because if you're cocky. about it you. kind of stop and stall and don't get. anywhere yeah but there's like a. hope that's unbreakable maybe that's.

better than confidence it's a kind of. wishful. hope and a little dream and you almost. don't want to do anything else you kind. of. keep doing it that's that seems to be. the story. and but with enough skepticism that. you're looking for where the problems. are and fighting. through them yeah because you know. there's got to be a way out of this. thing yeah. and for him it was probably there's. there's a bunch of little factors that. come into play it's funny how these. stories just all come together like. everything he did in his life. came into play which is like a love for.

video games. and also a connection to so the the 90s. had to happen with td gammon and so on. yeah. in some ways it's surprising maybe you. can provide some intuition to it. that not much more than td gammon was. done for quite a long time. on the reinforcement learning front yeah. is that weird to you. i mean like i said the the students who. i worked with we tried to get. basically apply that architecture to. other problems and. we consistently failed there were a.

couple. a couple really nice demonstrations that. ended up being in the literature there. was a. paper about controlling elevators right. where it's it's like okay. can we modify the heuristic that. elevators use for deciding like a bank. of elevators for deciding. which floors we should be stopping on to. maximize throughput essentially and you. can set that up as a reinforcement. learning problem and you can. you know have a neural net represent the. value function so that it's taking. where all the elevators where the button. pushes you know this high dimensional.

well at the time high dimensional input. um you know a couple dozen dimensions. and turn that into a prediction as to oh. is it going to be better if i stop at. this floor or not. and ultimately it appeared as though for. the. standard simulation distribution for. people trying to leave the building at. the end of the day. that the neural net learned a better. strategy than the standard one that's. implemented in. elevator controllers so that that was. nice. there was some work that satender singh. it all did on. uh handoffs with cell phones.

uh you know deciding when when should. you hand off from this cell tower to. this cell. okay communication networks yeah yeah. and so. a couple things seemed like they were. really promising none of them made it. into production that i'm aware of. and neural nets as a whole started to. kind of implode around then. and so there just wasn't a lot of air in. the room for people to try to figure out. okay how do we get this to work. in the rl setting and then they they. found their way back in.

in 10 in 10 plus years so you said. alphago was impressive like it's a big. spectacle is there right so then alpha. zero so i think i may have a slightly. different opinion. on this than some people so um i talked. to tinder saying in particular about. this so satinder was. uh like rich sutton a student of. antibartow so they came out of the same. lab. very influential machine learning. reinforcement learning researcher. uh now deep mind uh as just as is rich. though different sites the two of them.

he's in alberta. rich is in alberta and uh satinder would. be in england but i think he's in. england from michigan at the moment. uh but the but he was yes he was much. more impressed with. uh alphago zero. which is didn't didn't get a kind of a. bootstrap in the beginning with human. trained games yes just was purely. self-play. though the first one alpha go was also a. tremendous amount of self-play. right they started off they kick-started. the the action network that was making.

decisions. but then they trained it for a really. long time using more traditional. temporal difference methods. um so so as a result i didn't it didn't. seem. that different to me like it seems like. yeah. why wouldn't that work like once once it. works it works so. but he he found that that removal of. that extra information to be. breathtaking like that that's a game. changer to me the first thing was more. of a game changer. but the open question i mean i guess. that's the assumption.

is the expert games might contain with. them. within them a. humongous amount of information but we. know that it went beyond that. right we know that it somehow got away. from that information because it was. learning strategies i don't think it. i don't think alphago is just better at. implementing human strategies i think it. actually developed its own strategies. that were. that was more effective and so from that. perspective. okay well so it made at least one. quantum leap in terms of strategic.

knowledge. okay so now maybe it makes three like. okay but that first one is the doozy. right. getting it to to to work reliably and. and for the networks to to hold on to. the value well enough like that was. that was a big step well isn't maybe you. could speak to this on the reinforcement. learning front so the. starting from scratch and learning to do. something like the first like. like random behavior to like.

crappy behavior to like somewhat okay. behavior. it's not obvious to me that that's not. like impossible to take those steps like. if you just think about the intuition. like how the heck does. random behavior become somewhat. basic intelligent behavior not not human. level not super human level. but just basic but you're saying to you. kind of the intuition is like if. if you can go from human to superhuman.

level intelligence on the. uh on this particular task of game. playing then. so you're good at taking leaps so you. can take many of them. that the system i believe that the. system can take that kind of leap. yeah no and also i think that that. beginner knowledge. in go like you can start to get a feel. really quickly. for the idea that um you know certain. parts of the being in certain parts of. the board seems to be. more associated with winning right. because it's not. it's not stumbling upon the concept of.

winning it's told that it wins. or that it loses well it's self-play so. it both wins and loses it's told. which which side won and the information. is kind of there to start. percolating around to make a difference. as to. um well these things have a better. chance of helping you win and these. things have a worse chance of helping. you win and so. you know it can get to basic play i. think pretty quickly. then once it has basic play well now. it's kind of forced to do some search to. actually experiment with okay well what.

gets me that next. increment of of improvement how far do. you think. okay this is where you kind of bring up. the the elon musk and the sam harris is. right. how far is your intuition about these. kinds of self-playing mechanisms being. able to take us. because it feels one of the. ominous but stated. calmly things that when i talked to. david silver he said. is that they have not yet discovered a. ceiling. for alpha zero for example in the game.

of go or chess. it's it keeps no matter how much the. compute they throw at it it keeps. improving. so it's possible it's very possible that. you. if you throw you know some like 10x. compute that it will improve by 5x or. something like that. and when stated calmly it's so like. oh yeah i guess so but like and then you. think like well. can we potentially have like uh.

continuations of moore's law in totally. different way like broadly defined. moore's law right not the constitutional. improvement exponential improvement like. are we going to have an alpha zero that. swallows the world. uh but notice it's not getting better at. other things it's getting better at. go yeah and i think it's a that's a big. leap to say. okay well therefore it's better at other. things. well i mean the the question is how much. of the game of life. can be turned into right so that's of. that i think is a really good question.

and i think that we don't i don't think. we as a. i don't know community really know that. the answer to this but. um so okay so so i went i went to a talk. uh by some experts on. computer chess so in particular computer. chess is really interesting because. for you know for of course for a. thousand years humans were the best. chess playing. things on the planet um and then. computers like edge to head of the best. person and they've been ahead ever since. it's not like people have. have overtaken computers but um.

but computers and people together have. overtaken computers. right so at least last time i checked i. don't know what the very latest is but. last time i checked. that there were teams of people who. could work with computer programs to. defeat the best computer programs. in the game of go in the game of chess. in the game of chess right and so. using the information about how. these things called elo scores this sort. of notion of how strong a player are you.

there's a there's kind of a range of. possible scores and the. you you increment and score basically if. you can. beat another player of that lower score. 62 percent of the time or something like. that like there's some threshold of. if you can somewhat consistently beat. someone then you are. of a higher score than that person and. there's a question as to how many times. can you do that in chess. right and so we know that there's a. range of human ability levels that. cap out with the best playing humans and. the computers went a step beyond that.

and computers and people together have. not gone i think a full step beyond that. it feels the estimates that they have is. that it's starting to asymptote. that we've reached kind of the maximum. the best possible. chess playing and so that means that. there's kind of a. finite strategic depth right at some. point you just can't get any better at. this game. yeah i mean i i don't uh so i like to. check that. uh i think it's interesting because if. you have. somebody like uh magnus carlsen who's.

using these chess programs to train his. mind like to learn. to become a better chess player yeah and. so like that's a very. interesting thing because we're not. static creatures we're learning together. i mean just like we're talking about. social networks those algorithms are. teaching us just like we're teaching. those algorithms. so that's a fascinating thing but i. think. the best just playing programs are now. better than the pairs like they have. competition between.

paris but the it's still even if they. weren't. it's an interesting question where's the. ceiling so the the david the ominous. david silver. kind of statement is like we have not. found the ceiling. right but so the question is okay so i. don't i don't know. his analysis on that my from talking to. go experts the depth the strategic depth. of go seems to be. substantially greater than that of chess. that there's more kind of. steps of improvement that you can make. get getting better and better and better.

but there's no reason to think that it's. infinite yeah. and so it could be that it's that the. what david is seeing is a kind of. asymptoting that you can keep getting. better. but with diminishing returns and at some. point you hit. optimal play like in theory all these. finite games. they're finite they have an optimal. strategy there's a strategy that is the. minimax optimal strategy. and so at that point you can't get any. better you can't beat that that strategy. now that strategy may be. from an information processing.

perspective. intractable right the you need. the the all the situations are. sufficiently different that you can't. compress it at all it's this. giant mess of hard-coded rules. and we can never achieve that but but. that still puts a cap on how many levels. of improvement that we can actually make. but the the thing about self-play is if. you. if you put it although i don't like. doing that in the broader category of. self-supervised learning is that it. doesn't require.

too much or any human human labeling. yeah yeah human label or. just human effort the human involvement. past a certain point. and the same thing you could argue is. true for. the recent breakthroughs in natural. language processing with language models. oh this is how you get to gpt3. yeah see how that did the uh that was a. good good transition yeah yeah. i practiced that for days uh leading up. to this guy now. uh but like that's one of the questions. is. can we find ways to formulate problems.

in this world that are important to us. humans. like more important than the game of. chess. that uh to which self-supervised kinds. of approaches could be applied whether. it's self-play. for example for like maybe you could. think of like autonomous vehicles. in simulation that kind of stuff. or just robotics applications and. simulation. or in the self-supervised learning. where unannotated.

data or data that's generated by humans. naturally without extra cost like the. wikipedia or like all of the internet. can be used. to learn something about to create. intelligent systems that do something uh. really powerful that pass the turing. test or that. do some kind of superhuman level. performance. so what's your intuition like trying to. stitch all of it together.

about our discussion of agi. the limits of self-play and your. thoughts about maybe the limits of. neural networks in the context of. language models is there some intuition. in there that might be. useful to think about yeah yeah yeah so. so. first of all the the whole transformer. network. family of things um is really cool. it's really really cool i mean for you. know if you've ever.

back in the day you played with i don't. know mark off models for generating text. and you've seen the kind of text that. they spit out. and you compare it to what's happening. now it's. it's amazing it's so amazing now it. doesn't take very long. interacting with one of these systems. before you find the holes. right it's it's not smart in any. kind of general way. it's really good at a bunch of things. and it does seem to understand a lot of. the statistics of language extremely. well. and that turns out to be very powerful.

you can answer many questions with that. but it doesn't make it a good. conversationalist. right and doesn't make it a good. storyteller it just makes it good at. imitating of things it has seen. in the past the exact same thing could. be said by. people who voting for donald trump about. joe biden supporters. and people voting for joe biden about. donald trump supporters. is uh you know that they're not. intelligent they're just following. the yeah they're following things. they've seen in the past and uh. so it's very it doesn't take long to.

find the flaws. in their uh in their like. natural language generation abilities. yes yeah so we're being very. that's interesting critical of ass. right so so i've had a similar thought. which was that. the stories that gpt-3 spits out. are amazing and very human-like. and it doesn't mean that computers are. smarter than we realize necessarily it. partly means that people are dumber than. we realize.

or that much of what we do day to day. is not that deep like we're just we're. just kind of going with the flow we're. saying. whatever feels like the natural thing to. say next not a lot of it. is is is creative or meaningful or or. intentional but enough is that we. actually get. we get by right we we do come up with. new ideas sometimes and we do. manage to talk each other into things. sometimes and we do sometimes. vote for reasonable people sometimes. but um but it's really hard to see in.

the statistics because so much of what. we're saying is kind of rote. and so our metrics that we use to. measure how these systems are doing. don't reveal that because it's it's it's. in the interest this is. that that is very hard to detect but is. your. do you have an intuition that with these. language models. if they grow in size it's already. surprising that when you go from gpt2 to. gpg3. that there is a noticeable improvement. so the question now goes back to the.

ominous david silver and the ceiling. right so maybe there's just no ceiling. we just need more compute now. i mean okay so now i'm speculating yes. as opposed to before when i was. completely on firm yeah all right um. i don't believe that you can get. something that really. can do language and use language as a. thing that doesn't interact with people. like i think that it's not enough to. just take everything that we've said. written down and just say. that's enough you can just learn from.

that and you can be intelligent i think. you really need to. be pushed back at i think that. conversations. even people who are pretty smart maybe. the smartest thing that we know. not maybe not the smartest thing we can. imagine but we get. so much benefit out of talking to each. other and interacting. that's presumably why you have. conversations live with guests is that. that there's. something in that interaction that would. not be exposed by. oh i'll just write your story and then. you can read it later and i think. i think because these systems are just.

learning from our stories they're not. learning from. being pushed back at by us that they're. fundamentally limited into what they. could actually become. on this route they have to they have to. get. you know shut down like we like we have. to have an argument that. they have to have an argument with us. and lose a couple times before they. start to realize. oh okay wait there's some nuance here. that actually matters. yeah that's actually subtle sounding but. quite profound that the interaction with. humans is essential.

and the limitation within that is. profound as well. because the time scale like the. bandwidth at which you can really. interact with humans is very low. so it's costly so you can't one of the. underlying things about self self-plays. it has to do you know a very large. number of interactions. and so you can't really deploy. reinforcement learning systems. into the real world to interact like you. couldn't deploy a language model.

into the real world to interact with. humans because it would just not get. enough data. relative to the cost it takes to. interact like the time of humans is is. expensive. which is really interesting that's that. go that takes us back to reinforce and. learning and trying to figure out. if there's ways to make algorithms that. are more efficient at learning. keep the spirit and reinforcement. learning and become more efficient. in some sense this seems to be the goal. i'd love to hear what your.

thoughts are i don't know if you got a. chance to see a. blog post called bitter lesson oh yes. but rich sutton that makes an argument. hopefully i can. summarize it perhaps perhaps you can. yeah but. okay so i i mean i could try and you can. correct me which is uh. he makes an argument that it seems if we. look at the long arc. of the history of the artificial. intelligence field it calls you know 70. years. that the algorithms from which we've. seen.

the biggest improvements in practice are. the very simple. like dumb algorithms that are able to. leverage computation. and you just wait for the computation to. improve like all the academics and so on. have fun by. finding little tricks and and. congratulate themselves on those tricks. and sometimes those tricks can be like. big that feel in the moment like big. spikes and breakthroughs but in reality. over the decades it's still the same. dumb algorithm that just. waits for the compute to get faster and.

faster do you find. that to be an interesting argument. against the entirety of the field of. machine learning. that's an academic discipline that we're. really just a subfield of computer. architecture yeah. we're just kind of waiting around for. them to do we really don't want to do. hardware work so like that's right i. really don't want to. we're procrastinating yes that's right. just waiting for them to do their job so. that we can pretend to have done ours so. uh yeah i mean the argument reminds me a. lot of.

i think it was a fred jelinek quote uh. early computational linguist who said. you know we're building these. computational linguistic systems and. every time we fire a linguist. performance goes up by ten percent. something like that and so the idea of. us building the knowledge in. in that in that case um was much less he. was finding to be much less successful. than get rid of the people who know. about language as a. you know from a kind of scholastic. academic kind of perspective and replace.

them with more compute. and so i think this is kind of a modern. version of that story which is okay we. want to do better on. machine vision you could build in all. these you know. motivated part-based models that you. know that just feel like. obviously the right thing that you have. to have or we can throw a lot of data at. it and guess what we're doing better. with it with a lot of data. so i i hadn't thought about it until. this moment in this way. but what i believe well i've thought. about what i believe.

what i believe is that you know. compositionality and. what's the right way to say it the. complexity grows. rapidly as you consider more and more. possibilities. like explosively and so far moore's law. has also. been growing explosively exponentially. and so so it really does seem like well. we don't have to. think really hard about the algorithm. design or the way that we build the. systems. because the best benefit we could get is.

exponential and the best benefit that we. can get from waiting is exponential. so we can just wait it's got that's. gotta end right and there's hints now. that that moore's law is. is starting to feel some friction uh. starting to. the world is pushing back a little bit. um one thing i. i don't know do lots of people know this. i didn't know this i was i was trying to. write an essay and yeah moore's law has. been amazing and it's been it's enabled. all sorts of things but there's a. there's also a kind of counter moore's.

law which is that the development cost. for each. successive generation of chips also is. doubling. so it's costing twice as much money so. the amount of development money per. cycle or whatever is actually sort of. constant and at some point. we run out of money uh so or we have to. come up with an entirely different way. of. of doing the development process so like. i. i guess i always always a bit skeptical. of the look it's an exponential curve. therefore it has. no end soon the number of people going. to nurips will be greater than the.

population of the earth. that means we're going to discover life. on other planets no it doesn't it means. that we're in a. in a sigmoid curve on the front half. which looks a lot like an exponential. the second half is going to look a lot. like diminishing returns. yeah the i mean but the interesting. thing about moore's law if you actually. like look at the technologies involved. it's. hundreds if not thousands of s-curves. stacked on top of each other it's not. actually an exponential curve. it's constant breakthroughs and and.

then what becomes useful to think about. which is exactly what you're saying. the cost of development like the size of. teams the amount of resources that are. invested. in continuing to find new s-curves new. breakthroughs. and yeah it's uh it's an interesting. idea. you know if we live in the moment if we. sit here today. it seems to be the reasonable thing to. say that. exponentials end and yet in the software.

realm. they just keep appearing to be happy. anyway. and it's so i mean it's so. hard to disagree with elon musk on this. because. it it like i i've. you know i used to be one of those folks. i'm still one of those folks i've. studied autonomous vehicles that's what. i worked on and. and it's it's like you look what elon. musk is saying about autonomous vehicles. well obviously in a couple years. or in a year or next month we'll have.

fully autonomous vehicles like there's. no reason why we can't driving is pretty. simple. like it's just a learning problem and. you just need to convert. uh all the driving that we're doing into. data and just having you all know with. the trains on that data. and uh like we use only our eyes so you. can use cameras and. you can train on it and it's like yeah. that's that what that should work. and then you put that hat on like the. philosophical hat. and but then you put the pragmatic hat.

and it's like this is what the flaws of. computer vision are like. this is what it means to trans scale and. then you you put the. human factors the psychology hat on. which is like. it's actually driving us a lot the. cognitive science or cognitive whatever. the heck you call it. is it's really hard it's much harder to. drive than. than we realize there's much larger. number of edge cases. so building up an intuition around this. is uh. around exponential is really difficult. and on top of that.

the pandemic is making us think about. exponentials. making us realize that like we don't. understand anything about it we're not. able to intuit exponentials we're either. that's true ultra terrified some part of. the population and some part is like. uh the opposite of whatever the. carefree and we're not managing. everything. blase well wow that's that french. uh it seems so it's got so it's uh.

it's fascinating to think what what. the limits of this exponential. growth of technology not just moore is. law. it's technology how that rubs up against. the bitter lesson and gpt-3. and self-play mechanisms like it's not. obvious. i used to be much more skeptical about. neural networks now. at least give a slither possibility that.

we'll be. all though will be very much surprised. and. also you know uh. caught in a way that like. we uh are not prepared for. like in applications of um. social networks for example sure because. it feels like. really good transformer models that are. able to do. some kind of like very good. uh natural language generation of the. same kind of models that could be used.

to learn human behavior and then. manipulate that human behavior to. gain advertiser dollars and all those. kinds of things sure uh. feed the capitalist system and and right. so they arguably already are. manipulating human behavior. yeah yeah so but not for. self-preservation. which i think is a big that would be a. big step like if they were trying to. manipulate us. to convince us not to shut them off i. would be very freaked out. but i don't see a path to that from.

where we are now. they they don't have any of those. abilities that's not what they're trying. to do they're trying to keep people on. on the site but see the thing is this. this is the thing about life on earth. is they might be borrowing our. consciousness and sentience like. so like in a sense they do because the. creators of the algorithms. have like they're not you know if you. look at our body. okay we're not a single organism we're a. huge number of organisms with like tiny.

little motivations we're built on top of. each other. in the same sense the ai algorithms that. are they're not. it's a system that includes human. companies and corporations right because. corporations are. funny organisms in and of themselves. that really do seem to have self. preservation built in and i think that's. at the at the design level i think. they're designed to have. self-preservation. be a focus so you're right in that in. that broader. system that we're also a part of. and can have some influence on uh.

it's it's it is much more complicated. much more powerful yeah i agree with. that. uh so people really love it when i ask. what three books. technical philosophical fiction had a. big impact in your life maybe you. couldn't recommend. we went with movies we went uh. with uh billy joel and i forgot what you. uh. what music you recommended but i didn't. i just said i have no taste in music i. just like pop music that was actually. really uh skillful the way you thank you.

that question i'm going to try to do the. same with the books. so do you have a skillful way to avoid. answering the question about three books. you would recommend. i'd like to tell you a story. so um my first job out of college was at. bellcore i mentioned that before. where i worked with dave ackley the head. of the group was a guy named tom. landauer and i don't know. how well known he's known now but. arguably he's the. he's the inventor and the first. proselytizer of word embeddings. so they they developed a system shortly.

before i got to the group. yeah um that that uh called latent. semantic analysis that would take. words of english and embed them in you. know multi-hundred dimensional space. and then used that as a way of uh you. know assessing similarity and basically. doing reinforcement learning. not sorry not reinforcing information. retrieval you know sort of pre-google. information retrieval. and he was trained as an anthropologist. but then. became a cognitive scientist so i was in. the cognitive science research group.

it's you know like i said. i'm a cognitive science groupie um at. the time i thought i'd become a. cognitive scientist but then i realized. in that group. no i'm a computer scientist but i'm a. computer scientist who really loves to. hang out with cognitive scientists. and he said he studied language. acquisition in particular he said you. know humans have about. this number of words of vocabulary and. most of that is learned from reading. and i said that can't be true because i. have a really big vocabulary. and i don't read he's like you must i'm.

like i don't think i do i mean like stop. signs i definitely read stop signs. but like reading books is not it's not a. thing that i do. really though it might be just no i. might be the red color. do i read stop signs yeah no it's just. pattern recognition at this point i. don't sound it out. um so now i do. i wonder what that oh yeah stop the guns. so um that's fascinating so you don't uh. so i don't read very i mean obviously i. read and i've read.

i've read plenty of books um but like. some people like. charles my friend charles and and and. others like a lot of people in my field. a lot of. academics like reading was really a. central topic to them. in development and i'm not that guy in. fact i used to joke. that um when i got into college that it. was on. kind of a help out the illiterate kind. of program because i got to. like in my house i wasn't a particularly. bad or good reader but when i got to. college i was surrounded by these people. that were just.

voracious in their reading appetite and. they were like have you read this have. you read this have you read this. and i'd be like no i'm clearly not. qualified to be at this school like. there's no way i should be here. now i've discovered books on tape like. audiobooks. um and so i'm i'm much better uh i'm. more caught up i read a lot of books. a small tangent on that it is a. fascinating open question to me. on the topic of driving whether. you know supervised learning people.

machine learning people think you have. to like. drive to learn how to drive to me it's. very possible that just by. us humans by first of all walking but. also by. watching other people dr not even being. inside cars as a passenger. but let's say being inside the car as a. passenger but even just. like being a pedestrian and crossing the. road you learn. so much about driving from that it's. very possible. that you can without ever being inside.

of a car. be okay at driving once you get in it uh. or like watching a movie for example. yeah i don't know. something like that it's have you have. you taught anyone to drive. no so i have myself i have two children. and um i learned a lot about car driving. because my wife doesn't want to be the. one in the car while they're learning so. that's my job. yeah so i sit in the passenger seat and. it's really scary. um you know i have wishes to live um.

and they're you know they're figuring. things out now they start off. very very much better than i imagine. uh like a neural network would right. they get that they're seeing the world. they get that there's a road that. they're trying to be on they get that. there's a relationship between the angle. the steering. but it takes a while to not be very. jerky. and so that happens pretty quickly like. the ability to stay in lane. at speed that happens relatively fast. it's not zero shot learning but. it's pretty fast the thing that's.

remarkably hard and this is i think. partly why self-driving cars are really. hard. is the degree to which driving is a. social interaction activity yes. and that blew me away i was completely. unaware of it until i watched my son. learning to drive. and i was realizing that he was sending. signals to all the cars around him. and those in his case he's he's always. had. social communication challenges. he was sending very mixed confusing. signals to the other cars and that was. causing the other cars to drive.

weirdly and erratically and there was no. question in my mind. that he would he would have an accident. because. they didn't know how to read him there's. things you do with the the speed that. you drive the positioning of your car. that you're constantly like in the head. of the other drivers. and seeing him not knowing how to do. that and having to be taught explicitly. okay you have to be thinking about what. the other driver is thinking. was a revelation to me yeah i was. supposed to be really. so so creating kind of uh theories of.

mind of the other theories of mind of. the other cars. yeah yeah which i just hadn't heard. discussed in the self-driving car. talks that i've been to since then. there's some people who do do. consider those kinds of issues but it's. way more subtle than i think. there's a little bit of work involved. with that when you realize. like when you especially focus not on. other cars but on pedestrians for. example. it's it's a literally staring you in the. face yeah yeah yeah so that when you're. just like how do i interact with. pedestrians.

um yeah like pedestrians you're. practically talking to an octopus at. that point they've got all these. weird degrees of freedom you don't know. what they're going to do they can turn. around any second but the point is. we humans know what they're going to do. like we have a good. theory of mind we have a good mental. model of what they're doing. and we have a good model of the model. that have a view. and the model of the model of the model. like they're we're able to kind of. reason about this kind of. uh the social like game of it. uh all the hope is that it's quite.

simple actually. that it could be learned that's what i. just talked to the waymo i don't know if. you know that company it's. google south africa they i talked to. their cto. about this podcast and they like i wrote. in their car. and it's quite aggressive and it's quite. fast and it's good and it feels. great it make it also just like tesla. waymo made me change my mind about like. maybe driving is easier than i thought. maybe i'm just being. speciesist human.

maybe uh it's a speciesist argument yes. i don't know. but it it's fascinating to think about. like the same. as with reading which i think you just. said you avoided the question. but i still hope you answered in some. way we avoided it brilliantly. it is there's blind spots there's. artificial intelligence. that artificial intelligence researchers. have about what it actually. takes to learn to solve a problem have. you had anka dragon.

on yeah okay one of my favorites so much. energy. she's right oh she yeah she's amazing. fantastic and and in particular she. thinks a lot about this kind of. i know that you know that i know kind of. planning and. the last time i spoke with her she was. very articulate about the. ways in which self-driving cars are not. solved. like what's still really really hard but. even her intuition is limited. like we're all like new to this uh so in. some sense the elon musk approach of. being ultra confident and just like.

put it out there putting it out there. like some people say it's reckless and. dangerous and so on. but like partly it's like it seems to be. one of the only ways to make progress in. artificial intelligence so it's uh. it's you know these these are difficult. things you know democracy. is messy uh uh implementation of. artificial. intelligence systems in the real world. is messy so. many years ago before self-driving cars. were an actual thing you could have a. discussion about. somebody asked me like what if what if.

the what if we could use that robotic. technology and use it to drive cars. around. like isn't that aren't people going to. be killed and then it's not you know. blah blah blah. i'm like that's not what's gonna happen. i said with confidence. incorrectly obviously uh what i think is. gonna happen is we're gonna have a lot. more. like a very gradual kind of rollout. where people. have these cars in like closed. communities. right where it's somewhat realistic but. it's still. in a box right so that we can really get. a sense of what. what are the weird things that can.

happen how do we. how do we have to change the way we. behave around these vehicles like. it obviously requires a kind of. co-evolution. that you can't just plop them in and see. what happens. but of course we're basically popping. them in to see what happens so i was. wrong but i. do think that would have been a better. plan so that's but your intuition that's. funny. just zooming out and looking at the. forces of capitalism. and it seems that capitalism rewards. risk takers. and rewards and punishes risk takers.

like it. and like try it out. the academic approach. to let's try a small thing. and try to understand slowly the. fundamentals of the problem. and let's start with one and do two and. then see that. and then do the three uh you know uh the. the capitalist like startup. entrepreneurial dream is. let's build a thousand and let's right. and 500 of them fail but whatever the. other 500 we. learned from them but if you're good.

enough i mean one thing it's like your. intuition would say like. that's going to be hugely destructive to. everything but. actually it's kind of the the the forces. of capitalism. people are quite it's easy to be. critical but if you actually look at the. data. at the way our world has progressed in. terms of the quality of life. it seems like the competent good people. rise to the top. this is coming from me from the soviet. union and. so on it's like it's interesting.

that somebody like elon musk is the way. you uh you push progress in artificial. intelligence like it's forcing way more. to step this. their stuff up uh and waymo is forcing. uh. elon musk to step up. it's fascinating i because i have this. tension in in my. heart and just being upset by. the lack of progress in autonomous. vehicles and within academia. so there's a huge progress in the early.

days. of the darpa challenges and then it just. kind of stopped. like at mit but it's true everywhere. else. with an exception of a few sponsors here. and there. is is like it's not seen as a sexy. problem. uh thomas like the moment artificial. intelligence starts. approaching the problems of the real. world. like academics kind of like ah all right. let let the couple get really hard in a.

different way in a different way and. that's right i think yeah right some of. us are not. excited about that other way but i still. think there's fundamentals problems to. be. solved in those difficult things it's. not it's still. publishable i think like we just need to. it's the same criticism you could have. of all these conferences in europe's. cvpr where application papers. are often as powerful and as important. as like. uh theory paper even like theory just. seems. much more respectable and so on i mean.

machine learning community is changing. that a little bit. i mean at least in statements but it's. it's still. not seen as the sexiest of uh pursuits. which is like how do i actually make. this thing work in practice. as opposed to on this toy data set. all that to say are you still avoiding. the three books question is there. something on audiobook that you can uh. recommend. oh i've yeah i mean um i yeah i've read. a lot of really fun stuff. uh in terms of books that i find myself.

thinking back on that i read a while ago. like that have stood the test of time to. some degree i find myself thinking of. program or be programmed a lot by. douglas. roshkopf um which. was it basically put out the premise. that we all need to become. programmers in one form or another. and it was an analogy to once upon a. time we all had to become. readers we had to become literate and. there was a time before that when not. everybody was literate but once literacy.

was possible. the people who were literate had more of. a say. in society than the people who weren't. and so we made a big effort to get. everybody up to speed and now it's. it's not 100 universal but it's quite. widespread. like the assumption is generally that. people can read. the analogy that he makes is that. programming is a similar kind of thing. that uh. that we need to have a say. in right so being a reader being. literate being a reader means you can.

receive all this information. but you don't get to put it out there. and programming is the way that we get. to put it out there. that was the argument he made i think he. specifically has now. backed away from this idea he doesn't. think it's happening quite this way. and that might be true that it didn't. society. didn't sort of play forward quite that. way i still believe in the premise i. still believe that at some point. we have the relationship that we have to. these machines and these networks. has to be one of each individual can has.

the wherewithal. to make the machines help them. do do the things that that person once. done and as so you know as software. people we know how to do that and we. have a problem we're like okay i'll just. i'll hack up a perl script or something. and make it so. if we lived in a world where everybody. could do that that would be a. better world and computers would be have. i think. less sway over us and other people's. software would have less sway over us. as a group yeah in some sense software. engineering programming's power.

it's programming is power right it's. it's yeah it's like magic it's like. magic spells and. and it's not out of reach of everyone. but at the moment it's just a sliver of. the population who can. who can commune with machines in this. way so i don't know so that book had a. big. big impact on me currently i'm i'm. reading uh the alignment problem. actually by brian christian so i don't. know if you've seen this out there yet. is this similar to stuart russell's work. with the control problem. it's in in that same general. neighborhood i mean they take they have.

different. emphases that they're they're. concentrating on i think i think. stewart's book. did a remarkably good job like a just a. celebratory good job at describing. ai technology and sort of how it works i. thought that was great it was really. cool to see that in a book. yeah i think he has some experience. writing some books. you know that's probably a possible. thing he's maybe thought a thing or two. about how to explain. ai to people yeah yeah that's a really. good point um this book so far. has been remarkably good at telling the.

story. of the sort of the history the recent. history. of some of the things that have happened. uh this i'm in the first third he said. this book is in three thirds the first. third is. essentially ai fairness and you know. implications of ai on society that we're. seeing right now. and that's been great i mean he's. telling the stories really well he's he. went out and talked to the frontline. people who. whose names are associated with some of. these ideas and and it's been terrific. he says the second half of the book is. on reinforcement learning so.

maybe that'll be fun um and then the. third half. third third is on uh this is super. intelligence alignment problem. and i i suspect that that part will be. less fun for me to read. yeah it's yeah it's. it's an interesting problem to talk. about i find it to be the most. interesting just like. thinking about whether we live in a. simulation or not as a. as a thought experiment to think about. our own existence. so in the same way talking about.

alignment problem with agi. is a good way to think similarly like. the trolley problem with autonomous. vehicles. it's a useless thing for engineering but. it's a it's a nice little thought. experiment for. actually thinking about what are like. our own. human ethical systems our moral systems. to. to to uh by thinking how we engineer. these things you start to understand. yourself. so sci-fi can be good at that too so one. sci-fi book to recommend is.

exhalations by ted chang a bunch of. short stories. um this ted chang is the guy who wrote. the short story that became the movie. arrival. um and all his stories just from a. he's he was a computer scientist. actually he studied at brown. they all have this sort of really. insightful bit of. science or computer science that drives. them and so it's just. a romp right to just like he creates. these artificial worlds with these. by extrapolating on these ideas that.

that we know about. but hadn't really thought through to. this kind of conclusion and so his stuff. is. it's really fun to read it's mind. warping. so i'm not sure if you're familiar i. seem to mention this every other word. uh is i'm from the soviet union and i'm. russian. uh read way too much my roots are. russian too. but a couple generations back well it's. probably in there somewhere so maybe we. can uh. we can pull up that thread a little bit. of the existential dread.

that we all feel you mentioned that you. i think somewhere in the conversation. you mentioned they you don't. really pretty much like dying i forget. in which context. it might have been a reinforcement. learning perspective i don't know i know. you know what it was it was. in teaching my kids to drive. that's that's how you face your. mortality yes uh. from a human being's perspective or from. a reinforcement learning researcher's. perspective let me ask you the most. absurd question. what's uh what do you think is the. meaning of this whole thing.

the meaning of life on this spinning. rock. i mean i think reinforcement learning. researchers maybe think about this from. a science perspective more often. than a lot of other people right as a. supervised learning person you're. probably not thinking about the sweep of. a lifetime but reinforcement learning. agents are. having little lifetimes little weird. little lifetimes and it's. it's hard not to project yourself into. their world sometimes. but you know as far as the meaning of.

life so i when i turned 42. you may know from that's a that is a. book i read um. the the historical hitchhiker's guide to. the galaxy. that that is the meaning of life so when. i turned 42 i had a meaning of life. party. where i invited people over and um. everyone shared their meaning of life we. they. we had slides made up and so we had we. all sat down and. did a slide presentation to each other. about the meaning of life. and mine mine was balance.

i think that life is balance and um. so the activity at the party for a 42. year old maybe this is a little bit. non-standard. but i i found all the little toys and. devices that i had that where you had to. balance on them you had to. like stand on it and balance or pogo. stick i brought. a ripstick which is like a weird. two-wheeled skateboard. um i got a unicycle but i didn't know. how to do it i didn't know how to do it. i now can do it i love watching you try. yeah i'll send you a video.

i'm not great but i put but but i. managed. um and so uh so balanced yeah so so. my my wife has a really good one that. she. sticks to and is probably pretty. accurate and it has to do with. healthy relationships with people that. you love. and working hard for good causes but to. me yeah balance balance. in a word that's that that works for me. not too much of anything because too. much of anything is. iffy that feels like uh rolling stone.

song i feel like they must be. you can't always get what you want but. if you try sometimes. you can strike a balance yeah i think. that's how it goes. uh michael i'll write your parody it's a. huge honor to talk to you this. been a big fan of yours so um uh. can't uh can't wait to see what you do. next in the world of uh education the. world of parity in the world of. reinforcement learning thanks for. talking today my pleasure.

thank you for listening to this. conversation with michael littman and. thank you to our sponsors. simplisafe a home security company i use. to monitor. and protect my apartment expressvpn. the vpn i've used for many years to. protect my privacy and the internet. masterclass online courses that i enjoy. from some of the most amazing humans in. history. and better help online therapy with a. licensed professional. please check out the sponsors in the. description to get a discount and to. support this podcast.

if you enjoy this thing subscribe on. youtube review. five stars napa podcast follow on. spotify. support it on patreon or connect with me. on twitter. at lex friedman and now let me leave you. some. words from groucho marx if you're not. having fun. you're doing something wrong thank you. for listening and hope to see you. next time.

you.