Charles Isbell and Michael Littman: Machine Learning and Education | Lex Fridman Podcast #148

the following is a conversation with. charles isbell and michael littman. charles is the dean of the college of. competing at georgia tech. and michael is a computer science. professor at brown university. i've spoken with each of them. individually on this podcast. and since they are good friends in real. life we all thought it would be fun. to have a conversation together quick. mention of each sponsor. followed by some thoughts related to the. episode thank you to. athletic greens the all-in-one drink. that i start.

every day with to cover all my. nutritional bases. ate sleep a mattress that cools itself. and gives me yet another reason to enjoy. sleep master class online courses from. some of the most. amazing humans in history and cash app. the app i use to send money. to friends please check out the sponsors. in the description to get a discount. and to support this podcast as a side. note let me say that. having two guests on the podcast is an. experiment that i've been meaning to do. for a while. in particular because down the road i.

would like to occasionally. be a kind of moderator for debates. between. people that may disagree in some. interesting ways if you have suggestions. for who you would like to see debate. on this podcast let me know as with all. experiments of this kind it is a. learning process. both the video and the audio might need. improvement. i realized i think i should probably do. three or more cameras next time as. opposed to just two. and also try different ways to mount the.

microphone for the. third person also after recording this. intro. i'm going to have to go figure out the. thumbnail. for the video version of the podcast. since i usually put the guest's. head on the thumbnail and now there's. two heads. and two names to try to fit into the. thumbnail it's a kind of bin packing. problem. which in uh theoretical computer science. happens to be an np hard problem.

whatever i come up with if you have. better ideas for the thumbnail let me. know as well. and in general i always welcome ideas. how this thing can be improved. if you enjoy it subscribe on youtube. review it with five stars on apple. podcast. follow on spotify support on patreon or. connect with me on twitter. and lex friedman and now here's my. conversation. with charles isbell and michael littman. you'll probably disagree about this. question.

but what is your biggest would you say. disagreement. about either something profound and very. important or something completely not. important at all. i don't think we have any disagreements. at all ah i'm not sure that's true. we walked into that one didn't we yeah. so one thing that you sometimes mention. is that and we did this one on air too. as it were. whether or not machine learning is. computational statistics. it's not but it is well it's not. and in particular and more importantly.

it is not just computational statistics. so what's missing in the picture what. all the rest of it. what's missing that which is missing oh. because well you can't be wrong now well. it's not just the statistics he doesn't. even believe this. we've had this conversation before if it. were just the statistics then. we would be happy with where we are but. it's not just the statistics that's why. it's computational statistics or if it. were just the computation i agree that. machine learning is not just statistics. it is not just this we can agree on that. nor is it just computational statistics. it's computational statistics it is. computational what is the computational.

and computational statistics. does this take us into the realm of. computing it does but i think. perhaps the way i can get him to admit. that uh he's wrong. is that it's about rules it's about. rules. it's about symbols it's about all these. other things statistics it's not about. rules i'm going to say statistics is. about rules but it's not just the. statistics right it's not just a random. variable that you choose and you have a. probability i think you have a narrow. view of statistics. okay well then what would be the broad. view of statistics that would still. allow it to be statistics and not say. history that would make computational.

statistics okay. well okay so i i had my first sort of. research mentor. a guy named tom landauer taught me to do. some statistics. right sure and and i was annoyed all the. time because the statistics would say. that what i was doing was not. statistically significant. and i was like but but but and basically. what he said to me is. statistics is how you're going to keep. from lying to yourself. which i thought was really deep it is a.

way. to keep yourself honest in a particular. way i agree with that. yeah and so you're trying to find rules. i'm just kind of bringing back to rules. wait wait wait could you possibly try to. define. rules even regular statisticians. non-computational statisticians. do spend some of their time evaluating. rules right applying statistics to try. to understand is this you know is this. does this rule capture this does this. not capture. i mean like hypothesis testing kind of. or like confidence intervals like. like have like more like hypothesis like.

i feel like the word statistic literally. means like a summary. like a number that summarizes other. numbers right but i think the field of. statistics actually applies that idea. to things like rules to understand. whether or not a rule is. valid the software engineering. statistics. no programming languages statistics no. because i think there's a very it's. useful. to think about a lot of what ai and. machine learning is or certainly should. be. as software engineering uh as. programming languages. just if to put it in language that you.

might understand. in the hyper parameters beyond the. problem the hyper parameters has too. many syllables for me to understand. the hyperparameters of uh that's better. that goes around it right it's the. decisions you choose to make it's the. metrics you choose to use it's the loss. you want to say the practice. of machine learning is different than. the practice of statistics like the. things you have to worry about and how. you worry about them are different. therefore they're different right at a. very little i mean at the very least. it's that that much is true it doesn't. mean that statistics computational or.

otherwise aren't important. i think they are i mean i do a lot of. that for example. but i think it goes beyond and i think. that we could think about game theory in. terms of statistics but i don't think. it's very. as useful to do i mean the way i would. think about it or a way i would think. about it. is this way chemistry is just. physics but i don't think it's as useful. to think about chemistry as being just. physics it's useful to think about it as. chemistry the level of abstraction. really matters here so i think it is. there are contexts in which it is useful.

that way right so finding that. connection is actually helpful and i. think that's when i when i emphasize the. computational statistics thing i think. i think i want to befriend statistics. and not absorb them here's the here's. the a way to think about it beyond what. i just said right so. what would you say and i want you to. think back to a conversation we had a. very long time ago what would you say is. the difference between. say the early 2000's icml and what we. used to call nips nerfs. was there a difference a lot of the. particularly on the machine learning. that was done there.

icmo was around that long oh yeah so. iclear is the new conference. newish uh yeah i guess so and i see him. i was. around the 2000 oh i see male predates. that. i well i think my most cited icml paper. is from 94. yeah. michael knows this better than me. because of course he's significantly. older than i am but the point is. yeah what is the difference what was the. difference between icml and nureps. in the late 90s early 2000s i don't know. what everyone else's perspective would. be but i had a particular perspective at. that time. which is i felt like icml was more of a.

of a computer science. place and that nips nerfs was more of an. engineering place. like the kind of math that happened at. the two places. as a computer scientist i felt more. comfortable with the icml math. and the nurbs people would say that. that's because i'm dumb. and that's such an engineering thing to. say so i agree with that part of it but. i. do a little differently actually i had a. nice conversation with tom dietrich. about this. in public on twitter just a couple days. ago i put it a little differently which. is that icml was.

machine learning done by uh computer. scientists. and uh nurbs was machine learning done. by computer scientists. trying to impress statisticians. which was weird because it's the same. people at least by the time i started. paying attention. but it just felt very very different and. i think that that perspective of whether. you're trying to impress the. statisticians or you're trying to. impress the programmers is actually very. different and has real impact on. what yeah what you choose to worry about. and what kind of uh.

outcomes you come to so i think it. really matters in computational. statistics is a means to an end it is. not an end in some sense. um and i think that really matters here. in the same way that i don't think. computer science is just engineering or. just science or just math or whatever. but. okay so i'd have to now agree that now. we agree on everything. yes yes the important thing here is that. you know my opinions may have changed. but not the fact that i'm right. i think is what what we just came to. right now my opinions may have changed. and not the fact that i'm wrong. that's right i lost me i'm not i think i.

lost myself there too but anyway. this happens to us sometimes we're sorry. how does neural networks change. this just to even linger on this topic. change this idea of statistics. how big of a pi statistics is within the. machine learning thing. like because it sounds like hyper. parameters and also just the role of. data. you know this people are starting to use. this terminology software 2.0. which is like the act of programming as.

a as a. like you're a designer in the. hyperparameter space of neural networks. and you're also the collector and the. organizer and the cleaner. of the data and that's part of the. programming. uh so how did on the. versus icml topic what's the role of. neural networks and redefining. the size and the role of machine. learning i can't i can't wait to. hear what michael thinks about this but.

um i would add one well but. that's true i'll force myself to i think. the the. there's one thing i would add to your. description which is the kind of. software engineering part is what does. it mean to debug for example. but this is a difference between uh the. kind of computational statistics view. of machine learning and the. computational view of machine learning. which is i think one is worried about. the equation as it were and. by the way this is not a value judgment. i just think it's about. perspective but the kind of questions. you would ask when you start asking. yourself what does it mean to program. and develop and build the system. it's a very computer sciencey view of.

the problem. i mean when if you get on data science. twitter and econ twitter. you actually hear this a lot with the. uh you know the economist and the data. scientist complaining about the machine. learning people well. you know it's just statistics and i. don't know why they don't don't see this. but they're not even asking the same. questions they're not. thinking about it as a kind of. programming problem. and i think that that really matters. just asking this question i actually. think it's a little different from. programming and hyper parameter space.

and sort of collecting. the data but i do think that that. immersion really matters so i'll give. you a quick. a quick example the way i think about. this so i teach machine learning michael. and i have. co-taught a machine learning class which. has now reached i don't know 10 000. people at least over the last several. years or somewhere there's abouts and. my machine learning assignments are of. this form so the super. the first one is something like. implement these five algorithms you know. k. n and s you know svms and boosting and. decision trees and.

neural networks and maybe that's it i. can't remember and when i say implement. i mean steal the code. i am completely uninterested you get. zero points for getting the thing to. work. i don't want you spending your time. worrying about uh getting the corner. case right. of you know what happens when you are. trying to normalize distances and the. points on the thing and so you divide by. zero i'm not interested in that right. steal the code however you're going to. run those algorithms. on two data sets the data sets have to. be interesting. what does it mean to be interesting well. data says interesting if it reveals.

differences between algorithms which. presumably are all the same. because they can represent whatever they. can represent and two data sets are. interesting together if they. show different differences as it were. and you have to analyze them you have to. justify their interestingness and you. have to analyze in a whole bunch of ways. but all i care about is the data in your. analysis not the programming and i. occasionally end up in these long. discussions with students well. i don't really i copy and paste the. things that i've said the other 15. 000 times it's come up which is they go. but the only way to learn. really understand is to code them up.

which is a very. programmer software engineering view of. the world if you don't program it you. don't understand it. which is by the way i think is wrong in. a very specific way. but it is a way that you come to. understand because then you have to. wrestle with the algorithm. but the thing about machine learning is. it's not just sorting numbers where in. some sense the data doesn't matter what. matters is. well does algorithm work on these. abstract things and one less than the. other in machine learning. the data matters it does it matters more. than almost anything. and not everything but almost anything. and so.

as a result you have to live with the. data and don't get distracted by. the algorithm per se and i think that. that focus on the data. and what it can tell you and what. question it's actually answering for you. as opposed to the question you thought. you were asking is a key and important. thing. about machine learning and is a way that. computationalists as opposed to. statisticians bring a particular. view about how to think about the. process the statisticians by contrast. bring. i i think i'd be willing to say a better. view about the kind of formal.

math that's behind it and what an actual. number. ultimately is saying about the data and. those are both important but they're. also different. i didn't really think of it this way is. to build intuition about. the role of data the different. characteristics of data by having two. data sets that are different. and they reveal the differences in the. differences that's that's a really. fascinating that's a really interesting. educational. approach the students love it but not. right away. no they love it later i love it at the. end not at the beginning.

not even not even immediately after i. feel like there's a deep. profound lesson about education there. yeah. that uh you can't listen to students. about. whether what you're doing is the right. or the wrong thing. well as a wise uh michael litman once. said to me. about children which i think applies to. teaching is you have to give them. what they need without bending to their. will. and students are like that you have to. figure out what they need you're a. curator your whole job is to curate.

and to present because on their own. they're not going to necessarily know. where to search so you're providing. pushes in some direction and learn space. and you have to give them what they need. in a way that keeps them engaged enough. so that they eventually discover. what they want and they get the tools. they need to go and learn other things. what's your view let me put on my. russian hat. which believes that life is like russian. hats by the way if you have one i would. like. those are ridiculous yes but in a.

delightful way but sure. what do you think is the role of uh we. talked about balance a little bit. what do you think is the role of. hardship in education. like i think the biggest things i've. learned. like what made me fall in love with math. for example. is by being bad at it until i got good. at it. so like like struggling with a problem. which increased the level of joy i felt.

when i finally figured it out. and it always felt with me with teachers. especially modern discussions of. education how can we make. education more fun more engaging more. all those things. or from my perspective it's like you're. maybe missing the point. that education that life is suffering. education is supposed to be hard and. that actually what. increases the joy you feel when you. actually learn something. is that ridiculous do you like to see.

your students suffer. okay so this may be a point where we. differ i'd suspect not. i'm gonna do go on well what would your. answer be i wanna hear you first. okay well i would i was gonna not answer. the question. do you know what this dude is i wasn't. gonna hear them suffering no no no no no. i was i was gonna say that there's. i think there's a distinction that you. can make in the kind of suffering right. so. i think you can be in a mode where. you're you're suffering. in a hopeless way versus you're.

suffering in a hopeful way. right where you're like you can see. that if you that you still have you can. still imagine getting. to the end right and as long as people. are in that mindset where they're. struggling but. it's not a hopeless kind of struggling. that's. that's productive i think that's really. helpful but it's struggling like if you. break their will. if you leave them hopeless no that don't. sure some people are gonna whatever lift. themselves up by their bootstraps but.

like. mostly you give up and certainly it. takes the joy out of it and. you're not going to spend a lot of time. on something that brings you no joy. so it's it's it is a bit of a delicate. balance right you have to thwart people. in a way that they still believe that. there's a way through. right so that's a that we strongly agree. actually so i think. well first off struggling and suffering. aren't the same thing right. being poetic oh no no i actually. appreciate the poetry and. i one of the reasons i appreciate it is. that they are often the same thing and.

often quite different right so. you can struggle without suffering you. can certainly suffer and suffer. suffer pretty easily you don't. necessarily have to struggle to suffer. so i think that. you want people to struggle but that. hope matters. you have to they have to understand that. they're going to get through it on the. other side. and it's very easy to confuse the two. i actually think brown university has a. very just. philosophically has a very different. take on the relationship with their. students particularly undergrads from. say. a place like georgia tech which is which.

universities better. uh well i have my opinions on that i. mean remember charles said. it doesn't matter what the facts are i'm. always right the correct thing. is that it doesn't matter they're. different um but. clearly he went to a school. like the school where he is as an. undergrad i went to a school. specifically the same school though it. was it changed a bit in the in the. intervening years. brown or georgia tech no i was talking. about georgia tech and i went yeah. and i went to an undergrad place that's. a lot like the place where i work now. and so it does seem.

like we're more familiar with these. models there's a similarity between. brown and yellow. yeah there's a i think that i think. they're quite similar yeah and duke. duke has some similarities too but it's. got a little southern. draw you've kind of worked here you sort. of worked at universities that are like. the places where. you learned and. the same would be true for me are you. uncomfortable uh. venturing outside the box is that what. you're saying. journeying out what i'm saying yeah. charles is definitely he only goes to.

places that have institute in the name. right it has worked out that way well. academic places anyway. well no i was a visiting scientist at. upenn or visiting. visiting something at upenn oh wow i. just i just understood. your joke which one. five minutes later i like to set these. sort of time bombs. the institute is in the uh uh that. charles only goes to places that have. institute. in the name so i guess georgia. i forget that georgia tech is georgia. institute of technology the number of.

people who refer to it as georgia tech. university is large and incredibly. irritating. that's one of the few things that. generally gets under my schedule but. like schools like georgia tech and mit. have as part of the ethos like there is. i want to say there's a there's an. abbreviation. that someone taught me like i htfp. something like that like there's a. there's a. there's an expression which is basically. i hate being here which they say. so proudly and that is definitely not. the ethos at brown like brown is. there's a little more pampering and.

empowerment and stuff and it's not like. we're gonna crush you and you're gonna. love it. so yeah i think there's a i think the. ethos. are different mm-hmm that's interesting. yeah we had drone proofing. what's that trump graduate from georgia. tech this is a true thing feel free to. look it up. uh if you a lot of schools have this by. the way. no actually georgette was barely the. first brandeis has it had it. i feel like georgia tech was the first. in the look first of all. it was it was the first time i think um. had the first time.

stop that first masters in computer. science actually right online masters. well that too but way back in the 60s um. nsf yeah yeah. you're the first information and. computer science master's degree in the. country. um but the uh georgia tech it used to be. the case in order to graduate from. georgia tech uh you had to take a drown. proofing class. where effectively they threw you water. tied you up. if you didn't drown you got to graduate. i believe so. there were certainly versions of it but. i mean luckily they ended it just before.

i had to graduate because otherwise. would have never graduated. it wasn't going to happen uh i want to. say 84 or 83 someone around then they. they ended it but uh yeah you used to. have to prove you could tread water for. some ridiculous amount of time are you. two yeah you couldn't graduate no it was. more than two hours two minutes. okay it was in a bathtub. it was in a pool but it was a real thing. but that idea that you know push you. fully clothed yeah fully clothed i don't. think i bet it was that and not tied up. because like who needs to learn how to. swim when you're tied.

nobody but who needs to learn when to. swim when you're actually falling into. the water dressed that's a real thing. i think your facts are getting in the. way with a good story oh that's fair. that's fair i didn't. think all right so they didn't tell you. what the narrative mattered but whatever. it was you had to it was called drown. proofing for a reason the point of the. story. michael uh is struggle it it's well no. but that's good it doesn't. bring it back to struggle that's a part. of what georgia tech has always been and. we struggle with that by the way. uh about what we want to be as things go. but you you sort of how much can you.

be pushed without breaking and you come. out of the other end stronger right. there. there's this saying we said when i was. an undergrad there which is georgia tech. building tomorrow the night before. right kind of idea that you know. give me something impossible to do and. i'll do it in a couple of days because. that's what i just spent the last four. or five. or six years that ethos definitely stuck. to you. having now done a number of projects. with you you definitely will do it the. night before that's not entirely true.

there's nothing wrong with waiting until. the last minute the secret is knowing. when the last minute is. right that's brilliant that's. brilliantly put yeah that yeah that's. that is a definite. charles statement that i am trying not. to embrace. and i appreciate that because you helped. move my last minute that's the social. construct that we converge together what. the definition of last minute is and we. we figure that out all together in fact. mit. you know i'm sure a lot of universities. have this but mit has like mit time that.

yeah everyone has always agreed together. that. there is such a concept and everyone. just keeps showing up like 10 to 15. to 20 depending on the department late. to everything. so there's like a weird drift that. happens it's kind of fascinating yeah. we're five minutes. five minutes in fact the classes will. say you know well this is no longer true. actually. but it used to be a class was started. eight but actually started 805. yeah it ends at nine actually ends at 8. 55. uh everything's five minutes off and. nobody expects anything to start until. five minutes after the half hour or.

whatever it is. it still exists it hurts my head well. let's rewind the clock. back to the 50s and 60s when you guys. met. how did you i'm just kidding i don't. know but what can you tell the story of. how you met so. you've like the internet and the world. kind of knows you as. as as connected in some ways. in terms of education of teaching the. world that's. that's like the public facing thing but. how did you as human beings. and as collaborators. meet i think there's two stories one is.

how we met. and the other is how we got to know each. other. i'm not gonna say fellaini i'm gonna say. that we came to understand that we. had some common something yeah it's. funny because on the surface i think. we're. we're different in a lot of ways but. there's something yeah i mean that's. just consonant there you go afternoon. so i will tell the story of how we met. and i'll let michael tell the story of. how we. okay all right okay so here's how we met. um i was already at that point it was.

18t labs. there's a long interesting story there. but anyway i was there and uh. michael was coming to interview he was a. professor at duke at the time but. decided for reasons that he wanted to be. in new jersey. uh and so that would mean uh bell lab. slash att labs. uh and we were doing interview. interviews very much like academic. interviews uh and so i had to be there. uh we all had to meet with him. afterwards and so on one on one. but it was obvious to me that he was. gonna be hired. like no matter what because everyone.

loved him they were just talking about. all the great stuff he did and. oh he did this great thing and you just. won something at triple a i think or. maybe you got 18 papers in triple either. but. i got the best paper award at your play. for the crosswords right exactly. so that it all happened and everyone was. going on and on and on about actually. tinder was saying incredibly nice things. about you really yes so he can be very. grumpy yes that's very that's nice to. hear he was grumpily saying very nice. things oh that's that makes sense and. that does make sense so you know so. it was going to come so why were we why. was i meeting him i had something else i. had to do i came here what it was yeah.

it probably involved commenting he. remembers meeting me as inconveniencing. his afternoon. so he came so eventually came to my. office i was in the middle trying to do. something i can't remember what and he. came and he sat down and for. reasons that are purely accidental. despite what michael thinks. my desk at the time was set up in such a. way that had. sort of an l shape and the chair on the. outside was always lower than the chair. that i was in. and you know the kind of point was the. only reason i think that was on purpose. is because you told me it was on purpose. i don't remember that anyway the thing. is that you know it kind of his guest. chair was really low so that he could.

yeah he could look down at everybody the. idea was just to simply create a nice. environment that you were asking for a. mortgage and i was going to say no that. was a. very simple idea here anyway so we sat. there and we just talked for a little. while and i think he got the impression. that i didn't like him. that wasn't true strongly the talk was. really good. by the way it was terrible and after. right after the talk i said to my host. michael kearns who ultimately was. my boss i'm a huge fan i'm a friend and. a huge fan of michael yeah yeah he is a. remarkable. person um i i after my talk today.

i went into this i went back at ball. he's good at that. basketball no but basketball racquetball. squash which is not. racquetball yes squash no and i hope you. you hear that michael. you mean like your parents as a game not. his skill level because i'm pretty sure. he's all right there's some. competitiveness there but the point. is that it was like the middle of the. day i had full day of interviews like i. met with people but then in the middle. of the day i gave. a job talk and then um and then there.

was going to be more interviews but. i i pulled michael aside and i said. i think it's in both of our best. interests if i just leave now. because that was so bad that it's just. be embarrassing if i have to talk to any. more people like you look bad for having. invited me. like it's just let's just forget this. ever happened. so i don't think the talk went well it's. one of the most michael littman set of. sentences i think i've ever heard. he did great or at least everyone knew. he was great so maybe it didn't matter. i was there i remember the talk and i.

remember him being very much the way i. remember him now. in any given week so it was good and we. met and we talked about stuff he thinks. i didn't like him but because he was so. grumpy. must been the chair thing the chair. thing and the low voice i think. the like obviously and that like that. like slight like. skeptical look yeah i have no idea what. you're talking about. well i probably didn't have any idea. what you were talking about. anyway i liked him he asked me questions. i answered questions i felt bad about. myself it was a normal day.

then he left and then he left and that's. how you tell me can we take it. and then i got hired and i was in the. group can we take a slight tangent on. that on this topic of. it sounds like uh maybe you could speak. to the bigger picture. it sounds like you're quite. self-critical who charles. no you oh i think i can i can do better. i can do better i'll. try me again i'll i'll do better. yeah that was like a like a three out of. ten responses. so let's try to work it up to five and. six uh you know i remember.

uh marvin minsky said uh on on a video. interview. something that the key to success in. academic research is to hate everything. you do. for some reason i think i followed that. because i hate everything he's done. [Laughter]. uh it's a good line that's a success. maybe that's a keeper but um but do you. do find that resonates with you at all. in how you think about talks and so on. i would say it differently it's not.

really that's such an mit view of the. world though. so i remember i i remember talking about. this when uh as a student you know. you were basically told uh i will clean. it up for the purpose of the podcast. um uh my work is crap my work is crap my. work is crap my work is crap then you. like go to a conference or something. like everybody else's work is crap. everybody else is working crap and you. feel better and better about it yeah. uh relatively speaking and then you sort. of keep working on it. i don't hate my work that resonates with. me yes i've never hated my work but i. have.

i have been dissatisfied with it. and i think being dissatisfied being. okay with the fact that you've taken a. positive step the derivative is positive. maybe even the second derivative is. positive that's important because that's. a part of the the hope right. but you have to but i haven't gotten. there yet if that's not there that i. haven't gotten there yet. then you know it's hard to it's hard to. move forward i think so i buy that. which is a little different from hating. everything that you do yeah i mean. there's. there's things that i've done that i. like better than i like myself.

so it's separating me from the work. essentially so i think i am very. critical of myself. but sometimes the work i'm really. excited about and sometimes i think it. doesn't happen right away so i found the. work that i've. liked that i've done most of it. i liked it in retrospect more when i was. far away from it in time. i have to be fairly excited about it to. get done. no excited at the time but then happy. with the result or but years later or. even i might go back you know what. that actually turned out to be yeah that.

turned out to matter or oh gosh it turns. out i've been thinking about that. it's actually influenced all the work. that i've done since without realizing. it. but that guy was smart yeah that guy had. a future. yeah yeah he's going places. i think there's so yeah so i think. there's something to it i think there's. something to the idea you've got to. you know hate what you do but it's not. quite hate it's just being unsatisfied. and different people motivate themselves. differently i don't happen to motivate. myself with self-loathing. i happen to motivate myself so you're. able to sit back and.

be proud of in retrospect of the work. you've done. well and it's easier when you can. connect with other people because then. you can be proud of them. a lot of the people yeah and then the. questions you can still safely hate. yourself. it's a win-win michael or at least win. lose which is what you're looking for. oh wow there's so many brilliant lines. in this. there's levels uh so how did you. actually meet me. yeah so my the way i think about it is. because we didn't do much. research together at 18t but um but then.

we all got laid off. so so that was that by the way i decided. to interrupt but that was like. one of the most magical places. historically speaking. they did not appreciate what they had. and how do we uh i feel like there's a. profound lesson in there too uh. how do we get it like what was why was. it so magical is just the coincidence of. history. or is there something special some. really good managers and people who. really believed in. machine learning as this is going to be.

important. um let's get the the people who are. thinking about this in creative and. and insightful ways and put them in one. place and. stir yeah but even beyond that right it. was. it was bell labs at its heyday and even. when we were there which i think was. past it. to be clear he's gotten to be at bell. labs i never got to be at bell labs. i joined after that yeah i should have. been 91 as a grad student. so i was there for a long time um every. summer except. twice i worked for companies that had. just stopped being better labs right.

bell core and then att labs so about. labs was. several locations or for the for the. research or is it what like. jerseys are involved somehow they're all. in jersey yeah they're all over the. place but they're in a couple places. murray hill was the bell labs um. so you you had you had an office in mary. hill at one point in your career. yeah and i i played ultimate frisbee on. the cricket pitch at bell labs at murray. hill. uh and then it became 18t labs when. split off with loose during what we. called uh tri-vestiture supposedly.

better than michael koren's ultimate. frisbee yeah oh yeah. okay but i think that one's not boasting. i think that i think charles plays a lot. of ultimate and i don't think mike i was. yes but but that wasn't the point the. point is yes yes. sorry okay i have played on a. championship winning ultimate frisbee. team. or whatever ultimate team with charles. so i know. how good he is he's really good how good. i was anyway when i was younger but the. thing is i know how young he was when he. was yeah that's true. that was true so much younger than now. he's old enough yeah i'm older michael.

is a much. was a much better basketball player than. i was michael kearns yes no not michael. i'm very clear so you don't know how. terrible i am. but you have a probably pretty good. guess that you're not as good as michael. kearns. he's tall and and he cared about it very. outlet he's very good he's. probably competitive i love hanging out. with michael anyway but we were talking. about something else although i no. longer remember what it was what were we. talking about. but also labs so so uh this was kind of. cool about what was magical about it.

the first thing you have to know is that. bell labs was an arm of the government. right because att was an army of. government. it was a monopoly uh and you know every. month you paid a little. thing on your phone bill which turned. out was a tax for like all the research. that bell labs was doing. and you know they invented transistors. and the laser and whatever else is that. big bang or whatever the. cosmic background radiation yeah they. did all that stuff they had some amazing. stuff with directional microphones by. the way i got to go in this room. um where they they had all these panels. and everything. and we would talk and one another and he. moved some panels around and then.

he would have me step two steps to the. left and i couldn't hear a thing he was. saying because nothing was bouncing off. the walls. and then he would shut it all down and. you could hear your heartbeat yeah. deeply disturbing to hear your heart. beat you can feel it i mean you can feel. it now there's so much all this sort of. noise around anyway bill labs is about. pure research. it was a university in some sense the. purest sense of a university. but without students so it was all the. faculty working with one another. and students would come in to learn they. would come in for three or four months. you know during the summer and they. would go away.

but it was just this kind of wonderful. experience i could walk out my door. in fact i would often have to walk out. my door and deal with rich sutton and. michael kearns yelling at each other. about whatever it is they were yelling. about. the proper way to prove something or. another and i could just do that and. dave mcallister and evan. and peter stone and and all of these. other people including. satinder and then eventually michael and. it was just a place where you could. think. thoughts and it was okay because so long. as once every 25 years or so. somebody invented a transistor it paid.

for everything else you could afford to. take the risk. and then when that all went away it. became harder. and harder and harder to justify it as. far as the folks who were very far away. were concerned. and there was such a fast turnaround. among middle management. on the atnt side that you never had a. chance to really build the relationship. at least people like us didn't have a. chance to. to build relationships so when the. diaspora happened um. it was amazing right yeah everybody left. and i think everybody ended up at a. great place and. made a huge made a continued to do.

really good work with with machine. learning but it was a wonderful place. and people will ask me you know what's. the best job you you've ever had. and as a professor anyway the answer. that i would give is. um well probably. bell labs in some very real sense and i. would never have a job like that again. because bell labs doesn't exist anymore. and you know microsoft research is great. and google does good stuff and you can. pick ibm. you can tell if you want to but bell. labs was magical it was around for it. was an important time. and it represents a a high water mark in.

in basic research in the u.s is there. something you could say about the. physical proximity and the chance. collisions. like we live in this time of the. pandemic where everyone. is maybe trying to see the silver lining. and accepting the remote nature of. things. is is there one of the things that. people like faculty. that i talk to miss is the. the procrastination like the chance to. like everything is about meetings that.

are supposed to be there's not a chance. to just. uh you know talk about comic book or. whatever like go into discussion that's. totally pointless. so it's funny you say this because. that's how we met matt. it's exactly that so i'll let michael. say that but i'll just add one thing. which is just that uh. you know research is a social process. and it helps to have. random social interactions even if they. don't feel social at the time that's how. you get things done. one of the great things about the a lab. when i was there i. don't quite know what it looks like now. once they moved buildings but we had. entire walls that were whiteboards and.

people would just get up there and they. were just right. and people would walk up and you'd have. arguments and you'd explain things to. one another. and you got so much out of the freedom. to do that you had to be. okay with people challenging every. freaking word you said which i would. sometimes find. deeply irritating but most of the time. it was it was quite useful but the sort. of pointlessness and the interaction was. in some sense the point at least for me. yeah i mean you. i think offline yesterday i mentioned. josh tannenbaum and he's. very much he put he's a man he's such an.

inspiration. in in the child like. way that he pulls you in on any topic it. doesn't even have to be about machine. learning. it could or or the brain he'll just pull. you into a closest. writable surface which is uh still you. can find whiteboards at mit everywhere. and and just like uh like basically. cancel all meetings and talk for a. couple hours about some. some aimless thing and it it feels like. the whole world the time. space continuum kind of warps and that.

becomes the most important thing. and then it's just it's so true it's. it's. definitely something worth missing in. this in this world where everything's. remote. there's some magic to the physical. presence whenever i wonder myself. whether mit really is as great as i. remember it i just go talk to josh yeah. you know that's funny is. there's a few people in this world that. carry the. the best of what particular institutions. stand for right and it's uh it's josh. i mean i i don't i my guess is he's.

unaware of this. that's the point that the masters are. not. aware of their mastery so how do we all. meet. yes but but first a tangent no. how did you meet me so i'm not sure what. you were thinking of but my. when it started to dawn on me that maybe. we had a longer-term bond. was after we all got laid off and. you had decided at that point that there. we were still paid we were given an. opportunity to like do job search and.

kind of make a transition. but it was clear that we were done and. i would go to my office to work and you. would go to my office to keep me from. working. that was that was my recollection of it. and you had decided that there was no. really no point in working for the. company because the company our. relationship with the company was. was done yeah but remember i felt that. way beforehand it wasn't about the. company it was about the set of people. there doing really cool things and it. always. always been that way but we were working. on something together oh yeah. yeah that's right oh so at the very end. we all got laid off but then.

our boss came to our boss's boss came to. us. because our boss was michael kearns and. he had jumped ship. brilliantly like perfect timing like. things like right before the ship was. about to sink. he was like gotta go and and and. landed perfectly because michael kearns. because michael king. and um leaving the rest of us to go like. this is fine and then it was clear that. wasn't fine and we were all. toast so we had this sort of long period. of time but then our boss figured out.

okay wait maybe we can save a couple of. these people. if we can have them do something really. useful. and uh the useful thing was we were. going to make a. basically an automated assistant that. could help you with your calendar you. could like. tell it things and it would it would. respond appropriately it would just kind. of integrate across. all sorts of your personal information. and so me and charles and peter stone. were this were set up as the crack team. to actually solve this problem.

uh other people maybe were too. theoretical that they thought and. and but we could actually get something. done so we sat down to get something. done and there wasn't time. and it wouldn't have saved us anyway and. so it all kind of went downhill. but the interesting i think coda to that. is that our boss's boss is a guy named. ron brockman. and he when he left at t. because we were all laid off he went to. darpa. started up a program there that became. kalo. which is the program from which siri.

sprung. which is a digital assistant that helps. you with your calendar and a bunch of. other things. um it really you know in some ways got. its start. with me and charles and peter trying to. implement this vision that ron brockman. had that. he ultimately got implemented through. his role at darpa. so when i'm trying to feel less bad. about having been laid off from. what is possibly the greatest job of all. time. i think about well we kind of helped. birth siri.

that's something and he did other things. too but. the we got to spend a lot of time in his. office and. talk about we got to spend a lot of time. in my office yeah. yeah yeah and so uh so then we went on. our merry way. everyone went to different places. charles landed at georgia tech which was. uh what he always dreamed he would do. and so. um that worked out well yeah um i. came up with a saying at the time which. is luck favors the charles. it's kind of like luck favors the. prepared but charles like.

like he'd wish something and then it. would basically happen just the way he. wanted it was. it was inspirational to see things go. that way things worked out and we stayed. in touch. and then um i think it really helped. when you were working on i mean you kept. me in the loop for things like threads. and the work that you were doing at. georgia tech but then. when they were starting their online. master's program he knew that i was. really excited about. moocs and online teaching and he's like. i have a plan and i'm like tell me your. plan he's like i can't tell you the plan.

yet because they were. deep in in negotiations between georgia. tech and udacity to make this happen. and they didn't want it to leak so. charles would kept teasing me about it. but wouldn't tell me what was actually. going on and eventually it was announced. and he said i would like you to teach. the machine learning course with me. i'm like that can't possibly work um but. it was a great idea and it was. it was super fun it was a lot of work to. put together but it was it was really. great. and was that the first time you thought. about first of all. was it the first time you got seriously. into teaching.

i mean you know i'm trying to get the. feeling right i'll tell you this is. already. after you jump to so like there's a. little bit of. jumping around in time yeah sorry about. it there's a pretty big jump in time so. like the moocs. thing so charles got to georgia tech and. he i mean maybe charles maybe this is a. trick. in 2002. he got to georgia tech in 2002. and um but then and worked on things. like revamping the curriculum the. undergraduate curriculum so that it had. some kind of. semblance of modular structure because. computer science was. at the time moving from a fairly narrow.

specific set of topics to touching a lot. of other parts. of of of intellectual life and the. curriculum. was supposed to reflect that and so um. charles played a big role in. in kind of redesigning that and then and. for my and for my. my labors i ended up his associate dean. right he got to become an associate dean. of in charge of educational stuff well. this would be a valuable lesson if. you're. good at something uh they will give you.

responsibility to do more of that thing. well until you don't show confidence. don't show confidence if you. well you know what the responsibility. here's what they say yeah. the reward for good work is more work. the reward for bad work is less work. which i don't know depending about what. you're trying to do that week. one of those is better than the other. well one of the problems with the word. work sorry to interrupt. is that it's seems to be an antonym. in this particular language we have the.

opposite of happiness. but it seems like they're they're like. that's one of you know we talked about. balance it's uh it's always like. work-life balance it always rubbed me. the wrong way as. a terminology i know it's just words. right the opposite of work is play. but yeah ideally work is play oh i can't. tell you how much time i'd spend. certainly i was about labs except for a. few very key moments. uh as a professor i would do this too i. was just saying cannot believe they're. paying me to do that um because it's fun. it's something that i would i would do.

for a hobby. if i could anyway uh so that sort of. worked out i'm sure you want to be. saying that. when this is being recorded as a dean. that is not true at all. i need a raise yes but but i think here. with with this that even though a lot of. time passed. you know michael and i talked almost. every well we texted almost every day. during the period charles at one point. took me. there was the icml conference the. machine learning conference was in. atlanta i was the chair the general. chair of the conference.

charles was my publicity chair or. something like that or. something fundraising champion sure yeah. um but he decided it'd be really funny. if he didn't actually show up for the. conference in his own home city. uh so he didn't but he did at one point. picked me up at the conference in his. tesla and drove me to the atlanta mall. and forced me to buy an iphone because. he didn't like. how it was to text with me and thought. it would be better for him. if i had an iphone the text would be. somehow smoother.

and it was and it was and it is and his. life is better and my life is better and. so. death but but it was yeah charles. forcing me to get an iphone. so that he could text me more. efficiently i thought that was an. interesting moment it works for me. anyway so we kept talking the whole time. and then eventually we did the. we did the teaching thing and it was. great and there's a couple of reasons. for that by the way one is. i really wanted to do something. different like you've got this medium. here. people claim it can change things what's. a thing that you could do in this medium. that you could not do otherwise. besides edit right i mean what could you.

do and and being able to do something. with another person was that kind of. thing it's very hard. i mean you can take turns but teaching. together having conversations is very. hard right. so that was a cool thing the second. thing it gave me an excuse to do more. stuff with him yeah i always thought. he makes it sound brilliant um and it is. i guess but it's at the time it really. felt like. i've got a lot to do charles is saying. and it would be great if michael could. teach. the course and i could just hang out. yeah just kind of coast on that. well that's what the second class was.

more like that because the second time. that was explicit because. the first class it was at least half so. the structure. so that was that was kind of true yeah. that was sort of true for 7642 which is. the reinforcement learning class because. that was really his class you started. with reinforcement. no we started with i did the intro. machine learning 7641. uh which is supervised learning. unsupervised learning and.

reinforcement learning and decision. making cram all that in there the kind. of assignments that we talked about. earlier. and then eventually about a year later. we did a follow-on 7642 which is. reinforcement learning and decision. making. the first class was based on something. i'd been teaching at that point for well. over a decade. and the second class was based on. something michael had been teachers. actually i learned quite a bit. teaching that class with him but he. drove most of that but the first one i. drove most it was all my material. although i had stolen that material. originally from slides i found online. from michael who had originally stolen. that material from i guess slides he.

found online probably from andrew moore. because the jokes were the same. anyway at least some of the at least. when i found the slides some of the. stuff. yes every machine learning class taught. in the early 2000's stole from. andrew moore a particular joke or two. the at least the structure now i did. and he did actually a lot more with. reinforcement learning and such and game. theory and those kinds of things but. you know we all sort of know this world. no no no no i mean. teaching that class the coverage was. different than than what other people. were starting most people were just. doing supervised learning and maybe a. little bit of.

you know clustering and whatnot but we. took it all the way to a lot of it. just comes from tom mitchell's book oh. no yeah except well half of it comes. from tom mitchell's book right i mean. the other half doesn't. this is what this is why it's all. readings right because certain things. weren't invented when tom wrote. okay that's true right uh but it was it. was quite good but there's a reason for. that besides. you know just i wanted to do it i wanted. to do something new and i wanted to do. something with him. which is a realization which is despite. what you might believe. he's an introvert and i'm an introvert. or i'm on the edge of it.

or being an introvert anyway but both of. us i think. um enjoy the energy of the crowd right. there's something about. talking to people and bringing them into. whatever we find interesting that is. empowering energy energizing or whatever. and i found the idea. of staring alone uh at a computer screen. and then talking off of materials. less inspiring than i wanted it to be. and i had in fact done a mooc for. udacity on. algorithms and it was a week.

in a dark room talking at the screen. writing on the little pad and i didn't. know this was happening but they had. watched the. crew had watched some of the videos. while like in the middle of this and. they're like. something's wrong you're you're. sort of shutting down um and i think a. lot of it was i'll make. jokes and no one would laugh yeah and i. felt like the crowd hated me now of. course there was no crowd. so like it wasn't rational yeah but it's.

little each time i tried it and i got no. reaction it just was taking. the the the energy out of my performance. out of my. presentation fantastic metaphor for grad. school anyway by working together. we could play off each other and have it. and and have it keep the energy up. because you can't. you with you can't let your guard down. for a moment with charles he'll just. he'll just. overpower you i have no idea what's. wrong with but we would work really well. together i thought and we knew each. other so i knew that we could we could. sort of make it work plus i was the. associate dean so. they had to do what i told him to do we.

had to do that we had to make it work. and so it worked out very well i thought. um well enough that we. with great power comes great power. that's right and we became smooth and. curly. and uh that's when we we we did the. the um the uh overfitting thriller video. yeah we took yeah yep that's a thing. so what okay can we just like like uh. smooth and curly where was that so okay. so. that happened it was completely. spontaneous these are the nicknames you. go by yeah so. uh students call us he was he was. lecturing.

so the way that we structured the. lectures is one of us is the lecturer. and one of us is basically the student. and so the he was lecturing on the. lecturer prepares all the materials. comes up with the quizzes. and then the student comes in not. knowing anything so it's you know just. like being on campus. yeah uh and i was doing game theory in. particular. the prisoner's delivery's dilemma and so. he needed to set up a little prisoner's. dilemma grid so he drew it and i could. see what he was drawing. and the the prisoner's dilemma consists. of two players two parties so he decided. he would make little cartoons of the two.

of us and so. there was uh two criminals right that. were deciding whether or not to rat each. other out. um one of them he drew as you know a. circle with a smiley face. and a kind of goatee thing smooth head. and the other one with all sorts of. curly hair and he said this is smooth. and curly i said smooth and curly he. said no smoove. with a v it's very important that it. happened v and then talk actually. and the students really the students. really took to that like they've really. they found that relatable. he started singing smooth criminal by.

michael jackson yeah yeah yeah and. that's. those those names stuck so that so we. now have a video series the. an episode our kind of first actual. episode should be coming out today. um smoove and curly on video. where the two of us discuss uh west. episodes of westworld we watch westworld. and we're like. huh what does this say about computer. science and ai. and we've never we did not watch it i. mean i know it's on season three or. whatever we have. as of this recording it's on season. three and uh that's now two episodes. total yeah i think it was three.

what do you think about westworld two. episodes in so i can tell you guys. so far yeah i'm just guessing what's. going to happen next. it seems like bad things are going to. happen with the robot's uprisings. so i no i have not i mean you know i. vaguely remember a movie existing so i. assume it's it's related to that. but that was more my time than your time. charles that's right because you're much. older than i i think the important thing. here. is that uh it's narrative right it's all. about telling a story that's the whole. driving thing. but the idea that they would give these. reveries that they would make.

people they would make them remember. remember the awful things that happened. that happened. who could possibly think that was good i. got a i mean i don't know i've only seen. the first two episodes or maybe the. third one i think i've only said you. know what it was you know what the. problem is. that the robots were actually designed. by hannibal lecter that's true. they were so like what do you think is. going to happen it's a bad thing. it's clear that things are happening and. characters are being introduced and we. don't yet know anything. but still i was just struck by how it's. all driven by narrative and story and. there's all these implied things like.

programming hap the programming. interface is talking to them about. what's going on in their heads. which is both i mean artistically it's. probably useful to film it that way but. think about how it would work in real. life that just seems very crazy but. there was. we saw in the second episode there's a. screen you could see things they were. wearing like. in the world it was quite interesting to. just kind of ask this question. so far i mean i assume it veers often to. never neverland at some point. but uh we can't answer that question i'm. also a fan of a guy named alex garland. he's a director of ex machina.

and he is the first i wonder if kubrick. was like this actually. is he like studies. what would it take to program in ai. systems like he's he's curious enough to. go into that direction. on the west wall side i felt there was. more emphasis on the narratives than. like actually. asking like computer science questions. yeah like how would you. build this how would you uh and how. would you debug it i still think. to me that's the key issue they were.

terrible debuggers. yeah and well they said specifically so. we make a change and we put it out in. the world and that's bad because. something terrible could happen. like if you're putting things out in the. world and you're not sure whether. something terrible is going to happen. you're probably your process is probably. i just feel like there should have been. someone whose sole job it was to walk. around and poke his head. and say what could possibly go wrong. just over and over again. i would have loved if there was an and i. did watch a lot more i'm not giving. anything away. i would have loved it if there was like. an episode where like. like the new intern is like debugging a.

new model or something and like it just. keeps failing and. they're like all right and then it's. more turns into like a. episode of silicon valley or something. like that yes versus like. all this ominous ai systems that are. constantly like. threatening the the fabric of this world. that's been created. yeah yeah and you know the other the. this this reminds me of something that. so i agree with that that actually be. very cool at least well for the small. percentage of people who care about. debugging. systems but the other thing is debugging. the series. it falls into think of the sequels fear.

of the debate oh my gosh. and anyway so a nightmare show it's a. it's a horror movie. i think that's where we lose people by. the way early on is the people who. either decide either figure out. debugging or think debugging is terrible. this is where we lose people in computer. science this is part of the struggle. versus. suffering right you you get through it. and you kind of get the skills of it or. you're just like this is dumb and this. is a dumb way to do anything and. i think that's when we lose people but. um i. well i'll leave it at that but i think. that i think that that there's something.

really really neat about framing it that. way but what i. don't like about all of these pro all of. these things and i love text mocking and. by the way i thought the. ending was very depressing um but again. one of the things after talking to alex. about. he says that the thing that nobody. noticed he put in. is uh the at the end spoiler alert. the the robot turns and looks at the. camera. and smiles right briefly and to him.

he thought that his definition of. passing. the touring the general version of the. turing test or the consciousness test. is smiling for no one hmm. oh like like. not oh you know it's it's like the. chinese room kind of. experiment it's not always trying to act. for others right but just on your own. being able to have a relationship with. the actual experience. and just like take it in i don't know he.

said like nobody noticed. i mean the magic of it i had this vague. feeling that i remember the smile but. now you now you just put the memory in. my head so probably not but i do think. that. that's interesting although by looking. at the camera you are smiling for the. audience right you're breaking the. fourth wall. it seems i mean well that's a that's a. limitation in the medium but i. i like that idea but here's the problem. i have with all of those movies all of. them. um is that but i know why it's this way. and i enjoy those movies. um and westworld is uh it sets up the. problem of ai.

as succeeding and then having something. we cannot control. but it's that's not the bad part of ai. the bad part of ai is the stuff we're. living through now right it's the. using the data to make decisions that. are terrible it's not the intelligence. that's going to go out there and surpass. us and you know take over the world or. you know lock us into a room to starve. to death slowly. over multiple days it's instead uh the. the tools that we're building that are. allowing us. to make the terrible decisions we would.

have less efficiently made before. right you know computers are very good. at making us more efficient. including being more efficient at doing. terrible things and and that's the part. of the ai we have to worry about it's. not the. you know true intelligence that we're. going to build sometime in the future. probably long after we're around. um but you know i i i i just. i think that whole framing of it sort of. misses. the point even though it is inspiring. and i was inspired by those ideas right. that i got into this in part because i. wanted to build something like that.

philosophical questions were interesting. me but but you know that's not where the. terror comes from the terror comes from. the every day. and you can construct situations in the. subtlety of the interaction between ai. and. the human like with the with social. networks. all the stuff you're doing with uh. interactive artificial intelligence. but you know i i feel like cal 9000 came. a little bit closer to that when it's. in 2001 space odyssey because it felt. like. uh a personal assistant you know it felt. like closer to the ai systems we have.

today and. and the real things we might actually. encounter which is. over relying uh on in some fundamental. way on our. like dumb assistance or on social. networks like over. offloading too much of us onto uh. you know onto things that require. internet and power and so on and thereby. becoming powerless. as a stand-alone entity and then when.

that thing starts to misbehave. in some subtle way it creates a lot of. problems. and those problems are dramatized when. you're in space because. you don't have a way to walk away well. as the man said. um once you once we started making the. decisions for you it stopped being your. world. right that's the matrix michael in case. you don't i didn't generally i don't. remember. but on the other hand i could say no. because isn't that what we do with. people anyway. you know this kind of the shared. intelligence that is humanity is relying. on other people constantly to i mean we.

hyper-specialize right. as individuals we're still generally. intelligent we make our own decisions in. a lot of ways but we leave most of this. up to other people and. that's perfectly fine and by the way. everyone doesn't necessarily. share our goals sometimes they seem to. be quite against us sometimes we. make decisions that others would see as. against our own interests and yet we. somehow manage it. manage to survive i'm not entirely sure. why an ai. would actually make that worse. or even different really you mentioned.

the matrix. do you think we're living in a. simulation it does. feel like a thought game more than a. real scientific question well i'll tell. you why like i think it's an interesting. thought experiment see what you think. from a computer science perspective. it's a good experiment of how difficult. would it be. to create a sufficiently realistic world. that us humans would enjoy being in. it that that's almost like if we're. living in a simulation then i don't. believe that we were put in the.

simulation i believe that. it's just physics playing out and we. came out of that. like i don't i don't i don't think so. you think you have to build the universe. yeah i think. the universe itself we can think of that. as a simulation and in fact. what i try sometimes i try to think. about to understand. what it's like for a computer to. start to think about the world i try to. think about the world. things like quantum mechanics where it. doesn't feel very natural to me at all.

and it really strikes me as i don't. understand this thing that we're living. in. it has there's weird things happening in. it that don't feel natural to me at all. now if you want to call that as the. result of a simulator. okay i'm fine with that but like i don't. know the bugs in the simulation. there's the bugs i mean the interesting. thing about the simulation is that it. might have bugs i mean that that's the. thing that i the but there would be bugs. for the people in the simulation they're. just that's just. reality unless you were fair enough to. know that there was a bug. but i i think back to the matrix yeah.

the way you put this. i don't think that we live in a in a. simulation created for us. okay i would say that i think that's. interesting i've actually never thought. about it that way i mean. you the way you asked the question. though is could you create a world that. is. enough for us humans it's an. interestingly sort of self-referential. question because. the beings that created the simulation. probably have not created a simulation. that's realistic for them. but we're in the simulation and so it's. realistic for us. so we could create a simulation that is. fine for the. people in the simulation as it were that.

would not necessarily be fine for us as. the creators of the simulation. but well you can you can forget i mean. when you go into the. if you play video games in virtual. reality you can. if with some suspension of disbelief or. whatever yeah uh. it becomes a world it becomes the world. even like in brief moments. you forget that another world exists i. mean that's what like good stories do. they pull you in and the question is is. it possible to pull. you know our brains are limited is it. possible to pull the brain in.

to where we actually stay in that world. longer and longer longer and longer. and like not only that but we don't want. to leave. and so especially this is the key thing. about the. developing brain is if we journey into. that world early on in life. often how would you even know yeah yeah. so i. but like from a video game design. perspective from a west world. perspective it's. i think i think it's an important thing. for even. uh computer scientists to think about.

because it's clear that video games. are getting much better and virtual. reality. although it's been ups and downs just. like artificial intelligence. it feels like virtual reality will be. here in a very impressive form if we. were to fast forward 100 years into the. future. in a way that might change society. fundamentally like. if i were to i'm very limited in. predicting the future as all of us are. but if i were to try to predict.

like in which way i'd be surprised to. see the world. 100 years from now it'd be. that or impressed it'd be that we're all. no longer living in this physical world. that we're all living in a virtual world. you really need to be calculating god by. sawyer it's a. you'll read it in a night it's a very. easy read but it's. assuming you're that kind of reader but. it's a it's a good story. and it's kind of about this but not in a.

way that it appears. and i uh really enjoyed. the thought experiment um i think it's. pretty sure it's robert sawyer but. anyway he's he's. apparently canadian's top science. fiction writer which is why this story. mostly takes place in toronto. uh but it's a it's a very good uh it's a. very good sort of. story that sort of imagines this. very different kind of simulation. hypothesis sort of thing from say um. the egg for example you know you know. i'm talking about the short story.

um by the guy who did the martian. who wrote the martian mm-hmm you know. matt damon. no the book so we had this whole. discussion. that michael doesn't uh doesn't partake. in this exercise of reading. yeah he doesn't seem to like it which. seems very strange to me considering how. much he has to read. i read all the time i used to read 10. books at every week when i was a. when i was in sixth grade or whatever i. was a lot of it science fiction a lot of. it. a lot of history but i i love to read. but anyway you should recalculating god.

i think you'll. you'll it's very easy read like i said i. think you'll enjoy. sort of the ideas that it presents yeah. i think the the thought experiment is. quite interesting. uh one thing i've noticed about people. growing up now. i mean we'll talk about social media but. video games is a much bigger bigger and. bigger and bigger part of their lives. and then the video games have become. much more realistic i think. it's possible that the three of us. are not uh and maybe the two of you are.

not familiar exactly with the numbers. we're talking about here. the number of people it's bigger than. movies right it's it's. it's huge i used to do a lot of the. narrative computational narrative stuff. i understand that economists can. actually see the impact of video games. on the labor market that there are there. there's fewer young men of a certain age. participating in like paying jobs than. you'd expect and and that they trace it.

back to video games i mean the problem. with star trek. was not warp drive or teleportation. it was the holodeck like if you have the. holodeck. that's it that's it you go in the. holodeck you never come out i mean. it just never made once i saw that i. thought okay well. so this is the end of humanity as we. know it right they've invented the. holiday. because that feels like the singularity. not some agi or whatever. it's some possibility to go into another. world. that can be artificially made better.

than this one. and slowing it down so you live forever. or speeding it up so you appear to live. forever or. making the decision of when to die and. then. most of us will just be old people on. the porch yelling at the kids these days. in their virtual reality. [Music]. but they won't hear us because they've. got headphones on so. i mean rewinding back to moocs. is there lessons that you've uh speaking. of kids these days. uh that was a transition all right i'll.

edit i'll fix it and post. yeah that's charles's favorite phrase. fix it in post fix it in post exit in. post they said all when we were. recording all the time whenever. the editor didn't like something or. whatever i would say we'll fix it in. post. he hated that yeah he hated that more. than anything because charles's way of. saying i'm not going to do it again. [Laughter]. you know you're on your own for this one. but it always got fixed in post. exactly right so uh is there something. you've learned about. i mean it's interesting to talk about. moocs is there something you've learned.

about the process of education about. thinking about the present i think. there's two lines. of conversation to be had here is the. future of education. in general that you've learned about and. more presciently is the. education in the times of covid yeah. well the second thing in some ways. matters more than the first. um for at least in my head for the not. just because it's happening now but. because. um i think it's it's reminded us of a. lot of things coincidentally today.

there's an article out by a good friend. of mine um who's also a professor of. georgia tech but more importantly a. writer and editor at the atlantic i. named ian bogus. um and the title is something like. americans will sacrifice anything for. the college experience. and it's about why we went back to. college and why. people wanted us to go back to college. and it's not you know greedy presidents. trying to get the last dollar from. someone it's because they want to go to. college and what they're paying for is. not the classes what they're paying for.

is the college. experience it's not the education it's. being there i've believed this for a. long time. that we continually make this mistake of. people want to go back to college as. being people want to go back to class. they don't they want to go back to. campus they want to move away from home. they want to do all those things that. people experience it's a rite of passage. it's a. it's a identity if i can if i can steal. some of them. ian's words here and i think that's. right and i think what we've learned. through kovid. is it has made it the disaggregation was.

not the disaggregation of the education. from the place. universe the university place and that. you can get the best anywhere you want. to. it turns out there's lots of reasons why. that is not necessarily true the. disaggregation. is having it shoved in our faces that. the reason to go again that the reason. to go to college. is not necessarily to learn it's to have. the college experience. and that's very difficult for us to. accept even though we behaved that way. most of us when we were undergrads you. know a lot of us didn't go to every. single class. we learned and we got it and we look.

back on it and we're happy we had the. learning experience as well obviously. particularly us because this is. the kind of thing that we do and my. guess is that's true of. the vast majority of your audience but. that doesn't mean the. i'm standing in front of you telling you. this is the thing that people are. excited about um and that's why they. want to be there primarily why they want. to be there. so to me that's what coveted has forced. us to deal with. um even though i think we're still all. in deep denial about it. and hoping that it'll go back to that. and i think about 85.

of it will we'll be able to pretend that. that's really the way it is again and. we'll forget the lessons of this. but technically what will come out of it. or technologically will come out of it. is. a way of providing a more dispersed. experience through online education and. these kinds of. remote things that we've learned and. we'll have to come up with new ways to. engage. them in the experience of college which. includes not just the parties or. whatever kids do but the learning part. of it so that they actually come out for. five or six years later. with having actually having actually. learned something so.

um i think the world will be radically. different afterwards and i think. technology. will matter for that just not in the way. that the people who are building the. technology originally. imagined it would be and i think this. would have been true. even without covid but covet has. accelerated that. reality so it's happening in two or. three years or five years as opposed to. 10 or 15. that was an amazing answer that i did. not understand. so it was passionate and and i but i. don't know i just didn't.

no i'm not trying to criticize it i. think i'm i don't think i'm getting it. so. you mentioned disaggregation so what's. that well so. you know the power the power of. technology that if you go on the west. coast and hang out long enough is all. about we're going to disaggregate these. things together the books from the. bookstore you know that kind of a thing. and then suddenly amazon controls the. universe right and technology is a. disrupter. right and people have been predicting. that for uh higher education for a long. time but certainly. so is this is this the sort of idea like. students can aggregate on a campus. someplace and then take classes.

over the network anywhere yeah this is. what people thought was going to happen. or at least people claimed it was going. to happen right that you know because my. daughter is essentially doing that now. she's on one campus but learning in a. different campus sure and kobe makes. that possible. right um okovi makes that um league all. but avoidable. right but the idea originally was that. you know you and i were going to create. this machine learning class and it was. going to be great and then no one else. would. be the machine learning class everyone. takes right that was never going to. happen but. you know something like that but i feel. like you didn't address that so why why. why is it that why cue why i don't think.

that will be the thing that happens the. college experience maybe i maybe i. missed what the college experience was i. thought it was. peers like people hanging around a large. part of it is peers. well it's peers and independence yeah. but none of that. you can do classes online for all of. that no no no no. because no definitely we're social. people right so you want to be able to. take the classes that also has to be. part of an experience. it's in a context in the context of the. university and by the way it actually. matters. that georgia tech really is different.

from brown. i see because then students can choose. the kind of experience they think is. going to be best for them. okay i think we're giving too much. agency to the students in making an. informed decision. okay but the truth but yes they will. make choices and they will have. different experiences and some of those. choices will be made for them. some of them will be choices they're. making because they think it's this that. or the other i just don't want to say. i don't want to give the idea not. homogenous yes it's certainly not. homogeneous. right i mean georgia tech is different. from brown brown is different.

from pick your favorite state school in. iowa. iowa state okay which i guess is my. favorite state school in iowa sure. but you know these are all different. they have different contexts and a lot. of those contexts. are they're about history yes but. they're also about the location of where. you are. uh they're about the larger group of. people who are around you whether you're. in athens georgia and you're basically. the only thing that's there. as a university you're responsible for. all the jobs or whether you're at. georgia state university which is an. urban campus. where you're surrounded by you know six.

million people uh in your campus where. it ends and begins in the city ends it. begins we we don't know. it actually matters whether you're a. small campus or a large campus. why is it that if you go to georgia tech. you're like. forever proud of that and you like say. that to people. at dinner with like bars and whatever. and if you. not you know if you get a degree in an. online university somewhere you don't. that's not a thing that comes up at a. bar. well it's funny you say that so the.

students who take our online masters. by several measures are more loyal than. the students who come on campus. certainly for the master's degree. the reason for that i think and you'd. have to ask them but based on. my conversations with them i feel. comfortable saying this is because. this didn't exist before i mean we talk. about this online masters and that it's. reaching you know 11 000 students and. that's an amazing thing and we're. admitting everyone we believe we can. succeed we got a. 60 acceptance rate it's amazing right. it's also a 6600. degree the entire degree costs 6 600 7.

000 depending on how long you take. a dollar degree as opposed to 46 000. cost you to come on campus. so that feels and i can do it while i'm. working full-time and i've got a family. and a mortgage and all these other. things. so it's an opportunity to do something. you wanted to do but you didn't think. was possible. without giving up two years of your life. as well as all the money and everything. else the life that you had built. so i think we created something that's. had an impact. but importantly we gave a set of people. opportunities they otherwise didn't feel.

they had so i think people feel very. loyal about that my biggest piece of. evidence for that besides the surveys. is that we have somewhere north of 80. students. might be 100 at this point who graduated. but come back in ta for this class for. basically minimum wage. even though they're working full-time. because they believe they believe. in sort of having that opportunity and. they want to be a part of something now. will they. will generation 3 feel this way 15 years. from now will people have that same. sense i don't know but right now. they they kind of do and so it's not the.

online it's. it's a matter of feeling as if you're a. part of something right we're all very. tribal. yeah right um and i think there's. something. very tribal about being a part of. something. like that being on campus makes that. easier going through a shared experience. makes it. easier it's harder to have that shared. experience if you're alone. looking at a computer screen we can. create ways to make that is it possible. it is that's the question is it still is. the intuition to me. and it was at the beginning when i saw.

something like. the online master's program is. that this is going to replace. universities and it won't replace. universities better but like where is it. why because it's living in a different. part of the ecosystem. right the people who are taking it are. already adults they've gone through. their undergrad experience. their i think their their goals have. shifted from when they were 17. um they have other things that that are. going right but it does do something. really important something very social. and very important right. you know this whole thing about um you.

know don't build the sidewalks just. leave the grass and the students. or the people will walk and you put the. sidewalks where they create paths kind. of things. yeah um their architects apparently. believe that's the right way to do. things. the metaphor here is that we. we created this environment we didn't. quite know how to think about. the social aspect but you know we didn't. have time to solve all. do all the social engineering right um. the students did it themselves they. created. um you know these groups like on google. plus they're like 30 something groups. created in the first year because.

somebody had these google plus um and. they created these groups. and they divided up in ways that made. sense we live in the same state or we're. working on the same things we have the. same background or whatever. and they created these social things we. sent them t-shirts. and they were we have all these great. pictures of students putting on their. t-shirts as they travel around the world. i climb to this mountaintop i'm putting. this t-shirt on. i'm a part of this they were they were. part of them they created the social. environment on top of the social network. and the social media that existed. uh to create this sense of belonging and. being a part of something they found a.

way to do it. right and i think they had other. it scratched an itch that they had but. they had scratched some of that itch. that might have required they'd be. physically in the same place. long before right so i think. yes it's possible and it's more than. possible it's necessary. but i don't think it's going to replace. the university as we know it the. university as we know it will change. but there's just a lot of power and the. kind of rite of passage kind of going.

off to yourself now maybe there'll be. some other rite of passage that will. happen. right that's the best drive or somewhere. else you can separate so the university. is such a fascinating. uh mess of things so just even the. faculty position. is a fascinating mess like it doesn't. make any sense it it stabilizes itself. but like why are the world-class. researchers. spending a huge amount of time of their. time teaching. and service like you're doing like three. jobs yeah.

and and i mean it turns. it's maybe an accident of history or. human evolution i don't know it seems. like the people who are really good at. teaching. are often really good at research there. seems to be a parallel. there but like it doesn't make any sense. that you should be doing that. at the same time it also doesn't seem to. make sense. that your place where you party. is the same place where you go to learn. calculus or whatever the. but it's a safe space safe space for. everything yeah relatively speaking it's.

a safe space now by the way i feel. the need very strongly to point out that. we are living in a very particular weird. bubble. right most people don't go to college. and by the way the ones who do go to. college. they're not 18 years old right they're. like 25 or something i forget the. numbers you know. the places where we've been where we are. uh they look like whatever we think the. traditional movie version. of universities are but for most people. it's not that way at all by the way most. people who drop out of college it's. entirely. for financial reasons right the the so.

you know we were talking about a. particular experience. um and so for that set of people which. is very small but larger than it was. a decade or two or three or four. certainly ago. i don't think that will change my. concern. which i think is kind of implicit in. some of these questions is that. somehow we will divide the world up. further uh. into the people who get to have this. experience and get to have the network. and they sort of benefit from it. and everyone else while increasingly.

requiring that they have more and more. credentials in order to get a job. as a barista right you got to have a. master's degree in order to to work at. starbucks. i mean we're going to force people to do. these things but they're not going to. get to have that experience and there'll. be a small group of people who do who. continue to you know. positive feedback et cetera et cetera i. worry a lot about that. which is why for me um and by the way. here's an answer to your question about. faculty which is why to me that you have. to focus on. access and the mission i think the. reason whether it's good bad or strong i. mean i agree it's strange. but i think it's useful to have the.

faculty member particularly large r1. universities where. we've all had experiences uh that. you tie what they get to do and with the. fundamental. mission of the university and let the. mission drive what i hear when i talk to. faculty is. they love their phd students because. they're creating they're reproducing. basically right and it lets them do. their research and multiply. but they understand that the mission is. the undergrads. and so they will do it without complaint. mostly.

because it's a part of the mission and. why they're here and they have. experiences with it themselves and that. it was important to get them get them. where they were going the people tend to. get squeezed in that by the way are the. master students. right who are neither the phds who are. like us nor the undergrads we. we have already bought into the idea. that we we have to teach though. that's increasingly changing anyway i. think tying that mission in really. matters and it gives you a way to. unify people around making it an actual. higher calling. education feels like more of a higher. calling to me than than even research.

because education you cannot treat it as. a hobby. if you're going to do it well but but. that's the that's the pushback. on this whole system is that you should. education be a full-time job right and. like it's almost like. research is a distraction from that. yes although i think most of our. colleagues many of our colleagues would. say that research is a job and education. is. the distraction right but that's the. beautiful dance it seems to be that. that tension in itself is seems to work.

seems to bring out the best in uh. in the faculty or like that but i will. point out two things. one thing i'm going to point out the. other thing i want michael to point out. because i think michael is much closer. to the. to the to sort of the the ideal. professor. in some sense than i am well you're the. platonic sense of a performance. i don't know what he meant by that but. he's he is a dean so he has a different. experience i'm giving him. i'm giving him time to think of the. profound thing he's going to that's good. but let me. let me point this out which is that we.

have lecturers in the college of. computing where i am. uh there's 10 or 12 of them depending on. your account as opposed to the. 90 or so tenure track faculty those 10. lecturers who. only teach well they don't only teach. they also do service they some of them. do research as well but. primarily they teach they teach 50. over 50 of our credit hours and we teach. everybody. right so they're doing not just. they're doing more than eight times the. work of the tenure track faculty. uh by just if more closer to nine or ten.

and that's including our grad courses. right so they're doing this they're. teaching more they're touching more. more than anyone and they're beloved for. it i mean. so we recently had a survey we do these. alumni everyone does these alumni. surveys you hire someone from the. outside to do whatever and. and i was really struck by something you. saw these really cool numbers i'm not. going to talk about it because you know. it's all internal confidential stuff but. one thing i will talk about. is there was a single question we asked. our alum and these are people who. graduated you know born in the 30s and. 40s. all the way up to people who graduated. last week right um. well that's great okay good um.

and here's the question name this a. single person who had a. strong positive impact on you something. like that i think it was. special impact yeah special impact on. you and then so they got all the answers. from people and they created a word. cloud. there was clear word cop created by. people who don't do word clouds for a. living because they had one person whose. name like appeared. nine different times like philip phil dr. phil you know but whatever. but they got all this and i looked at it.

and i noticed something really cool. the five people from the college of. computing. i recognized were in that cloud and. um four of them were lecturers. the people who teach two of them. relatively. modern both were chairs of our division. of computing instruction one just. one retired one is going to retire soon. and the other two were lecturers i. remembered. from the 1980s um two of those four. by the way the fifth person was charles. that's not important the thing is.

i i don't tell people that but the two. of those people are teaching the words. are named after. thank you michael two of those are our. teaching awards are named after right. so when you ask students alumni people. who are now 60 70 years old even. you know who touch them they say the. dean of students they say the big. teachers who taught the big introductory. classes that got me into it there's a. guy named. richard bark who's on there who's who's. you know i who's known as a great. teacher. uh the the phil adler guy who um who uh. i probably just said his last name wrong. but i know the first name is phil.

because he kept showing up over and over. again. uh famous adler is what it said okay. good but different people spelled it. differently so he appeared multiple. times right. so he was a uh clearly he was a. professor in. um the business school um but when you. read about himself i went to read a box. i was curious who he was you know it's. all about his teaching and the students. that he touched right. so whatever it is that we're doing and. we think we're doing that's important or. why we think the universities function. the people who go through it yeah they. remember the people who are kind to them. the people who.

taught them something and they do. remember it they remember it later. i think that's important that's where. the mission matters. yeah not to completely lose track of the. fundamental problem of how do we replace. uh the the party aspect of universities. uh before we go to the what makes the. platonic professor. do you uh do you think. like what in your sense is the role of. moocs in this whole picture. during covid like are we should we.

desperately be clamoring to get back on. campus. or is this a stable place to be for a. little while i don't know i know that. it's that. it's the online teaching experience and. learning experience has been. really rough i think that that people. find it to be a struggle in a way that's. not. a happy positive struggle that when you. got through it you just feel like glad. that it's over as opposed to i've. achieved something so you know i worry. about that but.

um you know i worry about just even. before this happened i worry about. lecture. teaching is how how well is that. actually really working. as far as a way to do education as a way. to. inspire people i mean all the data that. i'm aware of. seems to indicate and this kind of fits. i think with charles's story. is that people respond. to connection right they actually feel. if they're if they feel connected to the. person teaching the class. they're more likely to go along with it.

they're more they're more able to retain. information they're more motivated to be. involved in the class in some way and. and that really matters it people can. mean to the human. themselves yeah so can't you do that. actually perhaps more effectively. online like you mentioned science. communication so i. i literally i think learned linear. algebra from gilbert strang. by watching mit open courseware when i. was in drugs like. and he was a personality he was a bit.

like a you know tiny. in his tiny little world of math there's. a bit of a rock star right so you kind. of look up. to that uh to that person can't that. replace. the in-person education it can help. i will point out something i can't share. the numbers but the we have surveyed our. students. and even though they have feelings about. what i would interpret as connection. i like that word um in the different. modes of classrooms. there's no difference between how they.

how well they think they're learning. for them the thing that makes them. unhappy. is the situation they're in and i think. the last lack of connection it's not. whether they're learning anything they. seem to think they're learning something. anyway. right in fact they seem to think they're. learning it equally well. uh presumably because the faculty are. putting in. or the instructors more generally. speaking are putting in. uh the energy and effort to try to make. certain that they're. what they've curated can be expressed to. them in a useful way but the connection.

is missing and so there's huge. differences in. what they prefer and as far as i can. tell what they prefer is more connection. not less that connection just doesn't. have to be physically in a classroom i. mean look. you know i used to teach 348 students in. a machine learning class on campus you. know why. that was the biggest classroom on campus. they're sitting in a theater they're. sitting in theater seats i'm literally. on a stage looking down on them. and talking to them right there's no. i mean we're not sitting down having a. one-on-one conversation.

reading each other's body language. trying to communicate and going we're. not doing any of that. so you know if you're on the if you're. past the third row it might as well be. online anyway is the kind of thing that. people. said daphne has actually said some. version of this um that online starts on. the third row or something like that. and i think that's that's not yeah i. like it i think it captures something. important. but people still came by the way they. even the people who had access to our. material would still come to class. i mean there's a certain element about. looking to the person next to you yeah. it's just like their presence there.

their their boredom. and like when the parts are boring and. their excitement when the parts are. exciting. like and sharing in that like unspoken. kind of uh. yeah communication like in part the. connection is with the other people in. the room. watching watching the circus on tv. alone is ever been to a movie theater. and been the only one there at a comedy. it's not as funny as when you're in a. room full of people all laughing. well you need maybe you need just.

another person it's like. as opposed to many maybe maybe there's. some kind of well there's different. kinds of connection right. and there's different kinds of comedy. well in the sense as we're learning. today. i wasn't sure if that was going to land. but um just the idea that. that different jokes i i've. i've now done a little bit of stand up. and so different jokes work in different. size crowds too right where sometimes if. you know if it's a. big enough crowd then even a really.

subtle joke can. take root someplace and then that cues. other people and it kind of. there's a whole statistics of i did this. terrible thing to my brother so when i. was really young. i decided that my brother was only. laughing at sitcoms when i laughed. like he was taking cues from me so i. like purposely didn't laugh. just to see if i was right laugh at. non-funny things yes i really want to do. both sides i did both sides. and uh and at the end of it i told him. what i did. he was very upset about this yeah and.

from that day on. he's he lost his sense of humor no no no. no well yes but from that day on he. he he laughed on his own he stopped. taking cues from me so i want to say. that you know it was a good thing that i. did but yes yes you. was making that man's life yes but it. was mostly me but it's true though it's. true right that people. i i think you're right but okay so. that's where does that get us that gets. us the idea. that i mean certainly movie theaters are. a thing right where people like to be. watching together. even though the people on the screen.

can't aren't really co-present with the. people in the audience the. audience is co-present with themselves. by the way and that point it's. it's an open question that's being. raised by this whether movies will no. longer be a thing because netflix's. the audience is growing so that's it's a. it's a very parallel question for. education. will move and theaters still be a thing. right in 2000. but i think i think the argument is that. there is a feeling of being in the crowd. that isn't replicated by being at home. watching it and that there's value in. that. and then i think just but it scales.

better on. right i feel like we're having a. conversation. about whether concerts will still exist. after. the invention of the record or the cd or. wherever it is right. you're right concerts are dead um well. okay. i think the joke is only funny if you. say it before. now right yeah like three years ago it's. like well no obviously. i'll wait to publish this until we have. a vaccine. you know we'll fix it in post but i.

think the the important thing is. the virus bust concerts changed right. first of all movie theaters weren't this. way right in like the 60s and 70s they. weren't like this like blockbusters. were basically what jaws and star wars. created blockbusters right. before then there weren't like the whole. summer shared summer experience didn't. exist. in our lifetimes right certainly you. were well into adulthood by the time. this was true right. so it's just a very different it was. it's very different so what the what. we're we've been experiencing in the. last 10 years is not like.

the majority of human history but more. importantly concerts right. concerts mean something different most. people don't go to concerts. anymore like there's an age where you. care about it you sort of stop doing it. you keep listening to music or whatever. so i think that's a. painful way of saying that um. it will change it's not the same things. are going away replace is too strong of. a word. but it will change it has to i actually. like to push back. i wonder because i think you're probably.

just throwing that your intuition out. oh i won't wait and turn it's possible. that concerts. more people go to concerts now but. obviously much more people listen to. well just dumb. when they then before there was records. uh. it's it's possible to argue that uh if. you look at the data. uh that it just expanded the pie of what. music listening means so it's possible. that like universities grow. in the parallel where the theaters grow.

but also more people get to watch movies. more people get to. like be educated yes i i hope that yeah. and to this extent that we can grow the. pie. and have education be not just something. you do for four years when you're done. with your other education but it'd be a. more lifelong. thing that would have tremendous. benefits especially as the. the economy and the world change rapidly. like people need opportunities to stay. abreast of these changes and so.

i don't know i could i could it's all. part of the the ecosystem it's all to. the good i mean. you know i'm not gonna have an argument. about whether we. we lost fidelity when we went from. laserdisc to dvds or. record players to cds i mean i'm i'm. willing to grant that that is true. but convenience matters. and the ability to do something that you. couldn't do otherwise because that. convenience. matters and you can tell me i'm only. getting 90 of the experience. but i'm getting the experience i wasn't. getting it before or wasn't lasting as. long or it wasn't.

i mean this just seems this just seems. straightforward to me it's gonna. it's going to change it is for the good. that more people get access. and it is our job to do two separate. things one to educate them. and make access available that's our. mission but also for. very simple selfish reasons we need to. figure out how to do it better so that. we individually stay in business. we can do both of those things at the. same time they are not in. they may be intentioned but they are not. mutually exclusive. so you've educated some scary number of.

people. so you've seen a lot of people succeed. find their. path to life is there advice that you. can give. to a young person today. about computer science education. about education in general about life. about uh whatever the. journey that one takes in there maybe in. their teens in their early twenties.

instead of in those underground years as. you try to. go through the essential process of. partying and not going to classes. and yet somehow trying to get a degree. if you get to the point. where you're you're you're far enough up. in the in the. hierarchy of needs that you can actually. make decisions like this. then find the thing that you're. passionate about and pursue it. and sometimes it's the thing that drives. your life and sometimes it's secondary. and you'll do other things because.

you've got to eat right. you got a family you got to feed you've. got people you have to help or whatever. i and i understand that and it's not. easy for everyone but. um always take a moment or two to. pursue the things that you love the. things that. bring passion and happiness to your life. and if you don't i know that sounds. corny but i genuinely believe it and. if you don't have such a thing then. you're lying to yourself you have such a. thing you just. have to find it and it's okay if it. takes you a long time to get there. rodney dangerfield became a comedian in. his 50s. i think certainly wasn't his 20s and.

lots of people failed for a very long. time before getting to. where they were going um you know i try. to have. hope and it it wasn't obvious i mean you. know we. you and i talked about the experience. that i had. um a long time ago with with a. particular police officer was it my. first one. it wasn't my last one um but you know in. my view. i wasn't supposed to be here after that. and i'm here so it's all gravy. so you might as well go ahead and grab. life as you can because of that that's. that's sort of how i see it.

while recognizing again the delusion. matters right allow yourself to be. deluded. allow yourself to believe that it's all. going to work out just don't be so. deluded that you. you miss the obvious and and you're. going to be fine. it's going to be there it's going to be. there it's going to work out. what do you think i like to say choose. your parents wisely. because that has a big impact on your. life. yeah i mean you know i mean there's a. whole lot of things that you don't get. to pick. um and and and whether you get to have.

you know one kind of life or a different. kind of life can depend a lot on things. out of your control. but i really do believe in the in the. passion excitement thing. my i was talking to my mom on the phone. the other day and. essentially what came out is that. computer science. is really popular right now and and and. i get to be a professor teaching. something that's very. uh attractive to people and she's she. was like.

trying to give me some appreciation for. how. forsightful i was for choosing this line. of work as if somehow i knew that this. is what was going to happen. in 2020 uh but that's not how it went. for me at all like i studied computer. science because i was. just interested it was just so. interesting to me i. didn't i didn't think it would be. particularly lucrative. yeah and i've done everything i can to. keep to keep it as unlucrative as. possible. yeah um some of my you know some of my.

friends and colleagues have. have have not done that and i pride. myself on my ability to. just to remain unrich but um. but but i think but but i do believe. that that. like i'm glad i mean i'm glad that it. worked out for me it could have been. like. oh what i was really fascinated by is. this particular kind of engraving that. nobody cares about. but um so i got lucky and the thing that. i cared about happened to be a thing. that other people. eventually cared about but i don't think. i would have had a fun time. choosing anything else like this was the. thing that kept me.

interested and engaged well one thing. that people. tell me especially around as an early. undergraduate. and the internet is part of the problem. here. is they say they're passionate about so. many things. how do i choose a thing which is a. harder thing. for me to know what to do with is there. any i mean. don't you know i mean you know. a long time ago i walked down the. hallway and i took a left turn. yeah i could have taken a right turn and.

my world. could be better or it could be worse i. have no idea i have no way of knowing is. there anything about this particular. hallway that's relevant or you're just. in general choices yeah you were on the. left it sounds like you regret. not taking the right oh no not at all. you brought it up well. because there was a turn there on the. left was michael lemon's office right. i mean these sorts of things happen. right yes but here's the right by the. way it was just a blank wall. it wasn't a huge choice it would have. really hurt he tried first. no but it's it's true right that you. know i i think about ron brockman right. i.

i went i took a trip i wasn't supposed. to take. and i ended up talking to to um. on about this and i ended up going down. this entire path uh that allowed me to i. think get tenure. but by the way i decided to say yes to. something that didn't make any sense and. i went down this educational path. but it would have been you know who. knows right maybe if i hadn't done that. i would be a billionaire right now i'd. be elon musk my life could be so much. better. my life could also be so much worse you. know you just got to feel that sometimes.

you have decisions you're going to make. you cannot know what's going to you. should think about it right some things. are clearly smarter than other things. you've got to play the odds a little bit. but in the end if you've got multiple. choices or lots of things you think you. might love go with the thing that you. actually. love the thing that jumps out at you and. sort of pursue it for a little while the. worst thing that will happen is you took. a left turn instead of a right turn. and you ended up merely happy beautiful. so so accepting so taking the step and. just accepting. accepting that that don't like question.

questions. life is long and there's time. to actually pursue every once in a while. uh you have to put on a leather suit. and make a thriller video every once in. a while. yeah uh i was told that you actually. danced but that part was edited out i. don't dance. there was a thing where we did do the uh. yeah the zombie thing yeah we did do the. zombies yeah.

but that wasn't edited out it just. wasn't able to put into the final thing. i'm quite happy but there was a reason. for that too right like i wasn't wearing. something right there was a reason for. that i can't remember what it was no i. love this suit. is that what it was i can't remember. anyway the right thing happened. exactly you took the left turn and then. the third of the right ended up being. the right thing. so a lot of people ask me that are a. little bit. tangential to the programming the. computing world. and they're interested to learn. programming like all kinds of. disciplines that are outside.

of the particular discipline of computer. science what advice. do you have for people that want to. learn how to program or want to. either taste this little. skill set or discipline or try to see if. it can be used somehow in their own life. what stage of life are they in. uh it feels one of the magic things. about the internet. of the people that write me is i don't. know. because my answer is different for for.

my daughter is taking ap computer. science right now. hi johnny um she's uh she's amazing and. doing amazing things and. my son's beginning to get interested and. i'll be really curious where he takes it. i think he's. his mind actually works very well for. this sort of thing and she's doing great. but one of the things i have to tell her. all the time she points well i want to. make a rhythm game. so i want to go for two weeks and then. build a rhythm game show me how to build. a rhythm game. and start small learn the building. blocks and hours take the time. have patience eventually you'll build a.

rhythm game i was in grad school. when i suddenly woke up one day over the. royal east. um and i thought wait a minute i'm a. computer scientist i should be able to. write pac-man in an afternoon and i did. not with great graphics it was actually. a very cool game i had to figure out how. the ghost moved and everything and i did. it in an afternoon in pascal. on an old apple 2gs um but if i had. started out trying to build pac-man. i think it probably would have ended. very poorly for me luckily back then. there weren't you know these magical. devices we call phones and. software everywhere to give me this.

illusion that i could create something. by myself from the basics inside of a. weekend like that. i mean that was a culmination of years. and years and years right before i. decided i should be able to write this. and i could. so you know my advice if you're early on. is. you know you've got the internet there. are lots of people there to give you the. information. find someone who cares about this. remember they've been doing it for a. very long time. take it slow learn the little pieces get. excited about it and then. keep the big projects you want to build. in mind you'll get there soon enough. because.

as a wise man once said life is long. sometimes it doesn't seem that long but. it is long. and you'll have enough time to to build. it all out. it all the information is out there but. start small. you know generate different object. numbers that's not exciting but. it'll program well there's only one. programming language it's lisp. but if you have to pick a programming. language i guess. in today's what would i do i guess i do. python is basically doing this but with. better syntax blasphemy yeah.

see with c syntax how about that so. you're going to argue that c syntax is. better than. anything anyway also i'll go i'm going. to answer python despite. tell me tell your story about the. somebody's dissertation that had a. lisp program in it it was so funny this. is a this is dave's dave's listening to. him he was like dave mcallister who was. a professor at mit for a while and. and then he came in our in our girl labs. now he's at um now he's a technology. technical institute of chicago uh. a brilliant guy uh such an interesting. guy anyway his thesis.

uh it was a theorem proverb. and he decided to have as an appendix. uh his actual code which of course was. in list because of course it was. it's like the last 20 pages are just. right parentheses. it's just wonderful it's like they. that's programming right there just like. pages of. pages of right parentheses anyway lisp. is the only real language but i. understand that that's not necessarily. the place where you start. python is just fine nah python is good. if you're you know of a certain age if.

you're really young and trying to figure. it out. graphical languages that let you kind of. see how the thing works and that's fine. too they're all fine it almost doesn't. matter but there are people who spend. a lot of time thinking about how to. build. languages that get people in the. questions are you trying to get in. and figure out what it is or do you. already know what you want. and that's why i asked you what stage of. life people are in because if you're. different stages of life you. you would you would attack it. differently the the answer to that. question of which language keeps. changing i mean there's some. value to exploring uh a lot of people.

write to me about julia there's there's. these like more modern languages they. keep being invented rust and. and kotlin and there's stuff that uh. for people who love functional languages. like lisp. there apparently there's echoes of that. but much better. in the modern languages and it's. worthwhile to uh. especially when you're learning. languages it feels like it's okay to try. one that's not. like the popular one oh yeah but you. know you i think you get that. you get that way of thinking almost no.

matter what language. and if you if you push far enough like. it can be assembly language. but you need to push pretty far before. you start to hit the really deep. concepts that you would get sooner in. other languages. but like i don't know computation is. kind of computation. it's kind of touring equivalent is kind. of computation and so it's so. it matters how you express things but. you have to build out that mental. structure. in your mind and you i don't i don't. think it super matters which language. i mean it matters a little because some.

things are just at the wrong level of. abstraction i think assembly's at the. wrong level abstraction for someone. coming in new. um i think that if you start someone. coming in new. yes for frameworks big frameworks or or. quite a bit um you got to get to the. point where i want to learn any language. means i just pick up a reference book. and. i think of a project and i go through it. in the weekend right you got it you got. to get there you're right though. the languages that are designed for that. are it almost doesn't matter pick the. ones that people have. built tutorials and infrastructure. around to help you get kind of. kind of eased into it because it's hard.

i mean we i did this little experiment. once. um i was teaching intro to cs in the. summer as a favor. uh which is anyway i was using. memories i was teaching introduces a. favor it was very funny because i'd go. in every single time and i would think. to myself. how am i possibly going to fill up an. hour and a half talking about for loops. right. and there wasn't enough time it took me. a while to realize this right. there are only three things right. there's reading from a variable writing. to a variable in conditional branching.

everything else is syntactic sugar right. the syntactic sugar matters but that's. it. and when i say that's it i don't mean. it's simple i mean it's hard. like conditional branching loops. variable. those are really hard concepts so you. shouldn't be discouraged by this here's. a simple experiment. i'm going to ask you a question now you. ready x equals three. okay y equals four. okay what is x three what is y. four y equals gonna mess this up no.

that's oh it's easier. y equals x y equals x what is y. uh three that's right x equals 7. what is y that's one of the trickiest. things to get. for programmers that there's a memory. and the variables. are pointing to a particular thing in. memory and sometimes the languages hide. that from you and they. bring it closer to the way you think. mathematics works right so in fact. mark gosdale who worries about these. sorts of things or used to worry about.

these sorts of things anyway. had this kind of belief that actually. people when they see these statements x. equals something y equals something y. equals x. that you have now uh made a mathematical. statement that y and x are the same. which you can if you just put like an. anchor in front of it yes but. people that's not what you're doing yeah. right i thought. and i kind of asked the question and i i. think had some evidence for this. hardly a study is that most of the. people who didn't know the answer. weren't sure about the answer they had.

used spreadsheets. ah and so it's a it's a na it's it's. it's you know um it's by it's by. reference. or by name really right and so depending. upon what you think. they are you get completely different. answers the fact that i could go. or one could go two thirds of the way. through a semester. and people still hadn't figured out in. their heads when you say y equals x what. that meant. tells you it's actually hard because all. those answers are possible in fact. when you said oh if you just put an.

ampersand in front of it i mean that. doesn't make any sense for an intro. class and of course. a lot of language don't even give you. the ability to think about it in terms. of ampersand. do we want to have a 45-minute. discussion about the difference between. equal eq. and equal in lisp yeah i know you do. [Laughter]. but you know you could do that it's this. is actually really hard stuff. so you shouldn't be it's not too hard we. all do it but you shouldn't be. discouraged it's why you should start. small. so that you can figure out these things. you have the right model in your head. so that when you write the language you.

can execute it and build the machine. that you you want to build right yeah. the funny thing about programming. on those very basic things is the the. the very basics are not often made. explicit. which is actually what drives everybody. away from. basically any discipline but program is. just another one like even a simpler. version of the equal sign that i kind of. forget. is in mathematics equals is not. assignment. yeah right like i think basically every. single programming language with just a.

few. handful of exceptions equals his. assignment. and you have some other operator for uh. equality. yeah and you know even that like. everyone kind of knows it. once you started doing it but like you. need to say that explicitly or you just. realize it like yourself. otherwise you'll be you might be stuck. for you said like half a semester you. could be stuck for quite a long time. and i think also part of the programming.

is being okay in that state of confusion. for a while it's it's to the debugging. point it's like. i just wrote two lines of code why. doesn't this work and staring at that. for like. hours and trying to figure out. and then every once in a while you just. have to restart your computer and. everything works again. and then and then you just kind of stare. into the. void with the tear slowly rolling down. your eye by the way the fact that they. didn't get this. actually had no impact on i mean they.

were still able to redo their. assignments right. because it turns out their. misunderstanding wasn't being revealed. to them. yes by the problem sets we were. found actually yeah i wrote a um a. program a long time ago. actually for my master's thesis and uh. in c. plus i think or c i guess we'll see and. it was uh all memory management and. terrible. um and it wouldn't work for a while. and it was some kind of it was clear to. me that it was overwriting memory. and i just couldn't i was like look i.

got a paper. time for this so i basically declared a. variable. at the front in the main that was like. 400k. just an array and it worked because. wherever i was scribbling over memory. it would scribble into that space and it. didn't matter and so i never figured out. what the bug was. but i did create something to sort of. deal with it to work around it. and it you know that's crazy that's. crazy it was okay because that's what i. wanted. but i knew enough about memory managed. to go you know management to go you know.

i'm just going to create an empty array. here and hope that that deals with the. scribbling memory problem. and it did that takes a long time to. figure out and by the way the language. you first learned probably this garbage. collection anyway so you're not even. going to come up across it you're going. to come across that problem. so we talked about the the minsky idea. of hating everything you do. and hating yourself so let's. end on a question that's gonna make both. of you very uncomfortable. okay which is what is your charles. what's your favorite.

thing that you're grateful for about. michael. and michael what is your favorite thing. that you're grateful for about. charles well that answer is actually. quite easy. his friendship he's still the easy. i did yeah i'll tell you what i hate. about charles that steals my good. answers. the thing i like most about charles he. sees the world in it in a. similar enough but different way that i. it's sort of like having. another life it's sort of like i get to. experience things that i wouldn't.

otherwise. get to experience because i would not. naturally gravitate to them that way. and so he just he just shows me a whole. other world it's awesome. yeah the the inner product is not zero. for sure it's not quite one point seven. maybe. just enough that you can learn. just enough that you can learn that's. the definition of friendship the inner. product is 0.7 yeah. i think so that's the answer to life. really charles sometimes believes in me. when i have not.

believed in me he can he also sometimes. works as an outward confidence. that he has so much so much confidence. and. self i don't know aware comfortableness. okay let's go with that um that i feel. better. a little bit if he if he thinks i'm okay. then maybe i'm not as bad as i think i. am. at the end of the day luck favors the. charles. it's a huge honor to talk with you thank. you so much for taking this time. wasting your time with me it was an.

awesome conversation you guys are an. inspiration to. a huge number of people and to me so. really enjoyed this thanksgiving i. enjoyed it as well thank you so much. and by the way if luck favors the. charleston is certainly the case that. i've been very lucky to know. i'm going to edit that part out. thanks for listening to this. conversation with charles isabel and. michael littman. and thank you to our sponsors athletic. greens. super nutritional drink ate sleep. self-cooling mattress master class.

online. courses from some of the most amazing. humans in history. and cash app the app i use to send money. to friends. please check out the sponsors in the. description to get a discount. and to support this podcast if you enjoy. this thing. subscribe on youtube review it with five. stars napa podcast. follow on spotify support it on patreon. connect with me on twitter. at lex friedman and now let me leave you. some words from desmond tutu.

don't raise your voice improve your. argument. thank you for listening and hope to see. you next time.