Jim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162

the following is a conversation with jim. keller his second time in the podcast. jim is a legendary microprocessor. architect. and is widely seen as one of the. greatest engineering minds. of the computing age in a peculiar twist. of space-time in our simulation jim is. also. a brother-in-law of jordan peterson we. talk about this. and about computing artificial. intelligence. consciousness and life quick mention of.

our sponsors. athletic greens all-in-one nutrition. drink brooklyn and sheetz. expressvpn and belcampo grass-fed. meat click the sponsor links to get a. discount and to support this podcast as. a side note let me say that jim is. someone who on a personal level. inspired me to be myself there was. something in his words. on and off the mic or perhaps that he. even paid attention to me at all. that almost told me you're all right kid. a kind of pat on the back that can.

make the difference between a mind that. flourishes and a mind that is broken. down. by the cynicism of the world so i guess. that's just my brief few words of thank. you to jim and in general. gratitude for the people who have given. me a chance on this podcast. in my work and in life if you enjoy this. thing. subscribe on youtube review on apple. podcast follow on spotify. support on patreon or connect with me on. twitter alex friedman. and now here's my conversation with jim.

keller what's the value and. effectiveness of theory versus. engineering. this dichotomy in uh building good. software. or hardware systems well it's. good designs both i guess that's pretty. obvious. but engineering do you mean you know. reduction of practice of known methods. and then science is the pursuit of. discovering things that people don't. understand. or solving unknown problems definitions.

are interesting here but. i was thinking more in theory. constructing models. that kind of generalize about how things. work and engineering. is like actually building stuff the. pragmatic like. okay we have these nice models but how. do we actually get things to work maybe. economics is a nice example like. economists have all these models of how. the economy works and. how different policies will have an. effect but then there's the actual. okay let's call it engineering of like.

actually deploying the policies. so computer design is almost all. engineering. and reduction of practice is known. message now. because of the complexity of the. computers we built. you know you could think you're well. we'll just go write some code. and then we'll verify it and we'll put. it together and then you find out that. the combination of all that stuff is. complicated. and then you have to be inventive to. figure out how to do it. right so that's that's definitely. happens a lot. and then every so often some big idea.

happens. but it might be one person and that idea. is in what. in the space of engineering or is it in. the space well i'll give you an example. so one of the limits of computer. performance is branch predictions. so and there's there's a whole bunch of. ideas about how good you could predict. the branch. and people said there's a limit to it. it's an asthmatic curve. and somebody came up with a better way. to do branch prediction it was a lot. better. and he published a paper on it and every.

computer in the world now uses it. and it was one idea so the the engineers. who build branch prediction hardware. were happy to drop the one kind of. training array and put it in another one. so it was it was a real idea and branch. prediction is. is one of the key problems underlying. all of sort of the lowest level of. software it boils down to branch. prediction. boils down to uncertainty computers are. limited by you know single thread. computers limited by two things. the predictability of the path of the.

branches and the predictability of the. locality. of data so we have predictors that now. predict both of those pretty well. yeah so memory is you know a couple. hundred cycles away local cache is. couple cycles away. when you're executing fast virtually all. the data has to be in the local cache. so a simple program says you know add. one to every element in array. it's really easy to see what the stream. of data will be. but you might have a more complicated. program that you know says get a get an.

element of this array look at something. make a decision. go get another element it's kind of. random and you can think that's really. unpredictable. and then you make this big predictor. that looks at this kind of pattern and. you realize. well if you get this data in this data. then you probably want that one. and if you get this one and this one and. this one you probably want that one. and is that theory or is that. engineering like the paper that was. written. was it uh well it was asymptotic kind of. kind of discussion or is it more like. here's a hack that works well. um it's a little bit of both like. there's information theory in it i think.

somewhere. okay so that's actually trying to prove. yeah but once once you know the method. implementing it is an engineering. problem. now there's a flip side of this which is. in a big design team. what percentage of people think their. their. their uh their their plan or their. life's work is engineering versus design. inventing things so lots of companies. will reward you for. filing patents yes some many big. companies get stuck because to get. promoted you have to come up with.

something new. and then what happens is everybody's. trying to do some random new thing. 99 which doesn't matter and the basics. get. neglected and or. they get to there's a dichotomy they. think like the cell library and the. basic cad tools. you know or basic you know software. validation methods. that's simple stuff you know they want. to work on the exciting stuff. and then they they spend lots of time. trying to figure out how to patent. something.

and that's mostly useless but the. breakthroughs are on the simple stuff. no no you no you have to do the simple. stuff really well. if you're brilliant building a building. out of bricks you want great bricks. so you go to two places to sell bricks. so one guy says yeah they're over there. in an. ugly pile and the other guy is like. lovingly tells you about the 50 kinds of. bricks and how hard they are and how. beautiful they are and. how square they are and you know which. one you can buy bricks from. which is going to make a better house so.

you're talking about the craftsman the. person who understands bricks who loves. bricks who loves their varieties. that's a good word you know good. engineering is great craftsmanship. and when you start thinking engineering. is about invention. and set up a system that rewards. invention. the craftsmanship gets neglected okay so. maybe one perspective is the theory the. science. over emphasizes invention and. engineering emphasizes craftsmanship and.

therefore. like so if you it doesn't matter what. you do in theory well everybody knows. like read the tech rags they're always. talking about some breakthrough or. intervention. innovation and everybody thinks that's. the most important thing. but the number of innovative ideas is. actually relatively low we need them. right and innovation creates a whole new. opportunity like when. when some guy invented the internet. right. like that was a big thing the million. people that wrote software against that. were mostly doing. engineering software writing so the.

elaboration of that idea was huge. i don't know if you know brendan ike he. wrote javascript in 10 days. and that's an interesting story it makes. me wonder. and it was you know famously for many. years considered to be. a pretty crappy programming language. still is perhaps it's been improving. sort of consistently. but the interesting thing about that guy. is. you know he doesn't get any awards you. don't get a nobel prize or a field medal.

or. uh a crappy piece of. you know that software code that is. currently the number one programming. language in the world that runs. now is cons increasingly running the. backhand of the internet what does he. end up does he know why everybody uses. it. like that would be an interesting thing. was it the right thing at the right time. because like when stuff like javascript. came out like there's a move from you. know writing c programs and c. plus plus to let's call what they call.

managed code frameworks. where you write simple code it might be. interpreted it has. lots of libraries productivity is high. and you don't have to be an expert. so you know java was supposed to solve. all the world's problems it. was complicated javascript came out you. know after a bunch of other scripting. languages. i'm not an expert on it but yeah but was. it the right thing at the right time. or was there something you know clever. because he wasn't the only one. there's a few elements maybe if he. figured out what it was. no then he'd get a prize like that.

destructive theory. yeah you know babies probably he hasn't. defined this. or he this needs a good promoter well. i think there was a bunch of blog posts. written about it which is like. wrong is right which is like doing the. crappy thing fast. just like hacking together the thing. that answers some of the needs. and then iterating over time listening. to developers like listening to people. who actually use the thing. this is something you can do more in. software.

but the right time like you have to. sense you have to have a good. instinct of when is the right time for. the right tool and make it super. simple and just get it. out there the problem is this is true. with hardware this is less true with. software is. this backward compatibility that just. drags behind you as. you know as you try to fix all the. mistakes of the past but the the timing. was good there's something about that it. wasn't accidental. you have to like give yourself over to.

the. you have to have this like broad sense. of what's needed now. and both scientifically and like the. community. and just like this it was obvious that. uh. there was no the interesting thing about. javascript is. everything that ran in the browser at. the time like java. and and i think other like scheme other. programming languages. they were all in a separate external. container.

and then javascript was literally just. injected into the web page it was the. dumbest possible thing. running in the same thread as everything. else and like. uh it was inserted as a comment so. javascript code is inserted as a comment. in the html code and it was i mean it's. there's it's either genius or super dumb. but it's like. right so it has no apparatus for like a. virtual machine and container. it just executed in the framework the. program is already running. and it was that's cool and then uh.

because something about that. accessibility. the ease of its use uh. resulted in then developers innovating. on how to actually use it. i mean i don't even know what to make of. that but. it does seem to echo across different. software. like stories of different software php. has the same story really crappy. language. they just took over the world i always. have a joke that. the random length instructions that. variable length instructions that's. always one.

even though they're obviously worse like. nobody knows why. x86 is arguably the worst architecture. you know on the planet is one of the. most popular ones well i mean isn't. isn't that also the story of risk versus. i mean is that simplicity there's. something about simplicity that. uh us in this evolutionary process. is valued if it's simple it's uh. gets it spreads faster it seems like. yeah or is that not always true. that's always true yeah it could be.

simple is good but too simple is bad. so why did risk win you think so far did. risk win. in the long arc of history maybe we. don't know so who who's going to win. what what's risk what's cisco who's. going to win in that space. in these instruction sets a ice offers. going to win but. they'll be little computers that run. little programs like normal. all over the place but but. we're we're going through another. transformation so but. you think instruction sets underneath it.

all will change. yeah they evolve slowly they don't. matter very much. they don't matter very much okay i mean. the limits of performance are. you know predictability of instructions. and data i mean that's the big thing. and then the usability of it is some you. know. quality of design quality of tools. availability. like right now x86 is proprietary with. intel and amd but they can change it any. way they want independently. right arm is proprietary to arm and they.

won't let anybody else change it. so it's like a sole point and risk 5 is. open source. so anybody can change it which is super. cool. but that also might mean it gets changed. in too many random ways that there's no. common. sub subset of it that people can use do. you like open. or do you like clothes like if you were. to bet all your money on one or the. other risk five versus. no idea it's case dependent well x86. oddly enough when intel first started. developing it they licensed. like seven people so it was the open.

architecture. and then they move faster than others. and also bought one or two of them. but there were seven different people. making x86 because at the time there was. 6502 and z80s and you know 8086. and you could argue everybody thought. z80 was the better instruction set. but that was propriety proprietary to. one place. oh and the 6800 so there's like five or. four or five different microprocessors. intel went open got the market share.

because people felt like they had. multiple sources from it and then over. time it narrowed down the two players. so why you as a historian uh well. why did intel win for so long. with the with their processors i mean. they were great their process. development was great. oh so it's just looking back to. javascript and brand nike is uh. microsoft and netscape and all these uh. internet browsers. microsoft won the browser game because.

they aggressively stole. other people's ideas like right after. they did it. you know i i don't know if intel was. stealing other people's ideas. they started making a just good way they. started making. rams random access memories. and then at the time when the japanese. manufacturers came up. you know they were getting out competed. on that and they pivoted the. microprocessors and they made the first. you know integrated microprocessor. programs uh 4004 or something.

who was behind that pivot that's a hell. of a pivot andy grove. and he was great that's a hell of a. pivot. and then they led semiconductor industry. like they were just a little company ibm. all kinds of big companies had. both loads of money and they out. innovated everybody. auto innovated okay yeah yeah so it's. not like marketing it's not yeah. their processor designs were pretty good. um. i think the you know core 2.

was probably the first one i thought was. great. it was a really fast processor and then. haswell was great. what makes a great processor in that. delay oh if you just look at it's. performance versus everybody else. it's you know the size of it the you. know usability of it. so it's not specific some kind of. element that makes it beautiful it's. just like literally just raw. performance is that how you think about. processors it's just like. raw performance of course.

it's like a horse race the fastest one. wins. now you don't care how. [Laughter]. just as long as it was well there's the. fastest in the environment. like right you know for years you made. the fastest one you could and then. people started to have power limits. so then you made the fastest at the. right power point and then. and then when we started doing. multiprocessors like. if you could scale your processors more. than the other guy you could be 10. faster on like a single thread but you. have more threads. so there's lots of variability and then.

arm. really explored like you know they have. the a series and the r. series and the m series like a family of. processors for all these different. design points from like unbelievably. small and simple. and so then when you're doing the design. it's sort of like this big palette of. cpus. like they're the only ones with a. credible you know top to bottom palette. and what do you mean incredible. top to bottom well there's people who. make microcontrollers that are small but. they don't have a fast one there's.

people make fast processors but don't. have a little a medium one or a small. one is it hard to do that full palette. that's. that seems like a it's a lot of. different so what's the difference in. uh the arm folks and intel in terms of. the way they're approaching this problem. well intel almost all their processor. designs were. you know very custom high end you know. for the last 15. 20 years the fastest force possible yeah. in one horseshoe. yeah and then architecture they're. really good but the company itself was.

fairly insular to what's going on in the. industry with cad tools and stuff. and there's this debate about custom. design versus synthesis. and how do you approach that i'd say. intel was slow on the. getting to synthesize processors arm. came in from the bottom and they. generated ip which went to all kinds of. customers. so they had very little say how the. customer implemented their. ip so arm is super friendly to the. synthesis ip environment whereas intel. said we're going to make this great.

client chip server chip with our own cad. tools with our own process with our own. you know other supporting ip and. everything only works with our stuff. so is that is arm winning the mobile. platform space in terms of processors. and so. in that and what you're describing is. why they're winning well they. had lots of people doing lots of. different experiments so they controlled. the processor architecture and ip. but they let people put in lots of.

different chips and there was a lot of. variability in what happened there. whereas intel when they made their. mobile their foray into mobile they had. one team doing one. part right so it wasn't 10 experiments. and then their mindset was pc mindset. microsoft software mindset and that. brought a whole bunch of things along. that. the mobile world embedded world don't do. do you think it was possible for intel. to pivot hard and win the mobile. market that's a hell of a difficult. thing to do right for a huge company to.

just pivot. i mean it's so interesting to because. we'll talk about your current work. it's like it's clear that pcs were. dominating for several decades. like desktop computers and then mobile. it's unclear it's a leadership question. like like apple under steve jobs when he. came back they pivoted multiple. times you know they built ipads and. itunes and phones and tablets and. great macs like like who knew computers.

should be made out of aluminum. nobody knew that that they're great it's. super fun that was steve yeah steve jobs. like they pivoted multiple times. and uh you know the old intel they they. did that multiple times. they made drams and processors and. processes and. i got to ask this what was it like. working with steve jobs i didn't work. with him. did you interact with him twice i said. hi to him twice in the cafeteria. what did he say hi he said hey fellas.

he was friendly he was wandering around. and. with somebody he couldn't find the table. because the cafeteria was. was packed and i gave my table but i. worked for mike colbert who talked to. like mike was the unofficial cto of. apple. and a brilliant guy and he worked for. steve for 25 years maybe more. and he talked to steve multiple times a. day. and he was one of the people who could. put up with steve's let's say brilliance.

and intensity. and and steve really liked him and steve. trusted. mike to translate the he thought up. into engineering products at work and. then mike ran a group called platform. architecture and i was in that group. so many times i'd be sitting with mike. and the phone would ring. it'd be steve and mike would hold the. phone like this because steve would be. yelling about something or other. yeah and he would translate it and he. translated and then he would say. steve wants us to do this so. was steve a good engineer or no i don't.

know he was a great idea guy. idea person he's a really good selector. for talent. yeah so that seems to be one of the key. elements of leadership right. and then he was a really good first. principals guy like like somebody say. something couldn't be done and he would. just think. that's obviously wrong right. but you know maybe it's hard to do maybe. it's expensive to do maybe we need. different people. you know there's like a whole bunch of. you know if you want to do something. hard. you know maybe it takes time maybe you.

have to iterate there's a whole bunch of. things. you could think about but saying it. can't be done is stupid. how would you compare so it seems like. elon musk is more. engineering centric but it's also i. think he considers himself a designer. too he has a design mind. steve jobs feels like he is much more. idea space design space versus. engineering yeah just make it happen. like the world should be this way just. figure it out but. but he used computers you know he had. computer people talk to him all the time.

like mike was a really good computer guy. he knew what computers could do. computer meaning computer hardware like. hardware software all of pieces the. whole thing. and then he would you know have an idea. about what could we do with this next. that was grounded in reality it wasn't. like he was you know just. finger painting on the wall and wishing. somebody would interpret it. like so he had this interesting. connection because. no he wasn't a computer architect or a. designer. but he had an intuition from the. computers we had to what could happen.

and essentially you say intuition. because. it seems like he was pissing off a lot. of engineers in. his intuition about what canada can't be. done those. like the what is all these stories about. like floppy disks and all that kind of. stuff like. yeah so in in steve the first round like. he'd go into a lab. and look at what's going on and hate it. and and. uh fire people or or ask somebody in the.

elevator what they're doing for apple. and you know not be happy when he came. back my impression was. is he surrounded himself with this. relatively small group of people. yes and didn't really interact outside. of that as much. and then the joke was you'd see like. somebody moving a prototype through the. quad with a. with a black blanket over it and that. was because it was secret. you know partly from steve because they. didn't want steve to see it until it was. ready. yeah the dynamic with johnny ive and. steve.

is interesting it's like you don't wanna. he ruins as many ideas as he generates. yeah. yeah it's a dangerous kind of line to. walk. if you have a lot of ideas like like. gordon bell was famous for ideas. right and it wasn't that the percentage. of good ideas was way higher than. anybody else. it was he had so many ideas and and he. was also good at talking people about it. and. and getting the filters right and you. know seeing through stuff.

whereas elon was like hey i want to. build rockets so. steve would hire a bunch of rocket guys. and elon would go read rocket manuals. so ian is a better engineer a sense like. or like more uh. like a love and passion for the manuals. yeah. and the details the details the. craftsmanship too right. well i guess you had craftsmanship too. but of a different kind. what do you make of the just the. standard for just a little longer what. do you make of like the anger and the. passion and all that the the firing and.

the. mood swings and the madness the. um you know being emotional and all that. that's. steve and i i guess elon too so what is. that a is that a bugger feature. it's a feature so there's a graph which. is. uh y-axis productivity yeah x-axis at. zero it's chaos. yeah and infinity is complete order yeah. right so as you go from the. you know the origin as you improve order.

you improve productivity. yeah and at some point productivity. peaks and then it goes back down again. yeah too much order nothing can happen. yes but the question is. is how close to the chaos is that no no. no here's the thing. is once you start moving the directional. order the force vector to drive you. towards order is unstoppable. oh this is the same every organization. will. move to the place where their. productivity is stymied by order. so you need uh so the question is who's. the counter force.

like because it also feels really good. as you get more organized. then productivity goes up the. organization feels it they orient it. towards it right to hire more people. they get more guys who can run process. you get bigger. right and then inevitably inevitably. the organization gets captured by the. bureaucracy that manages all the. processes. right and then humans really like that. and so if you just walk into a room. and say guys love what you're doing.

but i need you to have less order. if you don't have some force behind that. nothing will happen. i i can't tell you on how many levels. that's profound so. so that's why i'd say it's a feature now. could you be nicer about it. i don't know i don't know any good. examples of being nicer about it. well the funny thing is to get stuff. done you need people who can manage. stuff and manage people because humans. are complicated they need lots of care. and feeding and you need to tell them. they look nice and they're doing good.

stuff and pat them on the back. right i don't know do you tell me is. that is that needed. humans need that i had a friend he. started to manage the group and he said. i figured it out you have to praise them. before they do anything. i was waiting until they were done and. they're always mad at me. now i tell them what a great job they're. doing while they're doing it but then. you get stuck in that trap because then. when they're not doing something how do. you confront these people. i think a lot of people that had trauma. in their childhood who disagree with you. successful people. that you just first do the rough stuff.

and then be nice later. i don't know okay but you know nice. engineering companies are full of adults. who had all kinds of range of childhoods. you know most people had okay childhoods. well i don't know if uh and lots of. people only work for praise. which is weird you mean like everybody. i'm not that interested in this but uh. well you. you're you're probably looking for. somebody's approval um. even still yeah maybe i should think. about that. maybe somebody who's no longer with this.

kind of thing. i don't know i used to call it my dad. and tell him what i was doing he was he. was very excited about engineering and. stuff. you got his approval uh yeah a lot. i was lucky like he he decided i was. smart and unusual as a kid and that was. okay when i was really young. so when i like did poorly in school i. was dyslexic i didn't read until i was. third or fourth grade and they didn't. care my parents were like. oh he'll be fine so.

that was funny that was cool is he still. with us. you miss him sure yeah he had. parkinson's and then cancer his last 10. years were tough. and i killed him killing a man like. that's hard. the mind well it's pretty good um. parkinson's causes slow dementia and uh. the chemotherapy i think accelerated it. but it was like hallucinogenic dementia.

so he was. clever and funny and interesting and. was it was pretty unusual do you. remember conversations. uh of course from that time like where. do you have fond memories. of the guy yeah oh yeah anything come to. mind. a friend told me one time i could draw a. computer on the whiteboard faster than. anybody you'd ever met. and i said you should meet my dad like. when i was a kid he'd come home and say. i was driving by this bridge and i was. thinking about it and he pulled out a. piece of paper and he'd draw the whole.

bridge. he was a mechanical engineer yeah and he. would just draw the whole thing and then. he would tell me about it and. tell me how you would have changed it. and he had this you know. idea that he could understand and. conceive anything and i i just grew up. with that so that was natural. so if you know like when i interview. people i ask them to draw a picture of. something they did on a whiteboard. and it's really interesting like some. people draw a little box. you know and then they'll say and then. this talks to this and yeah. i'd be like this is frustrating and i.

had this other guy come in one time he. says. well i designed a floating point in this. chip but i'd really like to tell you how. the whole thing works and then tell you. how the floating point works inside of. it do you mind if i do that. he covered two whiteboards yeah like 30. minutes and i hired him. like yeah he was great this is craftsman. i mean that's the craftsmanship to that. yeah but also the the mental agility to. understand the whole thing. right put the pieces in contacts like. you know. real view of the balance of how the. design worked. because if you don't understand it.

properly when you start to draw it. you'll fill up half the white board with. like a little piece of it and. you know like your ability to lay it out. in an understandable way it takes a lot. of understanding so. and be able to zoom into the detail and. then zoom out to the zoom. really fast what about the impossible. thing you see your dad. believed that uh you could do anything. that's a weird feature for a craftsman. yeah. it seems that that uh echoes in your own.

behavior. like that's that's the well it's not. that anybody can do anything right now. right it's that if you work at it you. can get better at it. and there might not be a limit. and they did funny things like like he. always wanted to play piano so at the. end of his life he started playing the. piano. when he had parkinson's and he was. terrible. but he thought if he really worked out. in this life maybe the next life he'd be. better at it. he might be onto something yeah. he enjoyed doing it yeah so that's.

pretty funny do you think the. perfect is the enemy of the good in. hardware and software engineering. it's like we were talking about. javascript a little bit and. the messiness of the 10-day building. process. yeah it's you know creative tension. right. the creative tension is you have two. different ideas that you can't do both. right right and then but the fact that. you want to do both. causes you to go try to solve that. problem that's the creative part.

so if you're building computers like. some people say we have the schedule. and anything that doesn't fit in the. schedule we can't do. right so they throw out the perfect. because they have a schedule. i hate that then there's other people. who say we need to get this perfectly. right. and no matter what you know more people. more money. right and there's a really clear idea. about what you want some people are. going to articulate in it. right so let's call that the perfect.

yeah yeah. all right but that's also terrible. because they never ship anything you. never hit any goals. so now you have that now you have your. framework yes you can't throw out stuff. because you can't get it done today. because maybe you get it done tomorrow. or the next project. right you can't so you have to i work. with a guy that i really like working. with. but he over filters his ideas over. filters. he'd start thinking about something and. as soon as he figure out what's wrong. with it you'd throw it out. and then i start thinking about it like.

you know you come up with an idea and. then you find out what's wrong with it. and then you give it a little time to. set because sometimes you know you. figure out. how to tweak it or maybe that idea helps. some other idea. so idea generation is really funny so. you have to give your ideas space. like spaciousness of mind is key but you. also have to execute programs and get. done. and then it turns out computer. engineering is fun because it takes you. know 100 people to build a computer. 200 to 300 whatever the number is and.

people are so. variable about you know temperament and. you know skill sets and stuff. that you know in a big organization you. find that the people who love the. perfect ideas and the people that want. to get stuff done yesterday and. people like to come up with ideas and. people like to. let's say shoot down ideas and it takes. the whole. it takes a large group of people some. are good at generating ideas some are. good at filtering ideas and then. all in that giant mess you somehow.

i guess the goal is for that giant mess. of people to uh. find the perfect path through the. attention the creative tension. but like how do you know when you said. there's some people good at articulating. what perfect looks like what a good. design is like if you're sitting in a. in a room and uh you have a set of ideas. about like how to design uh a better. processor how do you know. this is this is something special here. this is a good idea. let's try this so have you ever.

brainstormed idea with a couple people. that were really smart. and you kind of go into it and you you. don't quite understand it. and you're working on it and then you. start you know. talking about it putting it on the. whiteboard. maybe it takes days or weeks and then. your brain start to kind of synchronize. it's really weird and like you start to. see what each other is thinking. and yeah and it starts to work like you. can see work like my talent in computer.

design is i can. i can see how computers work in my head. like really well. and i know other people can do that too. and when you're working with people that. can do that. like it is kind of a an amazing. experience. and then and every once in a while you. get to that place and then you find the. flaw. which is kind of funny because you you. can you can fool yourself in. but the two of you kind of drifted along. yeah. into the direction that was useless yeah. that happens too like you have to.

because you know well the nice thing. about computer design there's always. reduction in practice. like you come up with your good ideas. and i know some architects who really. love ideas. and then they work on them and they put. it on the shelf they go work on the next. idea and put on the shelf they never. reduce the practice. so they find out what's good and bad. because most. every time i've done something really. new by the time it's done. like the good parts are good but i know. all the flaws like. yeah would you say your career just your.

own experience. is your career defined by mostly by. flaws or by successes. like if again there's great tension. between those if you haven't. tried hard yeah right and done something. new. right then you're not going to be facing. the challenges when you build it then. you find out all the problems with it. and but when you look back you see. problems. okay oh when i look back um what do you. think earlier in my career. yeah like eb5 was the second alpha chip.

uh i was so embarrassed about the. mistake so i could barely talk about it. and it was in the guinness book of. worlds records and it was the fastest. processor on the planet. yeah so it was and at some point i. realized that was really a bad. mental framework to deal with like doing. something new we did a bunch of new. things and some worked out great and. some were bad. and we learned a lot from it and then. the next one we learned a lot. that also ev6 also had some really cool.

things in it i think the proportion of. good stuff went. up but it had a couple of fatal flaws in. it that were. painful and then uh. yeah you learn to channel the pain into. like pride. not pride really you know just uh. realization about how the world works. okay or how that kind of ideas that. works. life is suffering that's the reality. what uh. no it's not well i know the buddha said. that and a couple other people. are stuck on it no it's you know there's.

this kind of weird combination of good. and bad. you know light and darkness that you. have to tolerate and. you know deal with yeah there's. definitely lots of suffering in the. world. depends on the perspective it seems like. there's way more darkness but uh. that makes the light part really nice. what uh. computing hardware. or just any kind of even software design. are you. uh do you find beautiful from your own. work from.

other people's work that you're just uh. we were just talking about the. the battleground of flaws and mistakes. and errors but things that were just. beautifully done is there something that. pops to mind. well when things are beautifully done. usually there's a well thought out set. of abstraction layers. so the whole thing works in unison. nicely yes. and and when i say abstraction layer. that means two different components when.

they work together. they work independently they don't have. to know what the other one is doing. so that decoupling yeah so the famous. one was the the network stack like. there's a seven layer network. you know data transport and protocol and. all the layers. and the innovation was is when they. really wrote got that right. because networks before that didn't. define those very well the layers could. innovate independently and occasionally. the layer boundary would. you know the interface would be upgraded.

and that that let. you know the the design space breathe. and. you could do something new in layer. seven without having to worry about how. layer four. worked right and so good design does. that and you see it in processor designs. when we did the zen design at amd. we made several components very modular. and you know my insistence at the top. was i wanted all the interfaces to find. before we wrote the rtl for the pieces. one of the verification leads said if we.

do this right i can test the pieces. so well independently when we put it. together we won't find all these. interaction bugs because the floating. point knows how the cache works. and i was a little skeptical but he was. mostly right. that the modularity design greatly. improved the quality. is that universally true in general. would you say about good designs the. modularity is. like usually talked about this before. humans are only so smart like. and we're not getting any smarter right. but the complexity of things is going up.

yeah so you know a beautiful design. can't be bigger than the person doing it. it's just you know their piece of it. like the odds of you doing a really. beautiful. design of something that's way too hard. for you is slow. right if it's way too simple for you. it's not that interesting it's like well. anybody could do that. but when you get the right match of your. your expertise and you know mental power. to the right design size. that's cool but that's not big enough to.

make a meaningful impact in the world. so now you have to have some framework. to design the pieces. so that the whole thing is big and. harmonious. but you know when you put it together. it's. you know sufficiently sufficiently. interesting to to be used. and you know so that's like a beautiful. design is. matching the limits of that human. cognitive capacity. to uh to the module you can create and.

creating a nice interface between those. modules. and thereby do you think there's a limit. to the kind of beautiful complex systems. we can build with this kind of modular. design it's like uh. you know if we build increasingly more. complicated you can think of like the. internet. okay let's scale it up you can think of. like social network like twitter. as one computing system. and but those are little modules yeah. right.

but it's built on it's built on so many. components nobody at twitter even. understands right so. so so if an alien showed up and looked. at twitter he wouldn't just see twitter. as a beautiful simple thing that. everybody uses. which is really big you would see the. network. it runs on the fiber optics the data is. transported the computers the whole. thing is so bloody complicated. nobody twitter understands it and so i. think that's what the alienware sees. so yeah if an alien showed up and looked. at twitter or looked at the various. different.

networked systems that you can see on. earth so imagine they were really smart. they could comprehend the whole thing. and then they sort of you know evaluated. the human and thought this is really. interesting no human on this planet. comprehends the system they built. no individual or well would they even. see individual humans that's the. interest. like we humans are very human-centric. entity-centric. and so we think of us as the organ as. the central organism and the networks as. just the connection of organisms. but from a perspective of an alien from.

an outside perspective. it seems like yeah yeah i get it where. the ants and they'd see the ant colony. the ant colony yeah or the result the. production of the ant colony which is. like. cities and it's it's uh. yeah in that sense humans are pretty. impressive the modularity that we're. able to. and the and how robust we are to noise. and. mutation all that kind of stuff well. that's because it's stress tested all. the time. yeah you know you build all these cities.

with buildings and you get earthquakes. occasionally and. you know some you know wars earthquakes. viruses every once in a while. you know changes in business plans for. you know like. shipping or something like like as long. as it's all stress tested then. it keeps adapting to the the situation. so. the that's that's a curious phenomena. well let's go let's talk about moore's. law a little bit. uh at the broad view. of moore's law was just exponential.

improvement of uh. computing capability uh like openai for. example recently. published this kind of papers looking at. the exponential improvement in the. training efficiency of neural networks. for like image net and all that kind of. stuff we just got better on. this is purely software aside just. figuring out. better tricks and algorithms for. training neural networks. and that seems to be improving uh. significantly faster than the moore's.

law prediction. you know so that's in the software space. like. what do you think if moore's law. continues or. if the general version of moore's law. continues. do you think that comes mostly from the. hardware from the software some mix of. the two. some interesting totally uh so not the. reduction of the size of the transistor. kind of thing but more in the uh. in the totally interesting kinds of. innovations in the hardware space all. that kind of stuff.

well there's like a half a dozen things. going on in that graph. so one is there's initial innovations. that had a lot of had room to be. exploited. so you know the efficiency of the. networks has improved dramatically. and then the decomposability of those. and the use go you know they started. running on one computer then multiple. computers and multiple gpus and then. arrays of gpus and. they're up to thousands and at some. point. so so it's sort of like they were.

consumed they were going from like a. single computer application to a. thousand. computer application so that's not. really a moore's law thing that's an. independent vector how many computers. can i put on this problem. because the computers themselves are. getting better on like a moore's law. rate. but their ability to go from one to ten. to a hundred to a thousand. yeah you know was something and then. multiplied by. you know the amount of computes it took. to resolve like alex net to resnet the. transformers it's. it's been quite you know steady.

improvements. but those are like s cars aren't they. yeah that's the exactly kind of s-curves. that are underlying moore's law from the. very beginning. so so what what's the biggest what's the. most. uh productive uh rich. source of s-curves in in the future do. you think is it. hardware is it software or is it so. hardware is going to move along. relatively slowly like you know double. performance every two years. there are there's still i like how you. call that slow yeah that's the slow.

version. the snail's pace of moore's law maybe we. should we should. we should uh trademark that one. whereas the scaling by number of. computers you know can go much faster. you know i'm sure at some point google. had a you know their initial search. engine was running on a laptop you know. like. yeah and at some point they really. worked on scaling that and then they. factored. the indexer from you know this piece and. this piece and this piece and they. spread the data on more things and you.

know they did. a dozen innovations but as they scaled. up the number of computers on that. it kept breaking finding new bottlenecks. in their software and their schedulers. and. and made them rethink like it seems. insane to do a scheduler across a. thousand computers to schedule parts of. it and then send the results to one. computer. but if you want to schedule a million. searches that makes perfect sense. so so there's the the scaling by just. quantity is probably the richest. thing but then as you scale quantity.

like a network that was great on 100. computers may be. completely the wrong one you may pick a. network that's 10 times slower. on 10 000 computers like per computer. but if you go from a hundred to ten. thousand that's a hundred times. so that's one of the things that. happened when we did internet scaling is. the efficiency went down. not up the future of computing is. inefficiency not efficiency. but scales in efficient scale it's it's. scaling faster than inefficiency.

bites you and as long as there's you. know dollar value there like scaling. costs lots of money. yeah but google showed facebook showed. everybody showed that the scale was. where the money was. at it was and so it was worth it. financially do you think. is it possible that like basically the. entirety. of earth will be like a computing. surface like this table will be doing. computing. this hedgehog will be doing computing. like everything really inefficient dumb. computing would be. fiction books they call it computronium.

computing we turn everything into. computing. well most of the elements aren't very. good for anything like you're not going. to make a computer out of iron like. you know silicon and carbon have like. nice structures. you know we'll we'll see what what you. can do with the rest of it. people talk about well maybe we can turn. the sun into computer but it's. it's hydrogen and a little bit of helium. so what i mean is more like actually. just. adding computers to everything oh okay.

so you're just. converting all the mass of the universe. into a computer no no. so not using to be ironic from the. simulation point of view is like. the simulator build mass to simulate. like. yeah i mean yeah so i mean ultimately. this is all heading towards the. simulation yeah well. i i think i might have told you the. story a tesla they were deciding. so they want to measure the current. coming out of the battery and they. decide between putting the resistor in. there. and putting a computer with a sensor in. there. and the computer was faster than the.

computer i worked on in 1982. and we chose the computer because it was. cheaper than the resistor. so so sure this hedgehog you know it. costs 13. and we can put a you know an ai that's. the smartest you in there for five bucks. it'll have one. you know so computers will be you know. he'd be everywhere. i was hoping it wouldn't be smarter than. me because well everything's going to be. smarter than you. but you were saying it's inefficient i. thought it was better to have a lot of. doubt. well well moore's law will slowly.

compact that stuff so even the dump. things will be smarter than us. the dump things are going to be smart or. they're going to be smart enough to talk. to something that's really smart. you know it's like well just remember. like. a big computer chip yeah you know it's. like an inch by an inch. and you know 40 microns thick. it doesn't take very much very many. atoms to make a high power computer. yeah and 10 000 of them can fit in the. shoe box. but you know you have the the cooling.

and power problems but you know. people are working on that but they. still can't write. uh compelling poetry or music or uh. understand what love is or have a fear. of mortality so. so we're still winning neither can most. of humanity so. well they can write books about it so uh. [Laughter]. but but speaking about this uh you know. uh. this walk along the path of innovation. towards uh. the dumb things being smarter than. humans you are now.

the cto of uh. tens torrent as of two months ago. they uh build hardware for deep learning. how do you build scalable and efficient. deep learning this is such a fascinating. space yeah yeah so it's interesting so. um. up until recently i thought there was. two kinds of computers there are serial. computers. that run like c programs and then. there's parallel computers. so the way i think about it is you know. parallel computers you.

have given parallelism like gpus are. great because you have a million pixels. and modern gpus run a program on every. pixel they call the shader program. right so or like finite element analysis. you. you build something you know you make. this into little tiny chunks you give. each chunk to a computer. so you're giving all these chunks a. parallel something like that. but most c programs you write this. linear narrative. and you have to make it go fast to make. it go fast you predict all the branches. all the data fetches and you run that. more in parallel but that's found.

parallelism. ai is i'm still trying to decide how. fundamental this is. it's a given parallelism problem but the. way people. describe the neural networks and then. how they write them in pi torch it makes. graphs yeah. that might be fundamentally different. than the gpu kind of parallelism yeah it. might be. because the when you run the gpu program. on all the pixels. you're running like you know depends you. know this group of pixels say it's.

background blue and that runs a really. simple program this pixel is. you know some patch of your face so you. have some really interesting shader. program to give you impression of. translucency. but the pixels themselves don't talk to. each other there's no graph. right so you you do the image and then. you do the next image and you do the. next image. and you run 8 million pixels 8 million. programs every time and modern gpus have. like 6 000. thread engines in them so you know to.

get 8 million pixels each one runs a. program on you know 10 or 20. pixels and that's how that's how they. work but there's no graph. but you think graph might be a totally. new way to think about hardware so raja. gadori and i've been having this good. conversation. about giving versus found parallelism. and then. the kind of walk cause we got more. transistors like you know computers way. back when did. stuff on scalar data then we did it on. vector data famous vector machines.

now we're making computers that operate. on matrices. right and then the the category we we. said that was next was spatial. like imagine you have so much data that. you know you want to do the compute on. this data. and then when it's done it says send the. result to this pile of data run some. software on that. and it's better to to think about it. spatially than. to move all the data to a central. processor and do all the work. so especially i mean moving in the space.

of data as opposed to moving the data. yeah you have a you have a petabyte data. space spread across some huge array of. computers. and when you do a computation somewhere. you send the result of that computation. or maybe. a pointer to the next program some other. piece of data and do it. but i think a better word might be graph. and all the ai neural networks are. graphs. do some computations send the result. here do another computation do a data. transformation do a merging do a pooling. do another computation is it possible to.

compress and say. how we make this thing efficient this. whole process efficient. that's different so first uh. the fundamental elements in the graphs. are things like matrix multiplies. convolutions data manipulations and data. movements. so gpus emulate those things with their. little singles. you know basically running a single. threaded program. and then there's a you know nvidia calls. it a work where they group a bunch of. programs that are similar together. so for efficiency and instruction use.

and then at a higher level you kind of. you take this graph and you say this. part of the graph is a matrix multiplier. which runs on these. 32 threads but the model at the bottom. was. built for running programs on pixels not. executing graphs. so it's emulation yes so is it possible. to build something that natively runs. graphs. yes so that's what ten storm did. so where are we on that how like in the. history of that effort.

are we in the early days yeah i think so. tense torrance. started by a friend of mine labisha. bajak and i. i was his first investor so i've been. you know kind of. following him and talking to him about. it for years and. in the fall when i was considering. things to do. i decided you know the we we held a. conference last year with a. friend organized it and and we we wanted. to bring in thinkers. and two of the people were andre. carpassi and chris lattner.

and andre gave this talk on youtube. called software 2.0. which i think is great which is we went. from. programmed computers where you write. programs to data program computers. you know like the futures you know of. software as data programs the the. networks. and i think that's true and then. chris has been work he worked on llvm. the low-level virtual machine which. became the intermediate representation. for all compilers.

and now he's working on another project. called mlir which is mid-level. intermediate representation which is. essentially under the graph about how do. you represent that kind of computation. and then coordinate large numbers of. potentially heterogeneous computers. and and i would say technically tense. torrents. you know two pillars are those those. those two ideas software 2.0 and. mid-level representation. but it's in service of executing graph.

programs. the hardware is designed to do that so. it's including the hardware. piece yeah and then the other cool thing. is. for a relatively small amount of money. they did a test chip and two production. chips. so it's like a super effective teams and. and unlike some ai startups where if you. don't build the hardware to run the. software that they really want to do. then you have to fix it by writing lots. more software. so the hardware naturally does matrix. multiply convolution the data.

manipulations. and the data movement between processing. elements that. that you can see in the graph which i. think is all pretty clever. and that's that's what i'm i'm working. on now. so uh the i think it's called the grace. call. processor uh introduced last year it's. uh. you know there's a bunch of measures of. performance we're talking about horses. it seems to outperform 368 trillion. operations per second seems to.

outperform nvidia's tesla t4 system so. these are just numbers. what do they actually mean in real world. perform like what are the. metrics for you that you're chasing in. in your horse race like what do you care. about. well first so the the native language of. you know people who write ai network. programs is pie torch now by torch. tensorflow there's a couple others. the pi torch is one over tensor flows. it's just i'm not an expert on that. i i know many people have switched from. tensorflow to pi torch.

yeah and there's technical reasons for. it and i use both. both are still awesome both are still. awesome but the deepest. love is for pytorch currently yeah. there's more love for that and that that. may change. so the first thing is when they write. their programs. can the hardware execute it pretty much. as it was written. right so pi torch turns into a graph we. have a graph compiler that makes that. graph. then it fractions the graph down so if. you have a big matrix multiply we turn. it into right-sized chunks that run on.

the processing elements. it hooks all the graph up it lays out. all the data there's a couple mid-level. representations of it that are also. simulatable so that. if you're writing the code you can see. how it's going to go through the machine. which is pretty cool and then at the. bottom it schedules kernels like. math data manipulation data movement. kernels which do this stuff. so we don't have to run write a little. program to do matrix multiply because we. have a big matrix multiplier. like there's no cmd program for that.

but there is scheduling for that right. so the the. one of the goals is if you write a piece. of pytorch code that looks pretty. reasonable you should be able to compile. it run it on the hardware. without having to tweak it and and do. all kinds of crazy things to get. performance. there's not a lot of intermediate steps. right it's running directly as right. like on a gpu if you write a large. matrix multiply naively you'll get five. to ten percent of the peak performance. of the gpu. right and then there's a bunch there's a. bunch of people publish papers on this.

and i read them about. what steps do you have to do and it goes. from pretty reasonable. well transpose one of the matrices so. you wrote or not column ordered. you know block it so that you can put a. block of the matrix on different. sms you know groups of threads. but some of it gets into little details. like you have to schedule it just so so. you don't have registered conflicts. so the the the they call them cuda. ninjas.

i love it to get to the optimal point. you either write a pre. use a pre-written library which is a. good strategy for some things. or you have to be an expert in micro. architecture to program it. right so the optimization step is way. more complicated with the gpa so our. our goal is if you write pi torch that's. good pi torch you can do it. now there's as the networks are evolving. you know they've changed from. convolutional to matrix multiply. people are talking about conditional. graphs you're talking about very large. matrices they're talking about sparsity.

you're talking about problems that scale. across many many chips. so the the native you know. data item is a as a packet like so you. send a packet to a processor it gets. processed. it does a bunch of work and then it may. send packets to other processors and. and they execute like a data flow graph. kind of methodology. got it we have a big network on chip and. then 16. the next second chip has 16 ethernet. ports they hook lots of them together. and it's the same graph compiler across.

multiple chips so that's where the scale. comes in so it's built to scale. naturally. now my experience with scaling is as you. scale you run into lots of interesting. problems. so scaling is the mountain to climb yeah. so the hardware is built to do this and. then. we're in the process of is there a. software part to this. with ethernet and all that well the. you know the protocol at the bottom you. know we send you know it's an ethernet. phi but the protocol basically says. send a packet from here to there it's.

all point to point. the header bit says which processor to. send it to and we basically. take a packet off our on-chip network. put an ethernet header on it. send it to the other end to strip the. header off and send it to the local. thing it's pretty straightforward. human human interaction is pretty. straightforward too but when you get a. million of us we could. do some crazy stuff together it could be. fun. so is that the goal is scale so like for. example i've. been recently doing a bunch of robots at. home for my own personal pleasure.

uh am i going to ever use 10 story or is. this more for. there's all kinds of problems like. they're small inference problems or. small training problems there's big. training problems. what's the big goal is it the big. difference uh training problems or the. small training problems. well one of the goals is to scale from. 100 milliwatts to a. to a megawatt you know so like really. have some range on the problems and. the same kind of ai programs work at all. different levels. so that's cool the natural since the.

natural data item is a packet that. we can move around it's built to scale. but so many people have you know small. problems. right right but but uh you know like. inside that phone is a small problem. to solve so do you see that storm. potentially being inside a phone well. the power efficiency of local memory. local computation. and the way we built it is pretty good. and then. there's a lot of efficiency on being. able to do conditional graphs and.

sparsity. i think it for complicated networks i. want to go in a small factor it's been. quite good. um but we have to prove that that's a. that's a fun problem and that's the. early days of the company right it's a. couple years you said. but you think you invested you think. they're legit yeah as you join yeah well. that's well it's also it's a really. interesting place to be. like the ai world is exploding you know. and. i looked at some other opportunities. like build a faster processor which.

people want. yes but that's more on incremental path. than. what's going to happen in ai in the next. 10 years. so this is kind of you know an exciting. place to be part of. the revolutions will be happening in the. very space and then. lots of people are working on it but. there's lots of technical reasons why. some of them you know aren't going to. work out that well. and and you know that's that's. interesting and there's also the same. problem about getting the basics right. like we've talked to customers about. exciting features.

and at some point we realized that each. unit was realizing. they want to hear first about memory. bandwidth local bandwidth compute. intensity. programmability they want to know the. basics power management. how the network ports work what are the. basics do all the basics work. because it's easy to say we got this. great idea that you know the crack gbt3. but the the people we talked to want to. say. if i buy that so we have a pc express. card. with our chip on it if you buy the card. you plug it in your machine you download.

the driver how long does it take me to. get my network to run. right right you know that's a real. question it's a very basic question. so yeah is there an answer to that yet. or is it trying to our goal is like an. hour. okay when can i buy a tesla uh. pretty soon for my for the small case. training yeah pretty soon. months good i love the idea of you. inside the room with the. carpathi andre kapathi and chris ladner. uh very um.

very interesting very brilliant people. very out of the box. thinkers but also like first principles. thinkers well they both. get stuff done they only get stuff done. to get their own projects done they. they talk about it clearly they educate. large numbers of people and they've. created platforms for other people to go. do their stuff on. yeah the the clear thinking that's able. to be communicated is kind of. impressive it's kind of remarkable to. yeah i'm a fan well let me ask because.

uh. i talked to chris actually a lot these. days he's been uh. one of the cool just to give him a shout. out and he's been. so supportive as a human being so. everybody's quite different like great. engineers are different but he's been. like. sensitive to the human element in a way. that's been fascinating. like he was one of the early people on. this stupid podcast that i do to say. like. don't quit this thing and also.

talk to whoever the hell you want to. talk to. that kind of from a legit engineer to. get like. props and be like you can do this that. was. i mean that's what a good leader does. right they just kind of. let a little kid do his thing like go go. do it let's see. let's see see what turns out that that's. a that's a pretty powerful thing but. what do you um what's your sense about. he used to be. he no i think stepped away from google. right. he said sci-fi i think uh.

what what's really impressive to you. about the things that chris has worked. on because it's that we mentioned the. optimization. the compiled design stuff the llvm uh. then there's he's also a google work. that the tpu stuff. he's obviously worked on swift so the. programming language side. talking about people that work in the. entirety of the stack yeah. uh what uh from your time interacting. with chris. and knowing the guy what's really. impressive to you it just inspires you.

well well like llvm became. you know the platform the de facto. platform for. you know compilers like it's it's. amazing. and you know it was good code quality. good design choices he hit the right. level of abstraction. there's a little bit of the right time. in the right place. and then he built a new programming. language called swift. which you know after you know let's say. some adoption resistance became very. successful.

i don't know that much about his work at. google although i know that. you know that was the typical they. started. tensorflow stuff and they you know it. was new is you know. they wrote a lot of code and then at. some point it needed to be refactored. to be you know because it its. development slowed down why. pytorch started a little later and then. passed it so he did a lot of work on. that. and then his idea about mlir which is. what people started to realize is the. complexity of the software stack above.

the low level ir was getting so high. that forcing the features of that into a. level. was was putting too much of a burden on. it so he's splitting that into multiple. pieces. and that was one of the inspirations for. our software stack where we have. several intermediate representations. that are all executable. and you can look at them and do. transformations on them before you lower. the level. so that was i think we started before. moir. really got you know far enough along to.

use. uh but we're interested in that he's. really excited about that malaya. he's that's that's his like little baby. so he you know. and there seems to be some profound. ideas on that that are really useful so. so each one of those things has been. as the world of software gets more and. more complicated how do we. create the right abstraction levels to. simplify it in a way that. people can now work independently on. different levels of it. so i would say all all three of those. projects allovm. swift and mlir did that successfully so.

i'm interested what's. what he's going to do next in the same. kind of way yes. so on either the tpu or maybe the. nvidia gpu side how does 10 story you. think. or the ideas underlying it doesn't have. to be testosterone. just this kind of graph focused. uh graph centric hardware. deep learning-centric hardware beat. nvidia's. do you think it's possible for it to.

basically overtake nvidia. sure what's what's that process look. like what's that. a journey look like you think well gpus. were built around shader programs on. millions of pixels. not to run graphs yes so there's a. hypothesis that says. the way the graphs you know are built. is going to be really interesting to be. inefficient on computing this. and then the the primitives is not a cmd. program. it's matrix multiply convolution and.

then the data manipulations are fairly. extensive about. like how do you do a fast transpose with. a program i don't know if you've ever. written the transpose program. they're ugly and slow but in hardware. you can do really well. like i'll give you an example so when. gpu accelerators first started doing. triangles. like so you have a triangle which maps. on the set of pixels. so you build it's very easy. straightforward to build a hardware. engine that will find all those pixels. and it's kind of weird because you walk. along the triangle to get to the edge. and then you have to go back down to the.

next row and walk along and then you. have to decide on the edge. if the line of the triangle is like half. on the pixel. what's the pixel color because it's half. of this pixel and half the next one. that's called rasterization. because you're saying that could be done. in uh in hardware now. that's an example of that operation as a. software program is really bad. i've written a program that did. rasterization the hardware that does it. has actually less code than. the software program that does it and. it's way faster.

right so there are certain times when. the abstraction you have rasterize a. triangle. you know execute a graph you know. components of a graph. the right thing to do in the hardware. software boundary is for the hardware to. naturally do it. and so the gpu is really optimized for. the rasterization of triangles. well no that's just well like in a. modern you know. that's a small piece of modern gpus what. they did is. that they still rasterize triangles when. you're running the game but for the most.

part. most of the computation in the area the. gpu is running shader programs. but they're single threaded programs on. pixels not graphs. let's be honest to say i don't actually. know the the math behind shader. shading and lighting and all that kind. of stuff i don't know what. they look like little simple floating. point programs or complicated ones you. can have 8 000 instructions in a shader. program. but i i don't have a good intuition why. it could be parallelized so easily. no it's because you have 8 million. pixels in every single so when you have.

a light. right yeah that comes down the angle you. know the amount of light. like like say this is a line of pixels. across this table. right the amount of light on each pixel. is subtly different. and each pixel is responsible for. figuring out what figure it out so that. pixel says on this pixel i know the. angle of the light i know the occlusion. i know the color i am. like every single pixel here is a. different color every single pixel gets. a different amount of light. every single pixel has a subtly. different translucency.

so to make it look realistic the. solution was you run a separate program. on every pixel. see but i thought there's a reflection. from all over the place is every picture. yeah but there is. so so you build a reflection map which. also has some pixelated thing. and then when the pixel is looking at. the reflection map it has to calculate. what the normal of the surface is. and it does it per pixel by the way. there's both loads of hacks on that. you're like you may have a lower. resolution. light map reflection map there's all. these you know attacks they do. but at the end of the day it's per pixel.

computation. and it so happened that you can map uh. graph. like computation onto the this pixel. essentially. you could do floating point programs on. convolutions and matrices. and nvidia invested for years in cuda. first for hpc and then they got lucky. with the ai. trend but do you think they're going to. essentially not be able to. hardcore pivot out of their we'll see. that's always interesting how often do. big companies hardcore pivot.

occasionally. how much do you know about nvidia folks. so. some yeah well i'm i'm curious as well. who's ultimately as a. well they've innovated several times but. they've also worked really hard on. mobile they worked really hard on radios. you know you know they're fundamentally. a gpu company. well they tried to pivot it's an. interesting little uh. game and play in autonomous vehicles. right with. or semi-autonomous like playing with.

tesla and so on and seeing. that's a dipping a toe into that kind of. pivot. they came out with this platform which. is interesting technically. yeah but it was like a three thousand. watt you know. you know thousand watt three three. thousand dollar you know gpu platform. i don't know if it's interesting. technically it's interesting. philosophically i. i technically i don't know if it's the. execution that craftsmanship was there. i'm not sure but that i didn't get a. sense they were. repurposing gpus for an automotive. solution right it's not a real pivot.

they didn't they didn't build a. ground-up. solution right like the like the chips. inside tesla are pretty cheap like. mobile eye has been doing this. they're they're doing the classic work. from the simplest thing yeah you know. they were building 40 mil. square millimeter chips and nvidia their. solution had two. 800 millimeter chips and two 200. millimeter chips and. you know like boatloads are really. expensive drams and. and you know it's a really different. approach. the mobilelite fit the let's say. automotive cost and form factor.

and then they added features as it was. economically viable and. nvidia said take the biggest thing and. we're gonna go make it work. you know and and that's also influenced. like waymo there's a whole bunch of. autonomous startups where they have a. 5000 watt server in their trunk. right and but that's that's because they. think well 5000 watts and you know 10. 000. is okay because it's replacing the. driver elon's approach was that port has. to be cheap enough. to put it in every single tesla whether. they turn on it autonomous driving or.

not. which and mobileye was like we need to. fit in the bomb and you know cost. structure that. car companies do so they may sell you a. gps for 1500 bucks. but the bond for that's like 25. well and uh for mobile eye it seems like. neural networks. were not first-class citizens like the. computation they didn't start out as. a yeah it was a cv problem yeah and did. classic cv and found stop lights and. lines and they were really good at it.

yeah and they never i mean i don't know. what's happening now but they never. fully pivoted i mean it's like it's the. nvidia thing and then. as opposed to so if you look at the new. tesla work. it's like neural networks from the. ground up yeah right. yeah and even tesla started with a lot. of cv stuff in it and andre's basically. been. eliminating it move it move. everything into the network so uh. without. this isn't like confidential stuff but. you sitting on a porch. looking over the world looking at the.

work that andre is doing. that elon's doing with tesla autopilot. uh do you like the trajectory of where. things are going on the floor. they're making serious progress i like. the videos of people. driving the beta stuff like it's taking. some pretty complicated intersections. and all that but it's. it's still an intervention for drive i. mean i i have autopilot the current. autopilot my. my tesla i use it every day do you have. full self-driving beta or no. no so you you like where this is going. we're making progress it's taking longer.

than anybody thought. you know my wonder was. is you know hardware three is it enough. computing. off by two off by five off by ten off by. a hundred. yeah and and i i thought it probably. wasn't enough. but they're doing pretty well with it. now yeah and one thing is. the data set gets bigger the training. gets better and then there's this. interesting thing is. you sort of train and build an arbitrary. size network that solves the problem.

and then you refactor the network down. to the thing that you can afford. to ship right so the the goal isn't to. build the network that fits in the phone. it's to build something that actually. works. and then then how do you make that most. effective on the hardware you have. and they seem to be doing that much. better than a couple years ago. well the one really important thing is. also what they're doing well is. how to iterate that quickly which means. like it's not just about one time.

deployment one building is constantly. entering the network. and trying to automate as many steps as. possible right. and that's actually the principles of. the software 2.0 like you mentioned with. andre. is uh it's not just. i mean i don't know what the actual his. description of software 2.0 is. if it's just high-level philosophical or. their specifics but the. interesting thing about what that. actually looks in the real world. is it's that uh what i think andre calls.

the data engine it's like. it's the iterative improvement of the. thing you have a neural network. that uh does stuff fails on a bunch of. things and learns from it over and over. and over so you're constantly. discovering edge cases. so it's very much about uh like data. engineering like. figuring out it's it's kind of what you. were talking about with testosterone is. you have the data landscape. they have to walk along that data. landscape in a way that uh. that's constantly improving the the the.

neural network and that that feels like. that's the central piece of it yeah. itself and there's two pieces of it like. you you find edge cases that don't work. and then you define something that goes. get your data for that. but then the other constraint is whether. you have to label it or not like the. the amazing thing about like the gpt3. stuff is it's unsupervised. so there's essentially infinite amount. of data now there's obviously infinite. amount of data. available from cars of people. successfully driving. but you know the the current pipelines.

are mostly running on labeled data which. is human limited. so when that becomes unsupervised. right it it'll create unlimited amount. of data which then they'll scale. now the networks that may use that data. might be way too big for cars. but then there'll be the transformation. from now we have unlimited data i know. exactly what i want. now can i turn that into something that. fits in the car. and that pro that process is going to. happen all over the place. every time you get to the place where.

you have unlimited data and that's what. software 2.0 is about. unlimited data training networks to do. stuff. without humans writing code to do it and. ultimately also trying to discover like. you're saying the self-supervised. formulation of the problem so the. unsupervised formulation of the problem. like uh you know in driving there's this. really interesting. thing which is you look at a scene. that's before you and you have data. about what a successful human driver did.

in that scene you know one second later. it's a little piece of data that you can. use just like with gpt-3 as training. currently even even though tesla says. they're using that it's an open question. to me. how much how far can you can you sell. all of the driving with just. that self-supervised piece of data. and like i i think that's what comedy is. doing. that's what common ai is doing but the. question is how. how much data so what comedy ai doesn't.

have. is as good of a data engine for example. as tesla does that's where the. like the organization of the data i mean. as far as i know i haven't talked to. george. but they do have the data the question. is how much data is needed. because we say infinite very loosely. here uh it's it's. and then the other question which you. said i don't know if you think it's. still an open question is. are we in the right order of magnitude. for the compute necessary.

that is is this is it like what elon. said this. chip that's in there now is enough to do. full self-driving. or do we need another order of magnitude. i think nobody actually knows the answer. to that question. i like the confidence that elon has but. yeah we'll see. and there's another funny thing is you. don't learn to drive with infinite. amounts of data. you learn to drive with an intellectual. framework that understands physics and. color and horizontal surfaces and laws. and roads and.

you know all your your uh experience. from manipulating your environment. like look there's so many factors go. into that so then when you learn to. drive. like driving is a subset of this. conceptual framework that you have. right and so with self-driving cars. right now we're teaching them to drive. with driving data. you never teach a human to do that you. teach a human all kinds of interesting. things like. language like don't do that you know. watch out you know there's all kinds of.

stuff going on well this is where you i. think previous time with we talked about. where you poetically uh disagreed with. my naive. uh notion about humans i just think that. humans will will make this whole driving. thing really difficult. yeah all right like i said humans don't. move that slow. it's a ballistics problem it's a. ballistic human zero ballistics problem. which is like poetry to me. it's very it's very possible that in. driving they're indeed purely a. ballistics problem i. and i think that's probably the right. way to think about it but.

i still they still continue to surprise. me those. and damn pedestrians the cyclists other. humans and other cars. and yeah but it's going to be one of. these compensating things so. like when you're driving you have an. intuition about what humans are going to. do. but you don't have 360 cameras and. radars and you have an attention problem. so yeah. so so the self-driving car comes in with. no attention problems 360 cameras right. you know. a bunch of other features yeah so. they'll wipe out a whole class of.

accidents. right and you know you know emergency. braking with radar and. especially as it gets you know ai. enhanced will eliminate collisions. right but then you have the other. problems of these unexpected things. where. you know you think your human intuition. is helping but then the cars also have. you know a set of hardware features that. you're not even close to. and the key thing of course is uh if you. wipe out. a huge number of kind of accidents then. it might be just way. safer than the human driver even though.

even if humans are still a problem. that's hard to figure out yeah that's. probably what happens autonomous cars. will have. a small number of accidents humans would. have avoided but they'll wipe. they'll get rid of the bulk of them what. do you think about. uh like tesla's dojo efforts. or it can be bigger than tesla in. general it's kind of like the tense. torrent. uh trying to innovate like this is the. dichotomy like. should a company try to from scratch. build its own.

neural network training hardware well. first i think it's great. so we need lots of experiments right and. there's lots of. startups working on this and they're. pursuing different things. you know i was there when we started. dojo and it was sort of like. what's the unconstrained computer. solution. to go do very large training problems. and then there's fun stuff like you know. we said well we have this 10 000 watt. board to cool. well you go talk to guys at spacex and. they think 10 000 watts is a really.

small number not a big number. yeah and and there's brilliant people. working on it i'm. curious to see how it'll come out i i. couldn't tell you you know. i know it pivoted a few times since i. left so so the. cooling does seem to be a big problem i. do like what. elon said about it which is like we. don't want to do the thing. unless it's way better than the. alternative whatever the alternative. is so it has to be way better than like. racks of gpus yeah and the other thing. is just like. you know you know the tesla autonomous.

driving hardware. it was only serving one software stack. and the hardware team and the software. team were tightly coupled. you know if you're building a general. purpose ai solution then you know. there's so many different customers with. so many different needs. now something andre said is i think this. is amazing. 10 years ago like vision recommendation. language were completely different. disciplines. we said the people literally couldn't. talk to each other and three years ago.

it was all neural networks but the very. different neural networks. and recently it's converging on one set. of networks they vary a lot in size. obviously they vary in data. varying outputs but the technology has. converged a good bit. yeah these transformers behind gbt3 it. seems like they could be applied to. video they could be applied to a lot of. yeah and it's like and they're all. really. it was like to literally replace letters. with pixels. yeah it does vision it's amazing so. and then size actually improves the.

thing so the bigger it gets the more. compute you throw at it the better it. gets. the more data you have the better it. gets so. so so then you start to wonder well is. that a fundamental thing or is. is this just another step to some. fundamental understanding about this. kind of computation. which is really interesting us humans. don't want to believe that that kind of. thing will achieve conceptual. understandings you were saying like. you'll figure out physics but maybe it. will. maybe probably will well it's worse than.

that. it'll understand physics in ways that we. can't understand i like to hear stephen. will from. talk where he said you know there's. three generations of physics there was. physics by reasoning well big things. should false faster than small things. right. that's reasoning and then there's. physics by equations. like you know but the number of programs. in the world that are solved with the. single equations relatively low almost. all programs have you know. more than one line of code maybe 100. million lines of code. so you said that now we're going to. physics by equation which is his project.

which is cool. i might point out that there was there. was two. two generations of physics before. reasoning. habit like all animals you know know. things fall and you know birds fly and. you know predators know how to you know. solve a differential equation to cut off. a. accelerating you know curving animal. path yep and then there was uh you know. the gods did it. right so yeah right so.

you know there's five generations now. software 2.0 says programming things is. not the last step. data so there's going to be a physics. past stephen's. wolfram's com that's not explainable. and and actually there's no reason that. i can see while that even that's the. limit. like there's something beyond that i. mean they're usually like usually when. you have this hierarchy it's not like. well if you have this step in this step. in this step and they're all.

qualitatively different. and conceptually different it's not. obvious why you know six is the right. hand number of hierarchy steps in not. seven or eight or. well then it's probably impossible for. us to to comprehend. something that's beyond the thing that's. not explainable. yeah because i think but the thing that. you know understands the thing that's. unexplainable to us. we'll conceive the next one and like i'm. not sure why there's a limit to it. uh your brain hurts that's the sad story.

if if we look at our own brain which is. an interesting uh. illustrative example in your work with. testor. and trying to design deep learning. architectures. uh do you do you think about the brain. at all. maybe from a hardware designer. perspective. if you could uh change something about. the brain what would you change. or do funny question like how would you.

so your brain is really weird like you. know your cereal cortex where we think. we do most of our thinking. is what like six or seven neurons thick. yeah like. that's weird like all the big networks. are way bigger than that. like way deeper so that seems odd. and then you know when you're thinking. if it's if if the input generates a. result you can lose it goes really fast. but if it can't that generates an output. that's interesting which turns into an. input and then your brain. to the point where you mold things over.

for days and how many trips through your. brain. is that right like it's you know 300. milliseconds or something to get through. seven levels of neurons. i forget the number exactly but then it. does it over and over and over as it. searches. and the brain clearly is looks like some. kind of graph because you have a neuron. with you know connections and it talks. to other ones and. it's locally very computationally. intense but it's also. does sparse computations across a pretty. big area. there's a lot of messy biological type.

of things and it's. it's meaning like first of all there's. mechanical chemical and electrical. signals that's all that's going on. then the there's a the asynchronicity. of signals and there's like there's just. a lot of variability that seems. continuous and messy and just a. mess of biology and it's unclear whether. that's a. good thing yeah or it's a bad thing. because if. if it's a good thing that we need to run. the entirety of the evolution. well we're going to have to start with.

basic bacteria to create some imaging. we could you could build a brain with 10. layers would that be better or worse. or more more connections or less. connections or you know. we don't know to what level our brains. are optimized. but if i was changing things like yeah. like you know you can only hold like. seven numbers. in your head yeah like why not 100 or a. million. never thought of that like and why can't. like why can't we have like a floating. point processor that can compute. anything we want. like and see it all properly like that.

would be kind of fun. and why can't we we see in four or eight. dimensions. like because you know 3d is kind of a. drag. like all the hard mass transforms are up. in multiple dimensions. so there's that you know you could. imagine a brain architecture that. you know you could enhance with a whole. bunch of features that would be. you know really useful for thinking. about things it's possible that the. limitations you're describing are. actually. essential for like the constraints are. essential for creating.

like the depth of intelligence like that. the ability to reason you know it's hard. to say because. like your brain is clearly a parallel. processor. you know you know 10 billion neurons. talking to each other at a relatively. low clock rate. but it produces something that looks. like a serial thought process. it's a serial narrative in your head. that's true right but then. there are people famously who are visual. thinkers like. i think i'm a relatively visual thinker.

i can imagine any object. and rotate it in my head and look at it. and there are people who say they don't. think that way at all. and recently i read an article about. people people who say they don't have a. they don't have a voice in their heads. they. can talk but when they you know it's. like well what are you thinking. they'll they'll describe something. that's visual. so that's curious. now if if you're saying.

if we dedicated more hardware to holding. information like you know. 10 numbers or a million numbers like. would that. just distract us from our ability to. form this kind of. singular identity like it dissipates. somehow right. but but maybe you know future humans. will have many identities that. have some higher level organization but. can actually do lots more things in. parallel. yeah there's no reason if we're thinking. modularly there's no reason we can't. have multiple consciousnesses in one.

brain. yeah and maybe there's some way to make. it faster so that the. you know the the area the computation. could. could still have a unified feel to it. but while still having way more ability. to do parallel stuff at the same time. could definitely be improved it could be. improved okay. well it's it's pretty good right now. actually people don't give it enough. credit the thing is pretty nice the. the you know the the fact that the right. ends seem to be.

on give a nice like spark. of uh beauty to the whole experience. i don't know i don't know if it can be. improved easily it could be more. beautiful. i don't know how yeah what do you mean. what do you mean how. all the ways you can't imagine no but. that's the whole point. i wouldn't be able to i'm at the fact. that i can imagine ways in. in in which it could be more beautiful. means. so do you know you know ian banks his. stories. so the the super smart ais there.

live mostly live in the world of what. they call infinite fun. because they can create arbitrary. worlds so they interact and you know the. story has it they interact in the normal. world and they're very smart and they. can do all kinds of stuff. and you know a given mind can you know. talk to a million humans at the same. time because we're very slow and. for reasons you know artificial the. story they're interested in people and. doing stuff but. they mostly live in this this other land.

of thinking. my inclination is to think that the. ability to create infinite fun will. um will not be so fun. that's sad there are so many things to. do imagine be able to make a star. move planets around yeah yeah but. because we can imagine that as wildlife. is fun. if we can if we actually were able to do. it it'd be a slippery slope. where fun wouldn't even have a meaning. because we just consistently. desensitize ourselves by the infinite.

amounts of fun we're having. and the sadness uh the the dark stuff is. what makes it fun. i think i mean that could be the russian. it could be the. could be the fun makes it fun and the. sadnesses makes it bittersweet. yeah that's true fun could be uh the. thing that makes it fun. so what do you think about the expansion. not through the biology side but through. the bci the brain computer interfaces. yeah you got a chance to check out the. neural link stuff it's super interesting.

like like humans like like our thoughts. to manifest as action. you know like like as a kid you know. like shooting a rifle was super fun. driving a mini bike doing things and. then computer games i think. for a lot of kids became the thing where. they you know they can do what they want. they can fly a. plane they can do this they can do this. right but you have to. have this physical interaction now. imagine. you know you could just imagine stuff.

and it happens. right like really richly. and interestingly like we kind of do. that when we dream like dream. dreams are funny because like if you. have some control or awareness in your. dreams. like it's very realistic looking or not. realistic. it depends on the dream but you can also. manipulate that. and you know what what's possible there. is is. is odd and the fact that nobody. understands it's hilarious but. um do you think it's possible to expand.

that capability through computing. sure is there some interesting so from a. hardware designer perspective. is there do you think you'll present. totally new challenges and the kind of. hardware that. required that like so this hardware. isn't. standalone computing well this just. knows. today computer games are rendered by. gpus. right right so but you've seen the gans. stuff. yep right where trained neural networks. render realistic images but there's no.

pixels no triangles no shaders. no light maps no nothing so the future. of. graphics is probably ai right yes. now that ai is heavily trained by lots. of real. data right so if you have an interface. with a. aai renderer right. so if you say render a cat it won't say. well how tall is the cat and how big it. you know it'll render a cat and you. might say well a little bigger a little. smaller you know. make it a tabby shorter hair you know.

like you could tweak it. like the the amount of data you'll have. to. send to interact with a very powerful ai. renderer. could be low but the question is for. brain computer interfaces. would need to render not onto a screen. but. render onto the brain and. like directly so that there's a. bandwidth you could do it both ways i. mean our eyes are really good sensors. it could render onto a screen and. we could feel like we're participating.

in it you know they're gonna. they're gonna have you know like the. oculus kind of stuff it's gonna be so. good when a projection to your eyes you. think it's real. you know they're slowly solving those. problems. and i suspect when the renderer. of that information into your head is. also ai mediated. you know they'll be able to give you the. cues that you know you really want for. depth and all kinds of stuff like your. your brain is probably faking your your. visual.

field right like your eyes are twitching. around but you don't notice that. occasionally they blank you don't notice. that you know there's all kinds of. things like you think you see over here. but you don't really see there yeah it's. all fabricated. yeah so yeah peripheral vision is. fascinating. so if you have an ai renderer that's. trained to understand. exactly how you see and the kind of. things that enhance the realism of the. experience. it could be super real actually.

so i don't know what the limits that are. but. obviously if if we have a brain. interface that goes in. inside your you know visual cortex in a. better way than your eyes do. which is possible it's a lot neurons. yeah um maybe that will be even. cooler well the really cool thing is it. has to do with the. the infinite fun that you're referring. to which is our brains seem to be very. limited and like you said computational. so very plastic. very plastic yeah yeah so it's a it's a.

com interesting combination. now the the interesting open question is. the limits of that neuroplasticity like. how how flexible is that thing. because we don't we haven't really. tested it we know about that experiments. where they they put like a pressure pad. on somebody's head. and had a visual transducer pressurize. it and somebody slowly learned to see. yep that's like it's especially at a. young age. if you throw a lot at it like what what. can it uh.

uh can it completely so can you like. arbitrarily expand it with computing. power so. connected to the internet directly. somehow yeah the answer's probably yes. so the problem with biology and ethics. is like there's a mess there. like us humans are perhaps unwilling to. take. risks in uh into directions that are. full of uncertainty. so that's like 90 of the population is. unwilling to take risks the other 10. is rushing into the risks unaided by any.

infrastructure whatsoever. and you know and that that's where all. the fun. happens in you know society there's been. huge transformations. yeah in the last you know a couple. thousand years yeah. it's funny i mean i got a chance to. interact with uh uh. this is matthew johnson from johns. hopkins he's doing this large-scale. study of psychedelics. it's it's becoming more and more i've. gotten a chance to interact with that. community of scientists working on. psychedelics. but because of that that opened the door. to me to.

all these uh what are they called. psychonauts the. people who like you said the ten percent. who like. i don't care i don't know if there's a. science behind this i'm taking the. spaceship. to if i'm being the first on mars i'll. be uh. the you know you know psychedelic's. interesting in the sense that. in another dimension uh like you said. it's a way to explore the. with the limits of the human mind like. what is this thing capable of doing. because you kind of like when you dream.

you detach it i don't know exactly in. your science of it but. you detach your like reality. from what your mind the images your mind. is able to conjure up and your mind goes. into weird places. and like entities appear freudian type. of. like trauma is probably connected in. there somehow but you start to have like. these. weird vivid worlds that like so do you. actively dream. do you why not i had like six six.

hours of dreams and i it's like really. useful time i know. i do i haven't uh i don't for some. reason i just knock out. and uh i have sometimes like anxiety. inducing kind of like. very pragmatic like nightmare type of. dreams but not nothing fun nothing. nothing fun nothing fun i i try. i unfortunately have mostly have fun. in uh the waking world which is very. limited in the amount of fun you can. have. it's not that limited either yeah that's.

what we'll have to talk. yeah i need instructions uh yeah there's. like a manual for that you might wanna. i looked it up i'll ask elon what uh. what did you dream. you know years ago and i i read about. you know. like you know a book about how to have. you know become aware of your dreams. i worked on it for a while like there's. this trick about you know imagine you. can see your hands and look out. and and i got somewhat good at it like.

but my mostly when i'm thinking about. things or working on problems. i i i prep myself before i go to sleep. it's like. i i pull into my mind all the things i. want to work on or think about. and then that let's say greatly improves. the chances that i'll. i'll work on that while i'm sleeping. and then and then i also. you know basically asked to remember it.

and i often remember very detailed. within the dream yeah or outside the. dream well. to bring it up in in my dreaming and. then remember it when i wake up. it's just it's more of a meditative. practice you say. you know to prepare yourself to do that. like if you go to you know the sleep. still gnashing your teeth about some. random thing that happened. that you're not that really interested. in you'll dream about it. that's really interesting maybe but but. you can direct your dreams.

somewhat by prepping you know i'm going. to try that it's really interesting. like the most important the interesting. not like uh. what what did this guy send in an email. kind of like stupid worry stuff but like. fundamental problems you're actually. concerned about. prepping and interesting things you're. worried about or just you're reading or. you know some great conversation you had. or something. some adventure you want to have like. there's there's a lot of. space there and. and it seems to work that.

you know my percentage of interesting. dreams and memories went up. is there uh is that the source of uh. if you were able to deconstruct like. where some of your best ideas came from. do is there a process that's at the core. of that. yeah like so some people you know walk. and think some people like in the shower. the best ideas hit them. if you talk about like newton apple. hitting them on the head. no i i found that a long time ago i'm i.

process things somewhat slowly. so like in college i had friends that. could study at the last minute get an a. next day. i can't do that at all so i always front. loaded all the work. like i do all the problems early you. know. for finals like the last three days i. wouldn't look at a book. because i want you know because like a. new fact the day before. finals may screw up my understanding of. what i thought i knew so my. my goal was to always get it in and and. give it time to soak. and i used to you know i remember we.

were doing like 3d calculus i would have. these amazing dreams of 3d surfaces with. normal. you know calculating the gradient and. this is like all come up so. it was really fun like very visual and. uh. and if i got cycles of that that was. useful. um and the other is don't over filter. your ideas. like i like that process of. brainstorming where lots of ideas can. happen i like people who have lots of. ideas. and things but that's what's up then. there's a yeah let them sit and let it.

breathe a little bit. and then reduce it to practice like at. some point you really have to. does it really work like you know is. this real or not. right but you but you have to do both. there's creative tension there like how. do you. be both open and you know precise. if you had ideas that you just that sit. in your mind for like years. before the sure it's an interesting uh. way to is generate ideas and just let.

them sit. let them sit there for a while. i think i have a few of those ideas you. know that was so funny. yeah i think that's you know creativity. uh this one or something for the slow. thinkers in the. in the room i suppose as i some people. like you said are just like. like the yeah it's really interesting. like there's so much diversity in how. people think. you know how fast or slow they are how. well they remember don't.

like you know i'm not super good at. remembering facts but processes and. methods. like in our engineering i went to penn. state and almost all our engineering. tests were open book i could remember. the page and not the formula. but as soon as i saw the formula i could. remember the whole method if i. if i'd learned it yeah you know so it's. just a funny. or some people could you know i i. swatched friends like flipping through. the book trying to find the formula. even knowing that they'd done just as. much work and i would just open the book.

i was on page 27. about half i could see the whole thing. visually. yeah and you know and you have to learn. that about yourself and figure out what. to do with the. function optimally i had a friend who he. was always concerned he didn't know how. he came up with ideas. he had lots of ideas but he said they. just sort of popped up. like you'd be working on something. having this idea like where does it come. from. but you can have more awareness of it. like like. like like how you how your brain works. is a little murky as you go down.

from the voice in your head or the. obvious visualizations. like when you visualize something how. does that happen yes. you know if i say you know visualize. volcano it's easy to do right. and what does it actually look like when. you visualize it i can visualize to the. point where i don't see. very much out of my eyes and i see the. colors of the thing i'm visualizing. yeah but there's like a there's a shape. there's a texture there's a color but. there's also conceptual visualization. like. what are you actually visualizing when. you're visualizing volcano just like. with peripheral vision you think you see. the whole thing yeah yeah.

that's a good way to say it you know you. have this kind of. almost peripheral vision of your. visualizations they're like these ghosts. but if you know if you if you work on it. you can get a pretty high level of. detail. and somehow you can walk along those. visualizations to come up with an idea. which is uh. but weird but when you're thinking about. solving problems. like you're you're putting information. and you're exercising the stuff you do. know. you're sort of teasing the area that's. you don't understand and don't know.

but you can almost you know feel. you know that process happening you know. that's that's how i. like like like i know sometimes when i'm. working really hard on something like. like i get really hot when i'm sleeping. and you know it's like. we got the blank throw i wake up all the. blankets are on the floor. and you know every time it's while i. wake up and think wow that was great. you know are you able to uh to reverse. engineer what the hell happened there. oh sometimes it's vivid dreams and.

sometimes it's this kind of like you say. like shadow thinking that you you sort. of have this feeling you're. you're going through this stuff but it's. it's not that obvious isn't that so. amazing that the mind just. does all these little experiments i. never you know i thought i always. thought. it's like a river that you can't you're. just there for the ride but you're right. if you prep. it no it's all understandable meditation. really helps. you you got to start figuring out you. need to learn language of your own mind. and there's multiple levels of it but.

the abstractions again right it's. somewhat comprehensible and observable. and. feelable or whatever the right word is. no it's. you know you're not long for the ride. you are the ride. i have to ask you hardware engineer. working on neural networks now. what's consciousness what the hell is. that thing is that is that just some. little weird quirk of our particular uh. computing device. or is it something fundamental that we.

really need to crack open if we're to. to build like good computers do you ever. think about consciousness like. why it feels like something to be i know. it's it's it's really weird. so yeah i mean. everything about it is weird first it's. a half a second behind reality. right it's a post-hoc narrative about. what happened you've already. done stuff by the time you're conscious. of it. and your consciousness generally is a.

single threaded thing but we know your. brain is 10 billion neurons. running some crazy parallel thing. and there's a really big sorting thing. going on there. it also seems to be really reflective in. the sense that. you create a space in your head right. like we don't really see anything right. like. photons hit your eyes it gets turned. into signals it goes through multiple. layers the neurons. you know like i'm so curious that you. know that looks glassy and that looks. not glassy like.

like how the resolution of your vision. is so high you have to go through all. this processing. yeah where for most of it it looks. nothing like vision. okay like like there's no theater in. your mind. right so we we have a world in our heads. we're literally just isolated behind our. sensors. but we can look at it speculate about it. speculate about alternatives problem. solve what if you know there's so many.

things going on. and that process is lagging reality. and it's single threaded even though the. underlying thing is like. massively parallel so it's so curious. so imagine you're building an ai. computer if you wanted to replicate. humans well you'd have huge arrays of. neural networks and. apparently only six or seven deep which. clarious. they only remember seven numbers but i. think we can upgrade that a lot. right and then somewhere in there you. would train the network to create. basically.

the world that you live in right so like. tell. stories to itself about the world that. it's perceiving well. create this create the world tell. stories in the world. and then have many dimensions of. you know like sideshows to it like we. have an emotional structure. like we have a biological structure and. that seems hierarchical too. like like if you're hungry it dominates. your thinking if you're mad it dominates. your thinking. like and we don't know if that's. important to consciousness or not but it.

certainly. disrupts you know in truths in the. consciousness. like so there's lots of structure to. that and we like to dwell on the past we. like to think about the future we like. to imagine. we'd like to fantasize right and. the somewhat circular observation of. that is the thing. we call consciousness now if you created. a computer system it did all things. create world views created future. alternate histories. you know dwelled on past events you know. accurately or semi-accurately.

you know it's it's consciousness just. bring up like. natural well would that feel look and. feel conscious to you like do you think. do you think the thing that looks. conscious is conscious like. do you uh again this is like an. engineering kind of question i think. because. uh like. if we want to engineer consciousness is. it okay to engineer something that just. looks conscious.

or is it is there a difference between. well we have all consciousness because. it's a super effective way to manage our. affairs. yeah it's right the social development. yeah well it gives us the planning. system. you know we have a huge amount of stuff. like when we're talking. like the reason we can talk really fast. is we're modeling each other a really. high level of detail and consciousness. is required for that right and. well all those components together. manifest consciousness. right so if we make intelligent beings. that we want to interact with that we're. like.

you know wondering what they're thinking. you know you know looking forward to. seeing them. you know when they interact with them. they they're interesting surprising. you know fascinating you know they will. probably. feel conscious like we do and we'll. we'll perceive them as conscious. i don't know why not but you never know. another fun question on this because. in in from a computing perspective we're. trying to create something that's. human-like or superhuman-like. let me ask you about aliens aliens.

uh do you think there's intelligent. alien civilizations out there and do you. think. their technology their computing. their ai bots their uh their chips. are of the same nature as ours. yeah i got i have no idea i mean if. there's lots of aliens out there they've. been awfully quiet. you know there's there's speculation. about why.

there seems to be more than enough. planets. out there there's a lot yeah um. there's intelligent life on this planet. that seems quite different you know like. you know dolphins seem like plausibly. understandable octopuses don't seem. understandable at all. if they live longer than a year maybe. they would be running the planet. they seem really smart and their neural. architecture is completely different. than ours. now who knows how they perceive things i. mean that's the question is for us.

intelligent beings who might not be able. to perceive other kinds of intelligence. if they become sufficiently different. than us so yeah. we live in the current constrained world. that you know it's three-dimensional. geometry and. the geometry defines a certain amount of. physics. and you know you know there's like how. time works seems to work. like there's so many things that seem. like a whole bunch of the input. parameters to the you know another. conscious being are the same. yes like if it's biological biological. things seem to be in a relatively narrow.

temperature range. right because you know organic stones. aren't stable too cold or too hot. you know so so there's if you specified. the list of things that. input to that but as soon as we make. really smart. you know beings and they go solve about. how to think about a billion numbers at. the same time and. and how to think in n there's a funny. science fiction book where the all the. society. had uploaded into this matrix and at.

some point some some of the beans in the. matrix. thought i wonder if there's intelligent. life out there. so they had to do a whole bunch of work. to figure out like how to make a. physical thing. because their matrix was self-sustaining. and they made a little spaceship and. they traveled to another planet when. they got there. there was like life running around but. there was no intelligent life. and then they figured out that there was. these huge. you know organic matrix all over the. planet inside there where intelligent.

beings had uploaded themselves. into that matrix so everywhere. intelligent life was as soon as it got. smart. it up leveled itself into something way. more interesting than 3d geometry and. yeah it escaped whatever this is not. escaped. better yeah the the essence of what we. think of as an intelligent being. i tend to like the thought experiment of. the organism like humans aren't the. organisms.

i like the notion of like richard. dawkins and memes. that ideas themselves are the organisms. like that are just using our minds to. evolve. so like we're just like meat receptacles. for ideas to breed and multiply and so. on and. maybe those are the aliens yes so. uh jordan peterson has a line says you. know you think you have ideas but ideas. have you. yeah right good line which and and then.

we know about the phenomena of. groupthink. and there's so many things that. constrain us. but i think you can examine all that and. not be. completely owned by the ideas and. completely sucked into groupthink. and part of your responsibility as a as. a human. is to escape that kind of phenomena. which isn't you know it's. you know it's it's one of the creative. tension things again you're constructed. by it. but you can still observe it and you can. think about it and you can make choices.

about. to some level how constrained you are by. it. and you know it's useful to do that. and but but at the same time. and it could be by doing that that you. know the the the group and society. you're. you're part of becomes collectively even. more interesting. so you know so the outside observer will. think wow. you know all these lexus running around. with all these really independent ideas.

have created something even more. interesting. and uh aggregate so. so i uh so i don't know i'm those are. lenses to look at the situation. but i'll give you some inspiration but i. don't think they're constrained. right you know as a small little quirk. of history. it seems like you're related to jordan. peterson. like you mentioned he's going through. some rough stuff now. is there some comment you can make about. the the roughness of the human journey.

the ups and downs well. i i became an expert in benzo withdrawal. like which is you took benzodiazepines. and at some point. they interact with gaba circuits. you know to reduce anxiety and do 100. other things like there's actually. no known list of everything they do. because they interact with so many parts. of your body. and then once you're on them you. habituate to them and you're you're.

you have a dependency it's not like. you're a drug dependency we're trying to. get high it's a. it's a metabolic dependency and then if. you. discontinue them there's a funny thing. called. kindling which is if you stop them. and then go you know you'll have a. horrible it's for all symptoms if you go. back on them at the same level you won't. be stable. and that unfortunately happened to him. because it's so deeply integrated into. all the kinds of systems in the body it. literally changes the size and numbers.

of. neurotransmitter sites in your brain. yeah so there's a there's a. process called the ashton protocol where. you taper it down. slowly over two years to people go. through that goes through unbelievable. hell. and what jordan went through seemed to. be worse because. the on advice of doctors you know we'll. stop taking these and take this. it was the disaster and he got some. yeah it was pretty tough um he seems to. be doing. quite a bit better intellectually you. can see his brain clicking back together.

i spent a lot of time with i've never. seen anybody suffer so much well his. brain is also like this. powerhouse right so i wonder does a. brain. that's able to think deeply about the. world suffer more through these kinds of. withdrawals like. i don't know i've watched videos of. people going through withdrawal. they they all seem to suffer. unbelievably. and you know my work goes out to. everybody. and there's some funny math about this. some doctors said as best you can tell.

you know there's the standard. recommendations don't take them for more. than a month and then taper over a. couple of weeks. many doctors prescribe them endlessly. which is against the protocol but. it's common right and then. something like 75 percent of people when. they taper. it's you know half the people have. difficulty but 75. get off okay 20 have severe difficulty. and 5. have life-threatening difficulty and if. you're one of those it's really bad. and the stories that people have on this.

is. heartbreaking and tough so you put some. of the fault that the doctors that just. not know what the hell they're doing. oh no it's hard to say it's it's one of. those commonly prescribed things like. one doctor said what happens is if. you're prescribed them for a reason. and then you have a hard time getting. off the protocol basically says you're. either crazy or dependent. and you get kind of pushed into a. different treatment regime. you're a drug drug addict or a. psychiatric patient.

and so like one doctor said you know i. prescribed me for 10 years thinking i. was helping my patients and i realized i. was really harming them. and you know the awareness of that is. slowly coming up. the fact that they're casually. prescribed. to people is horrible. and it's bloody scary and some people. are stable on them but they're on them. for life. like once you know it's another one of. those drugs that. but benzo's long range have real impacts.

on your personality. people talk about the benzo bubble where. you get disassociated from reality and. your friends a little bit. it's it's it's it's really terrible the. mind is terrifying we were talking about. how. how the infinite possibility of fun but. like. it's the infinite possibility of. suffering too which is one of the. dangers of. uh like expansion of the human mind it's. like. i wonder if all the possible huma. experiences that a. intelligent computer can have is it.

mostly fun or is it mostly. suffering so like if you if you uh brute. force expand. the set of possibilities like are you. going to run into some trouble. in terms of like torture and suffering. and so on maybe our human brain is just. protecting us from much more possible. pain and suffering. maybe the space of pain is like much. larger than we could possibly imagine. and that the world's in the balance.

you know all the all the literature on. religion and stuff is. you know the struggle between good and. evil is is balanced versus. very finely tuned for reasons that are. complicated. but that's a that's a long philosophical. conversation. uh speaking of balance that's. complicated i i wonder because we're. living through one of the more important. moments in human history with this. particular virus. it seems like pandemics have at least. the ability to. uh kill off most of the human population.

at their worst and they're just. fascinating because there's so many. viruses in this world. there's so many i mean viruses basically. around the world in the sense that uh. they've been around very long time. they're everywhere. they seem to be extremely powerful and. they're just in a distributed kind of. way but. at the same time they're not intelligent. and they're not even living. do you have like high level thoughts. about this virus that uh. like in terms of you being fascinated. about terrified or. not somewhere in between so i believe in.

frameworks right. so like one of them is the evolution. like we're evolved creatures right yes. and. one of the things about evolution is. it's hyper competitive. and it's not competitive out of a sense. of evil it's competitive in the sense of. there's endless variation in variations. that work better when. and then over time there's so many. levels of that competition. you know like multi-cellular life partly. exists because. of you know the the competition between.

you know different kinds of life forms. and we know sex partly exists to. scramble our genes so that we have. you know genetic variation against. the invasion of the bacteria and the. viruses and it's endless like. i read some funny statistic like the. density of viruses and bacteria in the. ocean. is really high and one third of the. bacteria die every day because the virus. is invading them. like one-third of them wow like. like i don't know if that number is true.

but it was like it's. like there's like the amount of. competition and what's going on is. stunning. and there's a theory as we age we slowly. accumulate bacterias and viruses. and as our immune system kind of goes. down. you know that's what slowly kills us and. it just feels so peaceful from a human. perspective when we sit back and they're. able to have a relaxed conversation. uh and there's wars going on out there. like right now. you're you're harboring how many.

bacteria and you know the ones. many of them are parasites on you and. some of them are helpful and some of. them are modifying your behavior and. some of them are. you know it's just really it's really. wild but. you know this particular manifestation. is unusual. you know in the demographic how it hit. and the political. you know response that it engendered and. you know the health care. response it engendered and the. technology it's gendered it's kind of. wild. yeah the communication on twitter that. it uh every level all that kind of stuff.

at every single level yeah. but but what usually kills life the big. extinctions are caused by. meteors and volcanoes that's the one. you're worried about as opposed to. human-created bombs solar flares are. another good one. you know occasionally solar flares hit. the planet so it's nature. oh yes yeah it's all pretty wild on. another historic moment. this is perhaps outside but perhaps. within your uh.

space of frameworks that you think about. that just happened. i guess a couple weeks ago is um i don't. know if you're paying attention at all. it's the. the game stop and wall street bets. so it's really fascinating there's kind. of a theme to this conversation. today because it's like you know that. works the. it's cool how there's a large number of. people in a distributed way. almost having a kind of fun we're able.

to take on. the powerful elites elite hedge funds. centralized powers and overpower them. uh do you have thoughts i mean saga. i don't know enough about finance but it. was like the elon. you know robin hood guy when they talked. yeah what do you think about that. well the robin guy didn't know how the. finance system worked that was clear. right he was treating like the the. people who settled the transactions as a. black box. and suddenly somebody called him up and.

said hey black box calling you. your transaction volume means you need. to put up three billion dollars right. now and he's like i don't have three. billion dollars. like i don't even make any money on. these trades why do i owe three billion. dollars while you're sponsoring the. trade. so so there was a set of abstractions. that. you know i don't think either like like. now he understands it like. this happens in chip design like you buy. wafers from tsmc or samsung or intel. and you know they say it works like this. and you do your design based on that and. then chip comes back and doesn't work.

and then suddenly you start having to. open the black boxes like the. transistors really work like they said. you know what's the real issue. so so the. there's a whole set of things that. created this opportunity and somebody. spotted it. now people spot these kinds of. opportunities all the times there's been. flash crashes there's been. you know there's always short squeezes. are fairly regular every ceo i know. hates the shorts because they're they're. manipulating.

they're trying to manipulate their stock. in a way that they make money. and you know deprive value from both. this you know the company and the. investors. so the fact that. you know some of these stocks were so. short it's hilarious. that this hasn't happened before i don't. know why and i don't actually know why. some serious hedge funds didn't do it to. other hedge funds. and some of the hedge funds actually. made a lot of money on this yes. so my my guess is we know. five percent of what really happened and.

that a lot of the players don't know. what happened. and the people who probably made the. most money aren't the people that. they're talking about. yeah that's do you think there was. something uh. i mean this is the this is the cool kind. of uh. elon uh you're the same kind of. conversationalist which is like first. principles. questions of like what the hell happened. uh just very basic questions of like was. there something shady going on.

uh what you know who are the parties. involved it's the basic questions that. everybody wants to know about. yeah so like we're in a very hyper. competitive world right. but transactions like buying and selling. stock is a trust event. you know i trust the company. representing themselves properly you. know. i bought the stock because i think it's. going to go up i trust that the. regulations are solid. now inside of that there's all kinds of. places where. you know humans over trust and. you know this this expose let's say some.

weak points in the system. i don't know if it's going to get. corrected i don't know if the. i don't know if we have close to the. real story. yeah my suspicion is we don't yeah and. listen to that guy he was. like a little wide-eyed about and then. he did this and then they did that and. it was like. i think you should know more about that. your business than that. but again there's many businesses when. like this layer is really stable. you stop paying attention to it you pay.

attention to the stuff that's bugging. you. or new you don't pay attention to the. stuff that just seems to work all the. time you just. you know the sky's blue every day. california and where once a while the. continued rains there was like what do. we do. somebody go bring in the lawn furniture. you know like it's getting wet. we don't know it's getting wet yeah it. doesn't it was blue for like 100 days. and now it's. you know so but part of the problem here. with vlad this. the ceo of robin hood is the scaling is. that what we've been talking about is.

there's a lot of unexpected things that. happen with the scaling. and you have to be i think the scaling. forces you to then return to the. fundamentals. well it's interesting because when you. buy and sell stocks the scaling is you. know the stocks only move in a certain. range and if you buy a stock you can. only lose that amount of money. on the short short market you can lose a. lot more than you can. benefit like it has a it has a weird. cost you know cost function or whatever. the right word for that is. so he was trading in a market where he.

wasn't actually capitalized for the. downside. if it got outside a certain range. now whether something the various has. happened i have no idea. but at some point the. financial risk to both him and his. customers was way outside of his. financial capacity. and his understanding how the system. work was clearly. weak or or he didn't represent himself i. you know i don't know the person. when i listened to him nick yeah it. could have been the surprise question.

was like and then these guys called and. you know it sounded like he was treating. stuff as a black box. maybe he shouldn't have but maybe his. whole pilot expert somewhere else and it. was going on i don't i don't know. yep i mean this is uh this is one of the. qualities of. a good leader is under fire you have to. perform. and that means to think clearly and to. speak clearly. and he dropped the ball on those things. because. and understand the problem quickly learn. and understand the problem. like at this like basic level.

like what the hell happened and my guess. is. you know at some level it was amateurs. trading against. you know expert slash insiders slash. people with you know special information. outsiders is insiders yeah and the. insiders. you know my guess is the next time this. happens we'll make money on it. the insiders always win well they have. more. tools and more incentive i mean this. always happens like the outsiders are. doing this for fun the insiders are.

doing this 24 7. but there's numbers in the outsiders. this is the interesting thing. well there's numbers on the insiders too. like different kind of numbers different. kind of numbers. but this could be a new era because i. don't know at least i didn't expect that. uh a bunch of redditors could. you know there's uh you know millions of. people who can get together. the next one won't be a surprise but. don't you think the. the the crowd the people are planning. the next attack. we'll see but it has to be a surprise.

can't be the same game. as to the end it could be there's a very. large number of games to play and they. can be. agile about it i don't know i'm not an. expert right that's a good question how. the space of games how how restricted is. it. yeah and the system is so complicated it. could be relatively unrestricted. and also like you know during the last. couple financial crashes. you know what set it off was you know. sets of derivative events where.

you know the you know nasim talibs you. know thing is. they're they're they're trying to lower. volatility. in the short run but creating tail. events and. systems always evolve towards that and. then they always crash like. the gas curve is the you know star low. ramp plateau crash it's 100. effective in the long run. let me ask you some advice to put on. your profound hat.

what uh there's a bunch of young folks. to listen to this thing. for no good reason whatsoever. undergraduate students maybe high school. students maybe just young folks a young. at heart. looking for the next steps to taking. life what advice would you give to a. young person today about. life maybe career but also life in. general. get good at some stuff well get to know. yourself right. get good at something that you're. actually interested in you have to love.

what you're doing to get good at it. you really got to find that don't waste. all your time doing stuff that's just. boring or bland or numbing. right don't let old people screw you. well people get talked into doing all. kinds of and wrapping up huge. student. you know student debts and like there's. so much crap going on. you know and then it drains your time. and drains yeah the eric weinstein you. know. thesis that you know the older. generation will let go. yeah and they're trapping all the young.

people i think there's some truth to. that. yeah sure just because you're old. doesn't mean you stop thinking i know. lots of really original. yeah old people i'm an old person. so um but you have to be conscious about. it you can fall into the ruts and then. do that you know when i hear young. people spouting opinions that sounds. like they come from fox news or cnn i. think they've been captured by. groupthink and memes and supposed to. think on their own. you know so if you find yourself.

repeating what everybody else is saying. you're not going to have a good life. like like that's not how the world works. it may be. it seems safe but it puts you at great. jeopardy for. well being boring or unhappy or. how long did it take you to find the. thing that uh. you have fun with well i don't know. i've been a fun person since i was. pretty little so everything i've gone. through a couple periods of depression. in my life. for a good reason or for uh the reason.

that doesn't make any sense. yeah like some some things are hard. like you go through mental transitions. in high school i was. really depressed for a year and i think. i had my first midlife crisis at 26. i kind of thought is this all there is. like i was working at a job that i loved. and but i was going to work and all my. time is consumed. what's what's the escape out of that. depression what's the answer to is. is this all there is well.

a friend of mine i asked him because he. was working his ass off i said what's. your work-life balance. like like there's you know work friends. family. personal time are you bouncing in that. and he said work 80. family 20 and i try to i try to find. some time to sleep. like there's no personal time there's no. passion at a time. because you know young people are often. passionate about work so and i was. certainly like that. but you need to you need to have some. space in your life for different things.

and that's that creates uh that makes. you resistant to the whole. the the the dip the the deep dips into. depression kind of thing yeah well you. have to get to know yourself too. meditation helps some physical. something physically intense helps like. the weird places your mind goes kind of. thing. like and why does it happen why do you. do what you do like triggers like. the things that cause your mind to go to. different places kind of thing or.

events like you're upbringing for better. or worse whether your parents are great. people or not. you you you come into. you know adulthood with all kinds of. emotional burdens. yeah and you can see some people are so. bloody stiff and restrained and they. think you know the world's fundamentally. negative. like you maybe you have unexplored. territory. yeah or you're afraid of something uh. definitely afraid of quite a few things. then you gotta go face them like like.

what's the worst thing that happened. you're going to die right. like that's inevitable you might as well. get over that like a 100. death rate like people were worried. about the virus but. you know the human condition is pretty. deadly. there's something about embarrassment. let's see i've competed a lot in my life. and i think the if i'm too introspected. the thing i'm most afraid of is. being like humiliated i think nobody. cares about that. look you're the only person on the. planet zack cares about you being.

humiliated exactly. it's like a really useless thought it is. it's like uh you're all humiliating. something happened in a room full of. people and they walk out and they didn't. think about it one more second. or maybe somebody told a funny story to. somebody else and then it just hates it. throughout yeah. yeah now i know it too i mean. i've been really embarrassed about. that nobody cared about myself. yeah it's a funny thing so the worst. thing ultimately is just. uh yeah but that's the cage and then you.

have to get out of it yeah. like once you here's the thing once you. find something like that. you have to be determined to break it. because otherwise you'll just you know. slowly accumulate that kind of junk and. then you die as a. you know a mess so the goal i guess it's. so. it's like a cage with a cage i guess the. goal is to die in the biggest possible. cage. well ideally you have no cage. people do get enlightened i've got a few. it's great you found a few there's a few. out there i don't know of course.

um either that or they have you know. it's a great sales pitch there's like. enlightened people writing books and. doing all kinds of stuff. it's a good way to sell a book i'll give. you that you've never met somebody you. just thought. they just killed me like this like like. mental clarity humor no 100 but i just. feel like they're living in a bigger. cage they have their own. they don't think there's a cage they're. still okay you secretly suspect there's. always the case. ah there's no there's nothing outside. the the unit there's nothing outside the.

cage. [Laughter]. you were you worked in a bunch of. companies. uh you led a lot of amazing teams. um i don't i'm not sure if you've ever. been like at the early stages of a. startup. but do you have advice for. uh somebody that wants to uh do a. startup or build a company. like build a strong team of engineers.

that are passionate just want to. uh solve a big problem like is there uh. more specifically on that point well you. have to be really good at stuff. if you're going to lead and build a team. you better be really interested in how. people work and think. the people or the solution to the. problems there's two things right. one is how people work and the other is. actually. there's there's quite a few successful. startups it's pretty clear the founders. don't know anything about people. like the idea was so powerful that it.

propelled them. but i suspect somewhere early they they. hired some people who understood people. because people really need a lot of care. and feeding to collaborate and work. together and. feel engaged and work hard you know like. startups are. all about out producing other people. like you're nimble because you don't. have any legacy. you don't have you know a bunch of. people who are depressed about life. you know just showing up you know so. startups have a lot of advantages that. way. you know do you like the the steve jobs.

talked about this idea. of a players and b players i don't know. if you uh. know this formulation yeah no um. organizations that get taken over by pb. player leaders. often really underperform their rc. players. that said in big organizations there's. so much work to do. like and there's so many people who are. happy to do what you know like the. leadership or the. big idea people who can see it consider. menial jobs.

and you know you need a place for them. but you need an organization that. both values and rewards them but doesn't. let them take over the leadership of it. got it but so so you need to have an. organization that's resistant to that. but. in the early days the the notion with. with steve was that like one b player in. a room. of a players will be like destructive to. the whole. i've seen that happen i i don't know if. it's like always true. like you know you you run into people. who are clearly b players but they think.

they're very players and so they have a. loud voice at the table and they make. lots of demands for that. but there's other people are like i know. who i am i just want to work with you. know cool people and cool and just. tell me what to do and i'll go get it. done. yeah you know so you have to again this. is like people skills like. what kind of person is it you know i've. met some really great. people i love working with that weren't. the biggest id people the most. productive ever but they show up they. get it done. you know they create connection and. community that people. value it's it's it's pretty diverse so i.

don't think there's a recipe for that. i gotta ask you about love i heard. you're into this now. into this love thing yeah is this is you. think this is your solution to your. depression. no i'm just trying to like you said the. enlightened people on occasion trying to. sell a book i'm writing a book about. love you're writing a book about me no. i'm not i'm not. a friend of mine he's gonna. somebody said you should really write a. book about your you know your management. philosophy he said it'd be a short book.

[Laughter]. well that one was all pretty well uh. what role do you think. love family friendship all that kind of. uh. human stuff play in a successful life. you've been exceptionally successful in. the space of. like running teams building cool in. this world. creating some amazing things what uh did. love get in the way did love help. the family get in the way to family help. friendship you want the engineer's.

answer. please so but first love is functional. right. it's functional in what way so we. habituate ourselves to the environment. and actually jordan told me jordan. peterson told me this line. so you go through life and you just get. used to everything except for the things. you love. they they remain new like this is really. useful for you know. like like other people's children and. dogs and you know trees. you just don't pay that much attention. to your own kids you monitor them really. closely.

like and if they go off a little bit. because you love them if you're smart. if you're going to be a successful. parent you notice it right away. you don't habituate to. just things you love and if you want to. be successful at work if you don't love. it. you're not going to put the time in. somebody else it's somebody else that. loves it like. because it's new and interesting and. that lets you go to the next level. so it's the thing it's just a function. that generates newness.

and novelty and surprises you know those. kind of things. it's really interesting but and there's. people figured out lots of. you know frameworks for this you know. like like humans seem to go in. partnership go through. you know interest like somebody suddenly. somebody's interesting. and then you're infatuated with them and. then you're in love with them. and then you you know different people. have ideas about parental love or mature. love like you go through a cycle of that. which keeps us together and it's you. know super functional for creating. families and.

and creating communities and making you. support somebody despite the fact that. you don't love them. like and and. it can be really enriching you know. now in the work life balance scheme if. all you do is work. you think you may be optimizing your. work potential but if you don't. love your work or you don't have family. and friends and things you care about. your brain isn't well balanced like.

everybody knows the experience of you. works on something all week you went. home and took two days off and you came. back in. the odds of you working on the thing you. picking up right where you left off is. zero. your brain refactored it. but being in blood is great it's like. changes the color of the light in the. room. it creates a spaciousness that's that's. different it helps you think. it makes you strong buckowski had this.

line about. love being a fog that dissipates with. the first light of reality. in the morning it's that's depressing i. think it's the other way around. it lasts well you like you said it's a. function it's a thing that just. be the light that actually enlivens your. world and. creates the interest and the power and. the strength and the. to go do something well it's like. like that sounds like you know there's. like physical love emotional of. intellectual love spiritually yeah. right isn't it all the same thing kind. of nope.

you should differentiate that maybe. that's your problem in your book you. should you should refine that a little. bit. different chapters yeah there's. different chapters what's that what's. these are there aren't these are just. different layers of the same thing. the stack no physical people people. some people are addicted to physical. love and they have no idea about. emotional or intellectual love. i don't know if they're the same thing. so i think they're different that's true. they could be different it'd be. it i guess the ultimate goal is for it. to be the same well if you want. something to be bigger and interesting.

you should find. all its components and differentiate. them not climb it together. people do this all the time they yeah. and the modularity. get your abstraction layers right and. then you can you have room to breathe. well maybe you can write the forward to. my book about love. yeah or the afterwards and the after you. really tried. i feel like lex has made a lot of. progress in this book but. uh well you have things in your life. that you love yeah. yeah you know so and they are you're. right they're modular it's.

and you can have multiple things with. the same person or the same. thing and yeah but yeah. depending on the moment of the day yeah. there's like what bukowski. described as that moment you go from. being in love to having. a different kind of love yeah right and. that's the transition. but when it happens if you read the. owner's manual and you believed it. you would have said oh this happened it. doesn't mean it's not love it's a. different kind of love. but but maybe there's something better.

about. that as you grow old if all you do is. regret. how you used to be it's sad right. you should have learned a lot of things. because like who you can be in your. future self is. is actually more interesting and. possibly delightful than. you know being a mad kid in love with. the. the next person like that's super fun. when it happens but. that's that's you know five percent of. the possibility.

yeah that's right that there's a lot. more fun to be had in the long lasting. stuff. yeah or meaning you know if that's me. which is a kind of fun. it's a deeper kind of fun and it's. surprising you know that's. like like the thing i like is surprises. you know and you just never know what's. gonna happen yeah. but you have to look carefully and you. have to work at it you have to think. about it and. you know it's yeah you have to see the. surprises when they happen. right you have to be looking for it from. the branching perspective. you mentioned regrets uh.

do you have regrets about your own. trajectory oh yeah of course. yeah some of it's painful but you want. to hear the painful stuff. i'd say like in terms of working with. people. when people did say stuff i didn't like. especially if it was a bit nefarious. i took it personally and i also felt it. was personal about them but a lot of. times. like humans are you know most humans are. a mess right and then they act out and.

they do stuff. and i this psychologist i heard a long. time ago said. you tend to think somebody does. something to you. but really what they're doing is they're. doing what they're doing while they're. in front of you. it's not that much about you yeah right. and. as i got more interested in. you know when i work with people i think. about them and. probably analyze them and understand. them a little bit and then when they do. stuff i'm way less surprised and i'm. wait you know and if it's bad i'm way.

less. hurt and i react way less like i sort of. expect. everybody's got their yeah and it's. not about you. it's not about me that much it's like. you know. you know you do something and you think. you're embarrassed but nobody cares. like and somebody's really mad at you at. the odds of it being about. you yeah no they're getting mad the way. they're doing that because of some. pattern they learned. and you know and maybe you can help them. if you care enough about it. but or you could step you could see it. coming and step out of the way.

like like i wish i was way better at. that i'm i'm a bit of a hothead. and and you said with steve that was a. feature not a bug. yeah well he was using it as the counter. force the orderliness that would crush. his work well you were doing the same. yeah maybe i don't think i don't think. my uh my vision was big enough. it was more like i just got pissed off. and did stuff. i'm sure that's just yeah yeah you're. telling me. i don't know if it had the it didn't. have the amazing effect of creating the.

trillion dollar company it was more like. i just got pissed off and left. and or made enemies that he shouldn't. have been. yeah it's hard like i didn't really. understand politics until i worked at. apple. where you know steve was a master player. of politics and his staff had to be or. they wouldn't survive them and. it was definitely part of the culture. and then i've been in companies where. they say it's political but it's all. you know fun and games compared to apple. and it's not that. the people apple are bad people it's.

just they operated politically at a. higher level. you know it's not like oh somebody said. something bad about somebody. somebody else which is most politics. it's. you know they they had strategies about. accomplishing their goals. sometimes you know over the dead bodies. of their enemies. you know with some communication yeah. more game of thrones and sophistication. and like a big time factor rather than a. you know well that requires a lot of. control over your emotions i think.

uh to do to have a bigger strategy in. the way you behave. yeah and it's it's it's effective in the. sense that coordinating thousands of. people to do really hard things. where many of the people in there don't. understand themselves much less how. they're participating. yeah creates all kinds of. you know drama and problems that you. know our solution is political in nature. like how do you convince people how do. you leverage them how do you motivate. them how do you get rid of them how you.

know like there's. there's so many layers of that that are. interesting. and even though some some of it let's. say may be tough. it's not evil. unless you know you use that skill to. evil purposes which some people. obviously do but. but it's a skill set that operates you. know and i wish i'd. you know i was interested in it but i. you know it was sort of like i'm an. engineer i do my thing. and you know there's there's times when. i could have way bigger. impact if i you know knew how to.

if i paid more attention and knew more. about that. about the human layer of the stack yeah. that human. political power you know expression. layer of the stack. which is complicated and there's lots to. know about it i mean people are good at. it are just amazing. and when they're good at it and let's. say. relatively kind and oriented in a good. direction. you can really feel it can get lots of. stuff done and coordinate things you.

never thought possible. but all people like that also have some. pretty hard edges because. you know it's it's a heavy lift and i. wish i'd spent more time with that when. i was younger but. but maybe i wasn't ready you know i was. a wide-eyed kid for 30 years. it's a little bit of a kid i know what. do you. hope your legacy is when there's a. when there's a book like a hitchhiker's. guide to the galaxy and this is like a. one sentence entry ball jim caller from.

like that guy lived at some point. there's not many you know not many. people be remembered uh you're one of. the. sparkling little human creatures. that had a big impact on the world how. do you hold you'll be remembered. my daughter was trying to get uh she. added my wikipedia page to say that i. was a legend and a guru. but they took it out so she put it back. in she's 15. i think i think that was probably the. best part of my legacy.

[Laughter]. she got her sister they were all excited. they were like trying to put it in the. references because there's articles in. that and. they're telling you that so the eyes of. your kids your. uh legend well they're pretty skeptical. because they'll be better than that. they're like dad so yeah that's. that's stupid that kind of stuff is. super fun in terms of the big legends. stuff anchor. okay legacy i don't really care you're. just an engineer. no they've been thinking about building. a big pyramid.

so i had a debate with a friend about. whether pyramids or craters are cooler. and you realize that there's craters. everywhere but you know they built a. couple of pyramids five thousand years. ago in there. and they remember you think that would. be foreign. uh those aren't easy to build oh i know. and they don't actually know how they. built them which is great. it's either uh agi or aliens could be. involved so i think.

i think you're gonna have to figure out. quite a few more things than just. the basics of civil engineering. so i guess you hope your legacy is. pyramids. that would that would be cool and my. wikipedia page you know getting updated. by my daughter periodically. like those two things would pretty much. make it jim it's a huge honor talking to. you again i hope we talk many more times. in the future. i can't wait to see what you do with. tennis torrent. i can't wait to use it i can't wait for. you to revolutionize.

yet another space in computing. it's a huge honor to talk to you thanks. for talking today this was fun. thanks for listening to this. conversation with jim keller and thank. you to our sponsors athletic greens. all-in-one nutrition drink brooklyn and. sheetz. expressvpn and bel campbell grass-fed. meat. click the sponsor links to get a. discount and to support this podcast. and now let me leave you with some words. from alan turing. those who can imagine anything can.

create. the impossible thank you for listening. and hope to see you. next time. you.