Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241

the following is a conversation with. boris sofman who is the senior director. of engineering and head of trucking at. waymo the autonomous vehicle company. formerly the google self-driving car. project. before that boris was the co-founder and. ceo of anki a robotics company that. created cosmo which in my opinion is one. of the most incredible social robots. ever built. it's a toy robot but one with an. emotional intelligence that creates a. fun and engaging human robot interaction.

it was truly sad for me to see anki shut. down when he did. i had high hopes for those little robots. we talk about this story and the future. of autonomous trucks vehicles and. robotics in general. i spoke with steve vaseli recently on. episode 237 about the human side of. trucking this episode looks more at the. robotic side. this is the lex friedman podcast to. support it please check out our sponsors.

in the description. and now here's my conversation with. boris. sofman. who is your favorite robot in science. fiction books or movies. wally and r2d2 where they were able to. convey such an incredible degree of. intent emotion and kind of character. attachment without. having any language whatsoever. and just purely through the emotion. richness of emotional interaction so. those were fantastic and then uh.

the terminator series just like really. really. pretty wide wide range right uh but uh i. kind of love this uh dynamic where you. have this like incredible terminator. itself that arnold played but uh and. then he was kind of like the inferior. like previous generation version that. was like totally outmatched uh you know. in terms of kind of specs by the new one. but you know still kind of like held his. own and so it was kind of interesting. where you you realize how many how many. levels there are on the spectrum from. human to kind of potentials and ai and.

robotics to uh futures and so yeah that. movie really uh as much as it was like. kind of a dark world in a way was. actually quite fascinating gets the. imagination going well from an. engineering perspective both the movies. you mentioned wally and. terminator the first one. is probably achievable you know humanoid. robot. maybe not with like the realism in terms. of skin and so on but. that humanoid form we have the humanoid. form it seems like a compelling form.

maybe the challenge is just super. expensive to engine to build but you can. imagine maybe not a machine of war but. you could imagine terminator type robots. walking around. and then the same obviously with wall-e. you've basically so for people who don't. know you uh created the company anki. that created a small robot with a big. personality called coswell that just it. does exactly what wally does which is. somehow with. very few basic. visual tools is able to communicate a. depth of emotion and that's fascinating.

but then again the humanoid form is. uh super compelling so like uh cosmo is. very distant from a humanoid form and. then the terminator has a humanoid form. you can imagine both of those actually. being in our society it's true and it's. interesting because um it was very. intentional to go really far away from. human form when you think about a. character like cosmo or like wall-e. where. you can completely rethink uh the.

constraints you put on that character um. what tools you leverage and then how you. actually create a personality uh and a. level of intelligence interactivity that. actually. matches the constraints that you're. under whether it's mechanical or sensors. or ai of the day this is why i almost. was always really surprised by how much. energy people put towards trying to. replicate human form in a robot because. you actually take on some pretty. significant um kind of constraints and. downsides when you do that.

um the first of which is obviously the. cost where it's just the the. articulation of a human body is just so. like magical um in both the precision as. well as the dimensionality that to. replicate that even in this quote. reasonably close form takes like a giant. amount of joints and actuators and uh in. motion and and you know sensors and. encoders and so forth but then um you're. almost like setting an expectation that. the closer you try to get to human form. the more you expect the strengths to. match and that's not the way ai works is.

there's places where you're way stronger. and there's places where you're weaker. and by moving away from human form you. can actually. change the rules and embrace your. strengths and bypass your weaknesses and. at the same time the human form. like has way too many degrees of freedom. to play with it's it's kind of. counterintuitive just as you're saying. but when you have fewer constraints. it's almost harder to master the the. communication of emotion like you see. this with cartoons like stick figures.

you can communicate quite a lot with. just. very minimal like two dots for eyes. and a line for for a smile i think like. you can almost communicate arbitrary. levels of emotion with just two dots and. a line yeah and like that's enough and. if you focus on just that. you can communicate the full range and. then you like if you do that then you. can focus on the actual magic. of of uh. human and. dot line interaction versus all the. engineering mess that's right like.

dimensionality voice all these sort of. things actually become a crutch where. you get lost in a search space almost um. and so some of the best animators that. we've worked with um they almost like. study when they come up uh you know kind. of in building their expertise by. forcing these um projects where all you. have is like a ball that can like kind. of jump and manipulate itself or like. really really like aggressive. constraints for your force to kind of. extract the deepest level of motion and.

so in a lot of ways um you know we. thought when we thought about cosmos. like you're right like our if we had to. like describe it in like one small. phrase it was bringing a pixar character. to life in the real world it's uh it's. what we were going for and um in a lot. of ways what was interesting is that. with like wall-e which we studied. incredibly deeply and in fact some of. our team were you know kind of had. worked previously at um at pixar and on. that project um they intentionally. constrained wall-e as well even though. in an animated film you could do. whatever you wanted to because it forced.

you to like. really saturate the smaller amount of. dimensions but uh you sometimes end up. getting a far more beautiful output um. because you're pushing at the extremes. of this emotional space in a way that. you just wouldn't because you get lost. in a surface area if you have like. something that is just infinitely. articulable so if we backtrack a little. bit and uh you thought of cosmo in 2011. and 2013 actually uh designed and built. it what is anki what is cosmo.

i guess who is cosmo and. uh what was the vision behind this. incredible little robot we started. uh anki back in. like while we were still in graduate. school so myself and my two co-founders. we were phd students uh in the robotics. institute at carnegie mellon um and so. we were uh studying robotics ai machine. learning kind of different you know. different uh uh areas one of my. co-founders working on walking robots uh. you know for a period of time and so we. all had a um a bit of a.

really deep kind of a deeper passion for. applications of robotics and ai where um. there's like a spectrum where there's. people that get like really fascinated. by the theory of ai and machine learning. robotics where um whether it gets. applied in the near future or not is. less of a kind of factor on them but. they love the pursuit of like the. challenge and that's necessary and. there's a lot of incredible. breakthroughs that happen there we're. probably closer to the other end of the. spectrum where we love the technology. and the um and all the evolution of it. but we were really driven by. applications like how can you really. reinvent experiences and functionality.

and build value that wouldn't have been. possible without these approaches and. and that's what drove us and we had a. kind of some experiences through. previous jobs and internships where we. like got to see the applied side of. robotics and at that time there was. actually relatively few applications of. robotics um that were outside of um. you know peer research or industrial. applications um military applications. and so forth there were very few outside. of it so maybe you know my robot was. like one exception and maybe there were.

a few others but for the most part there. weren't that many and so we got excited. about consumer applications of robotics. where you could leverage. way higher levels of intelligence. through software to create value and. experiences that were just not possible. in. in those fields today. and we saw. kind of a pretty wide range of. applications. that varied in the complexity of what it. would take to actually solve those and. what we wanted to do was to. commercialize this into a company but. actually. do a bottoms-up approach where we could.

have a huge impact in a space that was. ripe to have an impact at that time and. then build up off of that and move into. other areas and entertainment became the. place to start because um. you had relatively little innovation in. a toy space. an entertainment space you had these. really rich. experiences in video games and uh and. movies but there was like this chasm in. between and so we thought that. we could really reinvent that experience. and there was a. really fascinating transition. technically that was happening at the. time where the cost of components was.

plummeting because of the mobile phone. industry and then the smartphone. industry and so the cost of a. microcontroller of a camera of a motor. of memory of. microphones cameras was dropping by. orders of magnitude and then on top of. that with the iphone coming out in 2000. uh i think it was 2007 i believe. um. it started to become apparent within a. couple of years that this could become a. really incredible. interface device and the brain with much.

more computation behind a physical world. experience that wouldn't have been. possible previously. and so. um we really got excited about that and. how we push all the complexity from the. physical world into software by using. really inexpensive components but. putting huge amounts of complexity into. the ai side and so cosmo became our. second product and then the one that. we're probably most proud of the idea. there was to create a physical character. that had enough understanding and. awareness of the physical world around. it in the context that mattered to feel.

like. like he was alive um and. to be able to have these like emotional. kind connections and experiences with. people that you would typically only. find uh inside of a movie and the. motivation very much was was pixar like. we had an incredible uh respect and. appreciation for what they were able to. um build in this like really beautiful. fashion and film um but it was always. like a you know when it was virtual and. two it was like a story on rails that. had no interactivity to it it was very. fixed and it obviously had a magic to it.

but where you really start to hit a. different level of experiences when. you're actually able to physically. interact with that robot and then that. was your idea with anki like the first. product was the cars. so basically you take. you take a toy. you add intelligence into it in the same. way you would add intelligence into ai. systems within a video game but you're. not bringing into the physical space so. the idea is is really brilliant which is. you're basically bringing video games to. life exactly that's exactly right we.

literally use that exact same phrase. because in the case of drive this was a. parallel of the racing genre and the. goal was to effectively have a physical. racing experience but have a virtual. state at all times that matches what's. happening in the physical world and then. you can have a video game off of that. and you can have uh different characters. different traits for your. the cars. weapons and interactions and special. abilities and all these sort of things. that you think of virtually but then you. can have it physically and um one of the.

things that we were like really. surprised by that really stood out and. immediately led us to really like kind. of accelerate the path towards um cosmo. is that things that feel like they're. really constrained and simple in the. physical world they have an amplified. impact on people where the exact same. experience virtually would not have. anywhere near the impact but seeing it. physically really stood out and so. effectively we've with with drive we. were creating a video game engine for. the physical world um and then with. cosmo we expanded that video game engine. to create a character and and.

kind of an animation and interaction. engine. on top of it that allowed us to start to. create these much more rich experiences. and a lot of those elements were uh. almost like a proving ground for what. would human robot interaction feel like. in a domain it's much more forgiving. where you can make mistakes in a game. it's okay if like uh if you know car. goes off the track or if if cosmo makes. a mistake um and what's funny is. actually we're so worried about that.

in reality we realized very quickly that. those mistakes can be endearing and if. you make a mistake as long as you. realize you make a mistake and have the. right emotional reaction to it it builds. even more empathy with the character. that's brilliant exactly so when uh the. the thing you're optimizing for is fun. you have so much more freedom to fail. to explore. and and also in the toy space like all. this is really brilliant like i got to. ask you backtrack. it seems for a roboticist. to take us jump. in into the direction of fun.

is a brilliant move because when you. have the freedom to explore to design. all those kinds of things. and you can also build cheap robots. like you don't have to like if you're. not chasing perfection. and like. toys it's understood that you can go. cheaper which means in robot it's still. expensive but it's actually affordable. by a large number of people so it's a. really brilliant space to explore yeah. that's right it's uh and in fact we. realized pretty quickly that like. perfection is actually not fun yeah.

because like in a traditional robotic. roboticist sense the first kind of path. planner and uh this is the you know the. part that i worked worked on out of the. gate was like a lot of the kind of ai. systems where you have these you know. vehicles and. you know cars racing kind of making. optimal maneuvers to try to kind of get. ahead and you realize very quickly that. like that's actually not fun because you. want the like. chaos from mistakes and the. and so you start to kind of. intentionally almost add noise to the. system uh in order to kind of create. more of a realism in the exact same way. the human player might start really.

ineffective and inefficient and then. start to kind of increase their quality. bar as they. as they progress and. there is a really really aggressive. constraint that's forced on you by. being a consumer product where the price. point matters a ton particularly in like. kind of an entertainment where um. you know you you can't make a thousand. dollar product unless you're going to. meet the qua like the expectations of a. thousand dollar product and so um in. order to make this work like your cost. of goods had to be like like you know. well under a hundred dollars uh uh in.

the case of cosmo we got it under fifty. dollars end-to-end fully packaged and. delivered and it was under two hundred. dollars. it cost the retail yeah. so uh okay if we sit down like at this. early stages. if you go back to that. and you're sitting down and thinking. about what kosovo looks like from a. design perspective and from a cost. perspective i imagine that was part of. the conversation. first of all what came first did you. have a cost in mind is there a target. you're trying to chase.

did you have a vision in mind like size. did you have because there's a lot of. unique qualities to cosmos so for people. who don't know they should definitely. check it out. there's a display there's eyes on the. little display and those eyes can it's. pretty uh low resolution eyes right but. they they still able to convey a lot of. emotion and there's this arm. like that out lift sort of lifts stuff. but there's something about arm movement. that adds even more kind of depth. it's like uh the face communicates.

emotion and sadness and disappointment. and happiness and then the arms. kind of communicates i'm trying here. yeah i'm doing my best. exactly so it's um. uh it's interesting because like um. all of cosmo's only four degrees of. freedom and two of them are the two. treads which is for basic movement and. so you literally have only. a head that goes up and down a lift that. goes up and down and then your two. wheels uh and you have sound uh and a.

screen yeah and a low resolution screen. and with that it's actually pretty. incredible what you can uh what you can. come up with where like you said it's a. uh it's a really interesting give and. take because there's a lot of ideas far. beyond that obviously as you can imagine. where like you said how big is it how. much degrees of freedom what does it. look like um uh what does he sound like. how does he communicate it's it's a. formula that actually scales way beyond. entertainment this is the formula for. human. kind of robot interface more generally. is you almost have this triangle between.

um the physical aspects of it the. mechanics the industrial design what's. mass producible the cost constraints and. so forth. you have the ai side of. how do you understand the world around. you interact intelligently with it. execute what you want to execute so. perceive the environment. make intelligent decisions and. and move forward and then you have the. character side of it. um. most uh companies have done anything in. human robot interaction really uh missed. the mark or under invest in the. character side of it um they over invest.

in the mechanical side of it uh. you know and then varied results on the. ai side of it and so the thinking is. that you put more mechanical flexibility. into it you're gonna do better um you. don't necessarily you actually create a. much higher bar uh for a high roi. because now your price point goes up. your expectations go up and if the ai. can't meet it or the overall experience. isn't there you missed the mark. um so who like how did you through those. conversations get the cost down so much. and make it made it so simple like that.

there's a big theme here because you. come from the mecca of robotics which is. carnegie mellon university robotics like. for all the people i've interacted with. that come from there or just from. you know the world experts at robotics. they don't. they would never build something like. cosmo yeah and so where did that come. from so the simplicity it came from this. combination of a team that we had it was. it was quite cool because like we and by. the way you ask anybody that's like. experienced in the like kind of you know.

toy entertainment space you'll never. sell a product over 99 um that was. fundamentally false and we believed it. to be false it was because experience. had to kind of you know meet the mark. and so we pushed past that amount but. there was a pressure where the higher. you go the more seasonal you become and. the tougher it becomes and so on the. cost side we very quickly partnered up. with some previous contacts that we. worked with where just as an example one. our head of mechanical engineering um. was one of the earliest heads of. engineering at logitech and has a.

billion units of consumer products and. circulation that he's worked on yeah so. like crazy low cost high volume consumer. product experience with a really great. mechanical engineering team and just a. very practical mindset where we were not. going to compromise on feasibility in. the market in order to chase something. that would be enabler and we pushed a. huge amount of expectations onto the. software team where yes we're going to. use cheap. noisy motors and sensors but we're gonna. fix it in the um on the software side. then we found on the design and. character side there was a faction that.

was more from like a game design. background that thought that it should. be very games driven cosmo where you. create a whole bunch of games. experiences and it's all about like game. mechanics and then there was um a. faction which my my co-founder and i the. most involved in this like really. believed in which was character driven. and the argument is that you will never. compete with what you can do virtually. from a game standpoint but you actually. on the character side put this into your. wheelhouse and put it more towards your. advantage. because a physical character has a. massively.

higher impact uh physically than. virtually this is okay i can't just. pause on that because this is so. brilliant when i uh for people who don't. know cosmo. plays games with you. but there's also a depth of character. and i actually when i was. you know playing with it. i wondered. exactly what is the compelling aspect of. this because to me obviously i'm i'm. biased but to me the character i get. what i enjoyed most honestly. or what. got me to return to it is the character.

that's right but that's that's a. fascinating discussion of uh. you're right ultimately. you cannot compete. on the. quality of the gaming experience too. restrictive the physical world is just. too restrictive and uh you don't have a. graphics engine it's like all this but. on the character side. we uh and clearly we moved in that. direction is like kind of the the the. winning path and um. we partnered up with this uh really we. immediately like went towards pixar and.

carlos bana he was um one of like had. been in pixar for nine years he'd worked. on tons of the movies including wally. and others and just immediately kind of. spoke the language and just clicked on. how you think about that like kind of. magic and drive and then he we built out. a team uh. you know with him as like a really kind. of prominent kind of driver of this with. different types of backgrounds and. animators and character developers where. um we put these constraints on the team. but then got them to really try to. create magic despite that and we.

converged on this system that was at the. overlap of character and the character. ai. that where if you imagine the. dimensionality of emotions happy. sad angry surprised confused uh um. scared like you think of these extreme. emotions. we almost like kind of put this. challenge to kind of populate this. library of responses on how do you show. the extreme. response that like goes to the extreme. spectrum on angry or frustrated or.

whatever and and so that gave us a lot. of intuition and learnings and um and. then we started parameterizing them. where it wasn't just a fixed recording. but they were parameterized and had. randomness to them where you could have. infinite permutations of happy and. surprised and so forth. and then we had a behavioral engine that. took the context from the real world. and would interpret it and then create. kind of probability mappings on what. sort of responses you would have that. actually made sense and so if cosmo saw. you for the first time in a day um he'd.

be really surprised and happy in the. same way that the first time you walk in. and like your toddler sees you they're. so happy but they're not gonna be that. happy for the entirety of your next two. hours but like you have this like spike. in response or if you leave him alone. for too long he gets bored and starts. causing trouble and like nudging things. off the table um or if you beat him in a. game um the most enjoyable emotions are. him getting frustrated and grumpy to a. point where our testers and our. customers would be like. i had to let him win because i don't. want him to be upset and.

so you start to like create this. feedback loop where you see how powerful. those emotions are and just to give you. an example something as simple as eye. contact um you don't think about it in a. movie just like it kind of happens like. you know camera angles and so forth um. but that's not really a prominent source. of interaction. what happens when a physical character. like cosmo when he makes eye contact. with you um. it built universal kind of connection. kids all the way through adults um and. it was truly universal it was not like.

people stopped caring after 10 12 years. old. and so. we started doing experiments and we. found something as simple as. increasing the amount of eye contact. like the amount of times in a minute. that he'll look over for your approval. to like kind of make eye contact just by. i think doubling it we increase the play. time engagement by 40. like you see these sort of like kind of. interactions where you build that. empathy and and so we studied pets we. studied um virtual characters there's. like a lot of times actually dogs are.

one of the perfect most perfect uh um. influencers behind these sort of. interactions and what we realized is. that the games were not there to. entertain you the games were to create. context to bring out the character and. if you think about the types of games. that you know that you played they're. relatively simple but they were always. once to create scenarios of either. tension or winning or losing or surprise. or whatever the case might be and they. were purely there to just like create. context to where an emotion could feel. intelligent and not random. and in the end it was all about the. character.

so yeah there's so many. elements to play with here so you said. dogs what lessons do we draw from cats. who don't seem to give a damn about you. is that just another character is this. another it's just another character and. so you you could almost like in early. aspirations we thought it would be. really incredible if you had a diversity. of characters where you almost help. encourage which direction it goes just. like in a role-playing game um and you. had uh like think of like the you know. seven dwarfs sort of and uh um and.

initially we even thought that it would. be amazing if like the other like. you know like their characters actually. help them be have strengths and. weaknesses and some you know like. whatever they end up doing like some are. scared some are. you know. arrogant some are uh. you know super warm and like kind of. friendly and in the end we focused on. one because it made it very clear that. hey we got to build out enough depth. here because you're. kind of trying to expand. it's almost like how long can you. maintain a fiction that this character. is alive um to where the person's.

explorations don't hit a boundary um. which happens almost immediately with. with typical toys um and you know even. with video games uh how long can we. create that immersive experience to. where you expand the boundary and one of. the things we realized is that you're um. just way more forgiving when something. has a personality and it's physical. that is the key. that unlocks. uh robotics interacting you know in the. physical world more generally is that. that uh the. when you have a when you don't have a.

personality and you make a mistake as a. robot the stupid robot made a mistake. why is it not perfect when you have a. character and you make a mistake you. have empathy and it becomes endearing. and you're way more forgiving and that. was the key that was like i think goes. far far beyond entertainment it actually. builds the depth of the personality the. mistakes so let me ask the the movie her. question then. how and so. cosmos seem feels like the early days of. something that will obviously be. prevalent throughout society at a scale.

that. we cannot even imagine. my sense is. it seems obvious. that these kinds of characters will. permeate society and they will be. friends with them we'll be interacting. with them in different ways the in the. way we i mean you don't think of it this. way but when you play video games. they're kind they're often cold and. impersonal. but but even then. uh you think about role-playing games. you become friends with certain.

characters in that game they're they. don't remember much about you they. they're they're just telling a story. it's exactly what you're saying they. they exist in that virtual world but if. they acknowledge that you exist in this. physical world if the characters in the. game remember that you exist that you. like for me like lex they understand. that i'm a human being who has like. hopes and dreams and so on it seems like. there's going to be a like.

billions. if not trillions of cosmos in the world. so if we look at that future. there are several questions to ask how. intelligent does that future cosmo need. to be. to create. fulfilling. relationships like friendships yeah it's. a great question and and part of it was. a recognition it's going to take time to. get there because it has to be a lot. more intelligent um because what's good. enough to. be a magical experience for. uh you know an eight-year-old um it's a.

higher bar to do that be a complaint. like a pet in the home or to help with. functional interface in an office. environment or in a home or uh and so. forth and so and the idea was that you. build on that and you kind of get there. and as technology becomes more prevalent. and less expensive and so forth you can. start to kind of work up to it um but. you know you're absolutely right at the. end of the day um we almost equated it. to how uh the touchscreen created like. this really novel interface to you know. physical kind of devices like this. this is the extension of it where you.

have. much richer physical interaction in the. real world this is this is the enabler. for it um and it shows itself in a few. kind of really obvious places so just. take something as simple as a voice. assistant um you will never most people. will never tolerate uh an alexa or a. google home just starting a conversation. um proactively uh when you weren't kind. of expecting it because it it feels. weird it's like you were listening and. like and then now you're kind of it. feels intrusive but if you had a. character um like a cat that touches you.

and gets your attention or toddler like. you never think twice about it what we. found really kind of immediately is that. um these types of characters like cosmo. and they would like roam around and kind. of get your attention and we had a. future version it was always on kind of. called vector people were way more. forgiving. and so you could initiate interaction in. a way that is not acceptable for. for machines and in general um you know. there's a lot of ways to customize it. but it makes people who are skeptical of. technology much more comfortable with it. there was like there were a couple of.

really really. prominent examples of this so when we. launched in europe and so we were in um. uh i think like a dozen countries if i. remember correctly but like we were we. went pretty aggressively in launching in. um germany and france and uh and uk and. we were very worried in europe because. there's obviously like a really a. socially higher bar for privacy and you. know security where you you've heard. about how many companies have had. troubles on uh uh that might things that. might have been okay in the u.s but like. are just not okay in germany and france. in particular um and so we were worried.

about this because you have um you know. cosmo who's um uh you know in our future. product veteran like where you have. cameras you have microphones it's kind. of connected and like you're playing. with kids and like in these experiences. and you're like this is like ripe to be. like a nightmare if you're not careful. yes um and. uh and the journalists are like. notoriously like really really tough on. on these sort of things um. we were shocked and we prepared so much. for what we would have to encounter we.

were shocked in that. not once from any journalists or. customer do we have any complaints. beyond like a really casual kind of. question. and it was because. of the character where um. when it conversation came up it was. almost like well of course he has to see. in here how else is he going to be alive. and interacting with you. and it completely disarmed um this like. fear of technology that enabled this. interaction to be much more fluid and. again like entertainment was a proving. ground but that is like a you know.

there's like ingredients there that. carry over to a lot of other uh. elements down the road that's hilarious. that we're a lot less concerned about. privacy if the if the thing is value and. charisma. i mean that's true for all of women to. human interaction too it's an. understanding of intent where like well. he's looking at me he can see me if he's. not looking at me he can't see me right. so it's almost like uh. um you're communicating intent and with. that intent. people are like kind of kind of more. understanding and calmer and it's a it's.

interesting we just it was just the. earliest kind of version of starting an. experiment with this but um it wasn't. enabler and um and then and then you. have like completely different. dimensions where like you know kids with. autism had like an incredible connection. with cosmo that just went beyond. anything we'd ever seen and we have like. these just letters that we would receive. from parents and we had some research. projects kind of going on with some. universities on studying this but um. there are like there's an interesting. dimension there that got unlocked that. just hadn't existed before um that has. these really interesting kind of links.

into society and. and a potential building block of future. experiences so. if you look out into the future do you. think we will have. beyond a particular game. you know a companion. like uh like her. like the movie her. or like a cosmo that's. kind of asks you how your day went too. right. you know like a friend. how many years away from that do you. think we are what's your intuition good.

question so. i think the idea of a different type of. character like more closer to like kind. of a pet style companionship it will. come way faster um. and there's a few reasons one is like to. to do something like in her that's like. effectively almost general ai and the. bar is so high that if you miss it by. bit you hit the uncanny valley where it. just becomes creepy and like and not um. not appealing um because the closer you. try to get to a human in form and. interface and. voice the harder it becomes whereas.

you have way more flexibility on. still landing a really great experience. if you embrace the idea of a character. and that's why um one of the other. reasons why we didn't have a voice uh. and also why like a lot of video game. characters uh like. sims for example does not have a voice. when you uh when you think about it it. was it wasn't just a cost savings like. for them it was actually for all of. these purposes it was because when you. have a voice you immediately narrow down. the appeal to some particular. demographic or age range or um kind of.

style or gender uh if you don't have a. voice people interpret what they want to. interpret. and an eight-year-old might get a very. different interpretation than a 40 year. old. but you create a dynamic range and so. you just you can lean into these. advantages much more um and something. that doesn't resemble a human and so. that'll come faster. i don't know when a human like that's. just uh still like ma just complete r d. at this point the the chat interfaces. are getting way more interesting and. richer but it's still a long way to go.

to kind of pass the test of. you know well let me like let's consider. like let me play devil's advocate. so google is a very large company that's. servicing. it's creating a very compelling product. that wants to provide a service a lot of. people but let's go outside of that you. said characters yeah it feels like and. you also said that it requires general. intelligence to be a successful. participant in a relationship which. could explain why i'm single this is.

very. but the i i honestly want to push back. on that a little bit. because i feel like is it possible. that if you're just good at playing a. character yeah you're in in a movie. there's a bunch of characters if you. just understand what creates compelling. characters. and then you you just are that character. and you exist in the world and other. people find you and they connect with. you just like you do when you talk to. somebody at a bar i like this character. this character is kind of shady i don't. like them you pick the ones that you.

like and you know maybe it's somebody. that's uh reminds you of your father or. mother i don't know what it is but the. the freudian thing but there's some kind. of connection that happens and that's. that that's the cosmo you connect to. that's the future cosmo you connect and. that's. so i guess the statement i'm trying to. make is it possible to achieve a depth. of friendship without solving. general intelligence i think so it's. about intelligent kind of constraints. right and just uh you set expectations.

and constraints such that in the space. that's left you can be successful and so. you can do that by having a very focused. domain that you can operate in for. example you're a customer support agent. for a particular product and you create. intelligence and a good interface around. that or uh you know kind of in the. personal companionship side you can't be. everything to. across the board. you you kind of solve those constraints. and i think uh i think it's possible my. my worry is like i. right now i don't see anybody that has. picked up on where kind of cosmo left.

off yes and is pushing on it in the same. way and so i don't know if it's a sort. of thing where similar to like how you. know in dot com there were all these. concepts that we considered like you. know that didn't work out or like failed. or like were too early or whatnot and. then 20 years later you have these like. incredible successes on almost the same. concept like it might be that sort of. thing where like there's another pass at. it that happens in five years or in 10. years but um it does feel like that. appreciation of that like that this the. three like it's duel if you will between.

like you know the hardware the ai and. the character um that balance it's hard. to. i'm not aware of of any pro anywhere. right now where like that same kind of. aggressive drive with the value on the. character is uh is happening and so to. me. just a prediction exactly as you said. something that looks awfully a lot like. cosmo not in the actual physical form. but in the three-legged stool. something like that in some number of. years would be a trillion dollar company. i don't understand like it's obvious to.

me yeah that. like. character not just as robotic companions. but in all our computers they'll be. there it's like uh. clippy was like. two legs of that stool or something like. that yeah i mean that those are all. different attempts and. what's really confusing to me is they. they're born. these attempts and they they everybody. gets excited and for some reason they. die and then nobody else tries to pick.

it up and then maybe a few years later. a crazy guy like you comes comes around. with just enough brilliance and vision. to create this thing and it's born a lot. of people love it a lot of people get. excited but maybe the timing is not. right yet. and then and then when the timing is. right it just blows up and it just keeps. blowing up more and more until it just. blows up and i guess everything in the. full span of human civilization. collapses eventually. and that wouldn't surprise me at all and.

like what's gonna be different in. another five years or ten years what not. physical component costs will continue. to come down uh in price and you know. mobile devices and computations going to. become more and more prevalent as well. as cloud as a big tool uh to offload. cost um ai is going to be a massive. transformation compared to what we dealt. with uh where um everything from voice. understanding to um uh to just. you know kind of a broader contextual uh. understanding and mapping of.

of semantics and uh understanding scenes. and so forth and then the character side. will continue to kind of you know. progress as well because that magic does. exist it just exists in different forms. and you have just the brilliance of uh. that's happening in animation and you. know these other areas where um that is. that was a big unlock in um you know in. film obviously uh and so i think yeah. the pieces can reconnect and the. building blocks are actually gonna be. way more impressive than they were five. years ago so. so in 2019.

uh anki the company that created cosmo. the company that you started had to shut. down. how did you feel at that time. yeah. it was tough uh that was a really. emotional stretch and it was really. tough year. like about a year ahead of that was. actually a pretty brutal stretch because. we were um. kind of light life or death on many many. moments um just navigating. these insane kind of just ups and downs.

and um barriers and the thing that made. it. like um. like just rewinding a tiny bit like what. you know what ended up being really. challenging about it as a business where. is um. from a commercial standpoint and. customer reception standpoint there's a. lot of things you could point to that. were like you know pretty big successes. sold millions of units uh like you got. to like pretty serious revenue like kind. of close to 100 million annual revenue. um uh number one kind of product in kind. of various categories.

but it was pretty expensive it ended up. being very seasonal where something like. 85 percent of our volume was in q4. because it was a you know a present and. and it was expensive to market it and. explain it and so forth um and even. though though the volume was like really. sizeable and like the reviews were. really fantastic um. forecasting and planning for it and. managing the cash operations was just. brutal like it was absolutely brutal you. don't think about this when you're. starting a company or when you have a. few million in. you know in revenue because it's just.

your biggest costs are kind of just your. head count and operations and. everything's ahead of you but we got to. a point where um. you know you if you look at the entire. year you have to. operate your company pay all you know. the people and so forth you have to pay. for the manufacturing the marketing and. everything else. to do your sales in mostly november. december and then get paid in december. january by retailers and those swings. were pretty um were really rough um and. just made it like so difficult because. the more successfully became the more.

wild those swings became. because you'd have to like spend. you know tens of millions of dollars on. inventory tens of millions of dollars on. marketing and tens of millions of. dollars on payroll and everything else. and then there's the bigger dip and then. you're waiting for the 204 yeah and it's. not a business that like is recurring. kind of month-to-month and predictable. and it's just and then you're walking in. your forecast in july um you know maybe. august. if you're lucky um. and uh and it's also like very hit. driven and seasonal where like you don't. have the sort of continued uh kind of.

slow growth like you do in some other uh. consumer electronics industries and so. before then like hardware kind of like. went out of favor too and so you had. fitbit and gopro dropped from 10 billion. revenue to 1 billion revenue and. hardware companies are getting valued at. like 1x revenue oftentimes um which is. tough right and so. we effectively kind of got caught in the. middle where we were trying to. quickly evolve out of entertainment and. move into some other categories but you. can't let go of that business because. like that's what you're valued on that's. what you're raising money on um but.

there's no path to prop kind of pure. profitability just there because it was. you know such you know uh specific type. of price points and so forth and so. um. we tried really hard to make that. transition and um yeah we had a. financing round that fell apart at the. last second and effectively there was. just no path to kind of get through that. and get to the next kind of like holiday. season and so we ended up um. uh selling some of the assets and kind. of winding down the company it was uh it. was brutal like we i was very. transparent with the company like in the.

the team while we were going through it. where actually despite how challenging. that period was very few people left i. mean like people loved the vision the. team the culture of the like kind of. chemistry and kind of what we were doing. there was just a huge amount of pride. there and we wanted to see it through. and we felt like we had a shot to kind. of get through these checkpoints um. we ended up uh and i mean by brutal i. mean like literally like days of cash. like three four different times uh. runway like in the year you know kind of. before it um where you're like.

playing games of chicken on negotiating. credit line timelines and. like repayment. terms and how to get like a bridge loan. from an investor it's just like. level of stress that like is as hard as. things might be anywhere else like. you'll never come you know come close to. that where you feel that like. responsibility for you know 200 plus. people right um. and so we were very transparent during. our fundraise on who we're talking to. the challenges um that we have. how it's going and when things are going. well when things were tough um and so it.

wasn't a complete shock when it happened. but it was just very emotional where. like i you know like. you know when we announced it finally. that like um you know we you know. basically we're just like watching kind. of like you know the runway and trying. to kind of time it and when we realized. that like we didn't have any more outs. we wanted to like kind of wind it down. make sure that it was like clean and you. know we could like kind of take care of. people the best we could but yeah like. broke down crying at all you know hands. and somebody else had to step in for a. bit and like it was just very very. emotional but the beautiful part is like. afterwards like everybody stayed at the.

office to like two three in the morning. just like drinking and hanging out and. telling stories and celebrating and it. was just like. one of the best uh for many people was. like the best kind of work experience. that they had and there was a lot of. pride in what we did and there wasn't. anything obvious we could point to that. like hey if only we had done that. different things would have been. completely different it was just like. the physics didn't line up uh and uh. um but the experience was pretty uh. incredible but it was hard like it was. uh it had this feeling that there was.

this like incredible beauty in both the. technology and products and the team. that um. uh. you know there's there's a lot there. that like in the you know right context. could have been. uh pretty incredible but it was um. emotional just. yeah just thinking i mean just looking. at this company like you said the. product and technology but the vision. the implementation you got the cost down. very low. yeah and the compelling the nature of.

the product was great so many robotics. companies failed at this at they. the robot was too expensive it didn't. have the personality it didn't really. provide any value like a sufficient. value to justify the price so like you. succeeded where basically every single. other robotics company or most of them. that are like going the category of. social robotics have kind of failed. and. i mean it's uh it's quite tragic i. remember uh.

reading that i'm not sure if i talked to. you before that happened or not. but i remember you know i'm distant from. this. i remember being heartbroken reading. that. because like. if. if cosmo's not going to succeed. what is going to succeed because that to. me was incredible. like. it was an incredible idea. cost is down the minimum the. the. it's just like the most minimal design.

in physical form that you could do it's. really compelling the balance of games. so it's a it's a fun toy it's a great. gift. for all kinds of age groups right it's. just it's compelling in every single way. and it seemed like uh it was a huge. success and it it failing was. i don't know there was heartbreak on. many levels for me just as an external. observer. is i was thinking how hard is it to run. a business.

that's that's what i was thinking like. if this failed this must have failed. because uh it's obviously not like. yeah it's b it's business yeah maybe. it's some aspect of the manufacturing. and so on but i'm now realizing it's. also not just that it's. yeah. sales marketing also it's everything. right like how do you explain something. that's like a new category to people. that like how all these previous. positions and so like uh you know it it. had some of the hardest elements of. if you were to pick a business it had. some of the hardest uh um customer.

dynamics because like to sell a 150. product you got to convince both the. child to want it and the parents to. agree that it's valuable so you're. having like this dual prong marketing. challenge you have manufacturing you. have like really high precision on the. components that you need you have the ai. challenges so there were a lot of tough. elements but is this feeling where like. just really great alignment of unique. strength across kind of like all these. different areas just an incredible like. you know kind of character and animation. team between this like carlos and. there's like a character director day.

that came on board and like you know. really great people there the ai side. the um uh. the manufacturing the you know where um. like never missing a launch right and. actually you know he kind of hitting. that quality was um yeah it was it was. heartbreaking but uh. here's one neat thing is like we we had. so much like fan mail from kind of kids. parents like i actually like there was a. bunch they collected in the end yeah. that um i actually saved and like i. never it was too emotional to open it. and i still haven't opened it um and so.

i actually have this giant envelope of. like a stack this much of like letters. from you know kids and families just. like every you know perpetration. permutation you can imagine and so. planning to kind of i don't know maybe. like a five year you know five year. eight some year reunion just inviting. everybody over and we'll just like kind. of dig into it and um kind of bring back. some memories but um you know good. impact and uh um well i i think there. will be companies uh maybe waymo and. google will be somehow involved that. will carry this flag forward and will uh.

will make you proud whether you're. involved or not i think this is one of. the greatest robotics companies in the. history of robotics so you should be. proud it's still tragic to know that. you know because you read all the. stories of apple and and. let's see spacex and like companies that. were just on the verge of failure. several times through that story and. they just it's almost like a roll of the. diet they succeeded and here's the role.

of the dice that just happened to go. and that's the appreciation that like. when you really like talk to a lot of. the. founders like everybody goes through. those moments and sometimes it really is. a matter of like you know timing a. little bit of luck like some things are. just out of your control and um. uh and you you get a much deeper. appreciation for um just the. dimensionality of of that challenge but. um the great thing is that like a lot of. the team actually like stayed together. and so um they were actually a couple of.

companies that we we kind of kept big. chunks of the team together and we. actually kind of helped align this uh um. you know to help people out as well um. and one of them was waymo where uh. a majority of the ai and robotics team. actually had the exact background uh. that you would look for in like kind of. a b space it was a space that a lot of. us like you know were you know worked on. in grad school were always passionate. about and ended up uh you know maybe the. time you know. serendipitous timings from another. perspective where like uh um kind of.

landed in a really unique um. circumstance it's actually been quite. exciting too. so it's interesting to ask you just your. thoughts uh cosmo still lives on. under dream labs i think. is that. are you tracking the progress there or. is it too much pain. is it are you is that something that. you're excited to see where that goes. so keeping an eye on it of course just. out of your curiosity and obviously just. kind of care for product line i think um.

it's deceptive how complex it is to. manufacture and evolve that product line. um and the amount of. experiences that are required to. complete the picture and be able to move. that forward and i think that's going to. make it pretty hard to do something. really substantial with it it would be. cool if like even the product in the way. it was was able to be manufactured yes. again that would be. yeah which would be neat um but uh it's. i think it was it's deceptive how tricky. that is on like everything from the.

quality control the details and um and. then like technology changes that forces. you to rick. reinvent and update certain things um so. uh i haven't been super close to it but. just kind of keeping an eye on it yeah. it's really interesting how. it's deceptively difficult just as. you're saying for example. those same folks uh and i've spoken with. them they're. they partnered up with rick and morty uh. creators to uh to do the butter robot.

yes i love the idea i just recently. i've kind of half-assed watch rick and. morty previously but now i just watched. like the first season it's such a. brilliant show i i like. i did not understand how brilliant that. show is and obviously i think in season. one is where the butter robot comes. along for just a few minutes or whatever. but i just fell in love with the butter. robot the sort of the. that particular character just like you. said there's characters you can create. personalities you can create and that.

particular. a robot. who's doing a particular task. realizes. you know. this like realizes that's the. existential question this the myth of. sisyphus question that uh camus writes. about it's like is this all there is. because he moves butter. but you know. that realization that's a that's a. beautiful little realization for a robot. that my purpose is very. limited with this particular task it's. abuse it's humor of course it's darkness.

it's a beautiful mix but so they want to. release that butter robot. but something tells me. that to do the same depth of personality. as cosmo had the same richness it would. be on the manufacturing on the ai. on the storytelling on the design it's. going to be very very difficult it could. be a cool sort of uh toy. for rick and morty fans. but to create the same depth of.

existential angst yeah that the butter. robot symbolizes is is really. that's the brave effort you succeeded at. with cosmo but it's not easy it's really. studies and. you can fail on almost any one of the. kind of dimensions and like uh and yeah. it takes you know. yeah unique convergence of a lot of. different skill sets to try to pull that. off yeah. on this topic let me ask you for some. advice. because uh as i've been watching rick.

and morty i i told myself i have to. build the butter robot just as a hobby. project and so uh i got a nice platform. for it with treads and and there's a. camera that moves up and down and so on. um i'll probably paint it. but. the question i'd like to ask there's. obvious technical questions i'm fine. with communication the personality. storytelling all those kinds of things. i think i understand the process of that. but how do you know.

when you got it right. so with with cosmo how did you know. this is great like or um something is. off like yeah is this brainstorming with. the team. do you know it when you see it is it. like. love at first sight it's like this is. right or like i guess if we think of it. as an optimization space. is there uncanny valley we're like. that's not right or this is right or are. a lot of characters right yeah. we stayed away from uncanny valley just.

by having such a different what like. mapping where it didn't try to look like. a dog or a human or anything like that. and so uh. you avoided having like a weird pseudo. similarity but not quite hitting the. mark um but you could like just fall. flat where just like a personality or a. you know character emotion just didn't. feel right and so it actually mirrored. very closely to kind of the iterations. that a character director of pixar would. have where you're. running through it and you can virtually. kind of like see what it'll look like we.

we created a plug-in to where we. actually used like like maya the sim you. know the animation tools and then we. created a plug-in that. perfectly. matched it uh to the physical one and so. you could like test it out virtually and. then push a button and see it physically. play out and there's like subtle. differences and so you want to like make. sure that that feedback loop is super. easy to be able to test it live. um and then sometimes like. you would just feel it that it's right. and intuitively no and then you'd also. do we did user testing but. it was very very often that like the.

into like if we found it magical it. would scale and be magical uh more. broadly there were not too many cases. where like. like we were pretty decent about not. like getting to it you know geeking out. or getting too attached to something. that was super unique to us um but. trying to kind of like you know put a. customer hat on and does it truly kind. of feel magical and so in a lot of ways. we just give a lot of um. autonomy to the. character team to really think about the. you know character board and mood boards.

and storyboards and like what's the. background of this character and how. would they react um and they went. through a process that's actually pretty. familiar but now had to operate under. these unique constraints um but. the moment where it felt right um. kind of took a fairly similar journey. than like a as a character in an. animated film actually it's quite cool. well the the thing that's really. important to me and i wonder if it's. possible well i hope it's possible. pretty sure it's possible is. for me. even though i know how it works. to make sure there's sufficient.

randomness in the process yeah. probably because it would be machine. learning based. that i'm surprised. that i don't i'm surprised by certain. reactions i'm surprised by certain. communication maybe that's in a form of. a question. um were you surprised by certain things. cosmo did like certain interactions. yeah we made it intentionally like. uh so that there would be some surprise. then like a decent amount of variability. in how.

he'd respond in certain circumstances. and so in the end like it's um. this is this isn't general ai this is a. giant like spectrum and library of like. parametrized kind of emotional responses. and an emotional engine that would like. kind of map. your current state of the game your. emotions the world the people are. playing with you all so forth to what's. happening um but we could make it feel. spontaneous by creating enough. diversity uh and randomness uh but still. within the bounds of what felt felt like.

very realistic um to make that work and. then what was really neat is that we. could get statistics on how much of that. space we were saturating um and then add. more animations and more diversity in. the places that would get hit more often. so that you stay ahead of the um you. know the curve and maximize the uh the. chance that it it stays feeling alive um. and so but then when you like combine it. like. the permutations and kind of like the. combinations of emotions stitched. together sometimes surprised us because. you see them in isolation but when you. actually see them and you see them live.

you know relative to some event that. happened in the game or whatnot like it. was kind of cool to see the combination. of the two and um uh and not too. different in other robotics applications. where like you get you get so used to. thinking about like the modules of a. system and how things progress through a. tech stack. that the real magic is when all the. pieces come together and you start. getting. the right emergent behavior um in a way. that's easy to lose when you just kind. of go too deep into any one piece of it. yeah when the system is sufficiently. complex there is something like emergent. behavior and that's where the magic is.

you as a human being you can still. appreciate the beauty of that magic of. the fine at the system level first of. all thank you for humoring me on this uh. it's really really. uh fascinating i think a lot of people. would love this i i'd love to just one. last thing on the butter robot i promise. in terms of uh speech yeah. cosmo is able to communicate so much. with just movement and face. do you think speech. is too much of a degree of freedom.

like a speech a feature or a bug of uh. deep. uh interaction. emotional interaction yeah. for a product. it's too deep right now it's just not. real uh it would immediately break the. fiction because the state of the art is. just not good enough um and. that's on top of just narrowing down the. demographic where like the way you speak. to an adult versus a way speak to a. child is very different. yet a dog is able to appeal to everybody.

and so right now there is no speech. system that is like rich enough and and. subtly realistic enough to feel. appropriate um and so we very very. quickly kind of like moved away from it. now speech understanding is a different. matter where understanding intent that's. a really valuable input um. but. giving it back requires like a you know. way way higher bar given kind of where. today's. world is and so. that realization that you can do.

surprisingly much with. uh either no speech or kind of tonal. like the way you know wally r2d2 and. kind of other characters are able to um. it's quite powerful and it generalizes. um across cultures and across ages. really really well i think we're going. to be in that. world for a little while where it's. still very much an unsolved problem on. how to like make something it touches on. kenny valley thing so if you have legs. and you're a big humanoid looking thing. you have very different expectations and. a much narrower degree of what's going. to be acceptable by society than if.

you're a you know robot like uh like. cosmo or wall and you can or some other. form where you can kind of like reinvent. the character speech has that same. property where speech is so well. understood um in terms of expectations. by humans that you have far less. flexibility on how to deviate from that. and lean into your strengths and avoid. weaknesses but i wonder if there is. obviously there's certain kinds of. speech that. activates the uncanny valley and breaks.

the illusion faster so. i guess my intuition is. we will solve. certain we would be able to. create some speech-based personalities. sooner than others so for example i. could i could think of. a robot that doesn't know english and is. learning english right yeah those kinds. of personalities where you're like uh. you're intentionally kind of like. getting a toddler level of uh speech so. that's exactly right so you can have. like. uh.

tie it into the experience where uh it. is a more limited character or you. embrace the lack of emotions as part or. the lack of sorry dynamic range in the. speech kind of capabilities emotions as. like part of the character itself and. you've seen that in like kind of. fictional characters as well yeah um but. that's why this podcast works and. yeah like you kind of had that with like. um i don't know i guess like you know. data and some of the other yeah. like um but yeah so you have to and that. becomes a constraint that lets you.

meet the bar um see i i honestly think. like also if you add uh drunk. and angry. that gives you more constraints that. allow you to be. dumber from an nlp perspective like. there's certain aspects so if you modify. human behavior like let's just so forget. the sort of artificial thing where you. don't know english toddler thing. we if you just look at the full range of. humans. i think we there's certain.

situations where we put up. with uh like lower level of intelligence. in our communication like if somebody's. drunk we understand this issue that. they're probably under the influence. like we understand that they're not. going to be making any sense anger is. another one like that i'm sure there's a. lot of other kind of situation. yeah maybe uh yeah again language loss. in translation. that kind of stuff that i think if you. if you play with that.

uh what is it the ukrainian boy that. passed the touring test you know play. with those ideas i think that's really. interesting and then you can create. compelling characters but you're right. that's a dangerous sort of road to walk. because uh you're adding degrees of. freedom that can get you in trouble yeah. and that's why like you have these um. big pushes that like for most of the. last decade plus like where you'd have. like. full like. human replicas of robots really being. down to like skin and like kind of in. some places um.

my personal feeling is like man like. that's not the direction that's most. fruitful right now um beautiful art yeah. it's not in terms of a. uh rich deep fulfilling experience yeah. you're right yeah and the way creating a. minefield of potential places to feel. off uh. and then and then you're sidestepping. where like the biggest kind of. functional ai challenges are to actually. have. you know kind of like really rich. productivity that actually kind of. justifies a you know kind of the higher. price points and that's that's part of.

the challenges like yeah like robots are. going to get to like thousands of. dollars tens of thousands of dollars and. so forth but you can imagine what sort. of expectation of value that comes with. it um and so that's where. you want to be able to invest the. the the time and uh and depth and so. going down the full human replica route. um. creates a gigantic uh. uh. distraction and really really high bar. that can end up sucking up so much of. your resources.

so it's weird to say but you happen to. be one of the greatest at this point. roboticist ever because you. created this little guy you were part. obviously of a great team that created. the the little guy with a deep. personality. and they're now. switching to. an entirely well maybe not entirely but. a different. fascinating impactful robotics problem. which is autonomous driving and more. specifically the biggest version of.

autonomous driving which is autonomous. trucking. so you are at waymo now can you give us. a big picture overview what is waymo. what is waymo driver what is waymo one. what is waymo via. can you give an overview of the company. and the vision behind the company for. sure waymo by the way it's just it's. been eye-opening on just how incredible. that that people and the talent is and. how in one company you almost have to. create i don't know 30 companies worth.

of like technology and capability to. like kind of solve the full spectrum of. it so um. yeah so i've been at weymouth since um. 2019 so about two and a half years so. waymo is uh focused on building what we. call a driver which is. creating the ability to have autonomous. driving across different environments. vehicle platforms domains and use cases. uh. you know as you know got started in uh. 2009 it was a lot almost like an. immediate successor to the grand.

challenge and urban challenges that were. like incredible uh kind of catalyst for. this whole space um and so google. started this project and then eventually. waymo spun out and so what waymo is. doing is creating uh the. systems both you know hardware software. infrastructure and everything that goes. into it to enable and to commercialize. autonomous driving this hits on consumer. transportation and ride sharing and kind. of vehicles and urban environments. and as you mentioned it hits on.

autonomous trucking to. to transport goods so in a lot of ways. it's transporting people and. transporting goods um but at the end of. the day the underlying capabilities are. required to do that are surprisingly. better aligned than one might expect. where it's the fundamentals of um of. being able to understand the world. around you process it make intelligent. decisions and prove that we are at a. level of safety that enables uh. large-scale autonomy so from a branding. perspective sort of uh waymo driver is.

the system that's irrespective of a. particular. uh. vehicle it's operating in there you have. a set of sensors that perceive the world. can act in that world and move this. whatever the vehicle is what's that. legal platform that's right and so in. the same way that you have a driver's. license and like your ability to drive. isn't tied to a particular make and. model of a car and of course there's. special licenses for other types of. vehicles but the fundamentals of a human. driver very very large you carry over. and then there's uniquenesses related to. a particular environment or domain or a.

particular um vehicle type that kind of. add some extra additive challenges but. that's exactly right it's the underlying. systems that enable uh. a physical vehicle without a human. driver to uh very successfully. accomplish the tasks that previously um. what wasn't possible um without um you. know 100 human driving. and then there's. way more one which is the transporting. people that's right from a brand.

perspective and just in case we refer to. it so people know and then there's waymo. via which is the trucking component why. via by the way what is that what is that. what's is it just like a cool sounding. name that just yeah uh like is there. does there an interesting story there. just it is a pretty cool sounding name. it's a cool sounding name i mean when. you think about it it's just like well. we're gonna transport it via this and. that like so it's just kind of like an. allusion to um the mechanics of. transporting something yes cool um and. uh and it is a pretty good grouping and.

the interesting thing is that even the. groupings kind of bore where waymo one. is like human transportation and uh. there's a fully autonomous service in. the phoenix area that like every day is. transporting people and it's pretty. incredible to like just you know see. that operate at reasonably large scale. and just kind of happen and then on the. via side it doesn't even have to be. like long-haul trucking is a like a. major focus of uh of ours but down the. road you can stitch together the vehicle. transportation as well for local. delivery um also and a lot of this.

requirements for local delivery overlap. very heavily with consumer. transportation um. obviously uh you know given that you're. operating on a lot of the same roads um. and uh. and navigating the same safety. challenges and so um. yeah and wave mode very much is a. multi-product company that. has. ambitions in both they have different. challenges and both are tremendous. opportunities but the cool thing is is. that there's a huge amount of leverage. and this kind of core technology stack.

now gets pushed on by both sides and. that adds its own unique challenges but. the success case is that um the. challenges that you push on um they get. leveraged across all platforms and also. from an engineering perspective the. teams are integrated. it's a mix so there's a huge amount of. centralized kind of core teams that. support all applications and so you. think of something like the hardware. team that develops the lasers the. compute integrates into vehicle. platforms this is an experience that. carries over across um you know any.

application that we'd have and they have. been flow with both then there's like. really unique um perception challenges. planning challenges like other you know. types of challenges where there's a huge. amount of leverage on a cortex stack but. then there's like dedicated teams that. think of how do you deal with a unique. challenge for example. an articulated trailer with varying. loads that completely changes the. physical dynamics of a vehicle that. doesn't exist on a car but becomes one. of the most important. kind of unique new challenges on a truck. so. what's the long-term dream.

of waymo via. uh the autonomous trucking effort that. waymo is doing yeah so we're starting. with developing. uh. l4 autonomy for. class 8 trucks these are 53-foot. trailers that. capture like a big perc a pretty sizable. percentage of the good transportation in. the country. long term the opportunity is obviously. to expand to much more diverse types of. vehicles. types of good transportation and start. to really expand in both the volume and.

the route feasibility that's possible. and so just like we did on the car side. you start with. a single route with a very specific. operating kind of domain and constraints. that allow you to solve the problem but. then over time you start to really try. to push. against those boundaries and open up. deeper feasibility across routes across. surface streets across environmental. conditions across the type of goods that. you carry the versatility of those goods. and how. little supervision is necessary to just.

start to scale this network. and. long term there's actually it's a pretty. incredible enabler where um. you know today you have. already a giant shortage of truck. drivers it's uh over 80 000 truck driver. shortage that's expected to grow to. hundreds of thousands in the years ahead. you have. really really quickly increasing demand. from e-commerce and just just. distribution of uh where people are. located. um you have one of the deepest safety. challenges of um.

of any profession in the u.s where um. there's a. huge huge kind of challenge around. fatigue and around kind of the long. routes that are driven. and even beyond kind of the cost and. necessity of it. there are fundamental constraints built. into our logistics network that are tied. to. the type of human constraints and. regulatory constraints that are. tied to trucking today for example our. limits on how long a driver can be. driving in a single day.

before they're they're not allowed to. drive anymore which is a very important. safety constraint. what that does is it enforces. limitations on how far. jumps with a single driver could be and. makes you very subject to availability. of drivers which influences where. warehouses are built which influences. how goods are transported which. influences costs and so um. you start to have an opportunity on. everything from plugging into existing. fleets and brokerages and the existing. logistics network and just immediately. start to have a huge opportunity to add.

value from. you know. cost and driving fuel insurance and. safety standpoint all the way to. completely reinventing the logistics. network um across the united states and. enabling something completely different. than what it looks like today yeah i had. uh. be published before this had a great. conversation with steve vicelli who we. talked about the manual driving and he. echoed many of the same things that you. were talking about but we talked about. much of the. the fascinating human stories of truck.

drivers he was also was a truck driver. for. for a bit as a grad student to try to. understand the depth of the problem he's. a fascinating wives. we have some drivers that have 4 million. miles of lifetime driving experience. it's pretty incredible and um. yeah it's uh. yeah learning from them like some of. them are on the road for 300 days a year. it's a very unique type of lifestyle so. there's fascinating stuff there just. like you said there's a shortage of. actually. people uh truck drivers. taking the job counter to what this i.

think is publicly believed. so there's an excess of jobs and a. shortage of people to take up those jobs. and just like you said it's such a. difficult problem. and these are experts at driving it's. solving this particular problem and it's. fascinating to learn from them to. understand. you know how hard is this problem and. that's the question i want to ask you. from a perception from a robotics. perspective. what's your sense of how difficult is. autonomous trucking maybe.

you can comment on which scenarios are. super difficult which are more. manageable is there is there a way to. kind of convert into words. how difficult the problem is yeah it's a. good question so there's um. and as you can expect it's a mix some. things become a lot uh uh. a lot. easier or at least more flexible um some. things are harder and so you know on the. things that are like uh the tailwinds. the benefits um. a big focus of.

automating trucking especially initially. is really focusing on the long-haul. freeway stretch of it where that's where. a majority of the value is captured on a. freeway you have a lot more structure. and a lot more consistency across. freeways across the u.s. compared to surface streets where you. have a way higher dimensionality of what. can happen lack of structural lack of. consistency and variability across. cities so you can leverage that. consistency to. tackle at least in that respect a more. constrained ai problem which has some.

benefits to it um you can itemize much. more of the sort of things you might. encounter and so forth and so. those are benefits is there a canonical. freeway. and city we should be thinking about. like. is there is there a standard thing. that's brought up in conversation often. like here's a stretch of road. um what is it like when people talk. about traveling across country they'll. talk about. new york this is san francisco. is that the route like is there a. stretch of road that's like nice and.

clean. and then there's like cities with. difficulties in them that you kind of. think of as the canonical problem to. solve here right uh so starting with the. car side um. well waymo very intentionally picked the. phoenix area and the san francisco area. as a follow once we hit driverless where. when you think of consumer. transportation and ride sharing you know. kind of economy a big percentage of that. market is captured in the densest cities. in the united states and so really. pushing out and solving san francisco.

becomes a really huge opportunity and uh. importance and um. and you know places one dot on kind of. like the spectrum of like kind of. complexity uh the phoenix area starting. with chandler and then like kind of. expanding more broadly in the phoenix uh. metropolitan area it's i believe the. fastest growing city in the us it's a uh. kind of a higher medium-sized city but. growing quickly. and still captures a really wide range. of kind of like complexities and so. getting to driverless there actually. exposes you a lot of the building blocks.

you need for the more complicated. environments and so in a lot of ways. there's a thesis that if you start to. kind of place a few of these kind of. dots where san francisco has these types. of unique challenges dense pedestrians. all this like complexity especially when. you get into the downtown areas and so. forth and phoenix has like. a really interesting kind of spectrum of. challenges maybe you know other ones. like la kind of add freeway focus and so. forth you start to kind of cover the. full set of features that you might. expect and it becomes faster and faster. if you have the right systems in the. right.

organization to then open up the fifth. city and intensity in the 20th city on. trucking there's uh similar properties. where um obviously there's uniquenesses. and freeways when you get into really. dense environments and then. the real opportunity uh to then you know. get even more uh value is to think about. how you expand with like some of the. service street challenges but for. example right now we're looking um we. have a big facility that we're uh. finishing building in q1 in uh dallas. area um that'll allow us to do testing. from the dallas area on routes like.

dallas to houston dallas to phoenix um. going out east and dallas to austin. austin so that triangle um waymo should. come to austin. well waymo the car side was in austin. for a while yes i know yeah come back. yeah but uh trucking is actually texas. is one of the best places to start uh. because of both volume regulatory. weather there's a lot of benefits um on. trucking a huge opportunity is port of. la going east so in a lot of ways a lot. of the work is to start to stitch.

together a network and converge to. port of la where you have the biggest. port in the united states um and the. amount of goods going east from there is. pretty tremendous and then obviously. there's you know kind of channels. everywhere and you have extra. complexities as you get into like snow. and inclement weather and so forth but. um what's interesting about trucking is. every single route segment that you add. increases the value of the whole network. and so it has this kind of network. effect and cumulative effect that's very. unique and so there's all these. dimensions that we think about um and so.

in a lot of ways dallas has a really. unique hub that opens up a lot of. options has become a really valuable. weber so the million questions i get. asked first of all. you mentioned level four. for people who totally don't know. there's these levels of automation. that uh level four refers to uh. kind of the first step that you could. recognize is fully autonomous driving. level five is really fully autonomous. driving level four is kind of fully. autonomous driving and then there are.

specific definitions depending on who. you ask what that actually means but. for you what does the level four mean. and you mentioned freeway let's say like. there's three parts of long-haul. trucking maybe i'm wrong in this but. there's freeway driving. there's like. truck stop. and then there's. more urban-y type of area so which of. those do you want to tackle. which of them do you include under level. four like how do you think about this.

problem what do you focus on where's the. biggest impact to be had in the short. term. so the goal is to we get we got to get. to market as fast as we can because the. moment you get the market you just learn. so much and it influences everything. that you do and it is um. uh i mean one of the experiences that. carried over from before is that you add. constraints you figure out the right. compromises you do whatever it takes. because getting the market like is so. critical right and here with autonomous. driving you can get to market in so many. different ways that's right and so one.

of the simplified simplifications that. we intentionally have put on is using. what we call transfer hubs where you can. imagine. depots uh that are. uh. at the entry points to metropolitan. areas like let's say dallas like the hub. that we're building which does a few. things that are very valuable so from a. first product standpoint you can. automate transfer hub to transfer hub. and that path from the transfer hub to. the. you know the full freeway route can be a. very intentional single route that you.

can select for the features that you. feel you want to handle at that point in. time then you build the hub. specifically designed. for time tracking and that's what's. going to happen actually like and you. get you need to come out in january and. check it out because it's going to be. really cool it's the not only is it our. main operating um headquarters for our. fleet there but it will be the first uh. fully ground-up design driverless hub. for autonomous drivers autonomous trucks. in terms of where do they enter where do. they depart how do you think about the. flow of people goods everything it's. like it's quite cool and it's really.

beautiful on how it's thought through. and so early on it is totally reasonable. to do the last. five miles manually to get to the final. kind of depot to avoid having to solve. the general surface street problem which. is obviously very complex now when the. time comes and we are increasingly we're. already we're pushing on some of this. but we will increasingly be pushing on. surface street capabilities to build out. the value chain to go all the way deeper. to depot instead of transfer hub the. transfer hub and we have probably the. best advantages in the world because of. all the waymo experience on surface.

streets but that's not the highest roi. right now where the highest roi is. hub the hub and get the routes going and. so when you ask what's l4. l4 can be applied to any domain. operating domain or scope but it's. effectively for the places where we say. we're ready for autonomous operation we. are 100. operating uh with uh through the as a. self-driving truck with no uh human. behind the wheel that is l4 autonomy and. it doesn't mean that you operate in.

every condition it doesn't mean you. operate on every road but for a. particularly well-defined area uh. operating conditions routes kind of. domain you are fully autonomous and. that's the difference between l4 and l5. and most people would agree that at. least any time in the foreseeable future. l5 is just not even really worth. thinking about because there's always. going to be these extremes. and so it's a race and a almost like a. game where you think of what is the. sequence of expanded capabilities that. create the most value and teach us the. most and create this feedback loop where.

we're building out and unlocking more. and more capability over time i gotta. ask you just curious so first of all i. have to when i'm allowed to visit the. dallas facility because it's super cool. it's like robot. on the giving and the receiving end it's. the truck is a robot and the the hub is. a robot yeah it's got to be very robot. friendly so yeah that's great. i will feel at home uh. the what's the sensor suite like on the. hub if you can just high level mention.

it is. does the hub have like lidars and like. is is it is the truck doing most of the. intelligence or is the hub also. intelligent yeah so most of it will be. the truck and uh everything is like. connected like so we uh we have our. servers where we know exactly where. every truck is we know exactly what's. happening at a hub and so you can. imagine like a large back-end system. that over time starts to manage uh. timings goods delivery windows all these. sort of things and so you don't actually.

uh. need to um there might be special cases. where that is valuable to equip some. sensors in the hub but a majority of the. intelligence is going to be on the truck. because um. whatever is relevant to the truck. relevance should be seen by the truck. and can be relayed. uh remotely for any sort of kind of. cognizance or decision making but. there's a distinct type of workflow. where um where do you check trucks where. do you want them to enter what if. there's many operating at once where's. the staging area to depart how do you. set up the flow of humans and human cars.

and traffic so that you minimize the. interaction between humans and kind of. self-driving trucks uh and then how do. you even intelligently select the. locations of these transfer hubs that. are both really great service locations. for a metropolitan area and there could. be over time many of them for a. metropolitan area. while at the same time leaning into. the path of least resistance to lean. into your current capabilities and. strengths so that you minimize the. amount of work that's necessary to. unlock the next kind of big bar i have a.

million questions so first is the goal. to have no human in the truck. the goal is to have no human in the. truck now of course right now we're. testing with expert operators and so. forth but um the goal is to um now there. might be circumstances where it makes. sense to have a human or uh and and. obviously these trucks can also be. manually driven so sometimes like our we. talk with our fleet partners about how. um you can buy a waymo equipped diamor. truck down the road and on the routes.

that are autonomous it's autonomous on. the routes that are not it's um human. driven maybe there's l2 functionality. that add safety systems and so forth but. as soon as they become. as soon as we expand in software the. availability of driverless routes the. hardware is forward compatible to just. now start using them um in real time and. so you can imagine uh this mixed use but. at the end of the day the largest value. proposition is where you're um able to. have no constraints on how you can. operate this truck um and it's 100. autonomous with nobody inside oh that's.

amazing so the. let me ask on the logistics front. because you mentioned that also. opportunity to revamp or for builds from. scratch some of the ideas around. logistics. i don't want to throw too much shade but. from talking to steve my understanding. is. logistics is not perhaps as great as it. could be in the current uh trucking uh. environment i'm not maybe you can break. down why but there's probably competing. companies. there's just a mess maybe some of it is.

literally just it's old school like they. it's just like it's not computer it's. not computerized. like. truckers are almost like contractors. there there's an independence and. there's not a nice interface where they. can communicate where they're going. where they're at. you know all those kinds of things and. so there it just feels like there's so. much opportunity to digitize everything. to where you could optimize the use of. human time optimize the use of all kinds. of resources. how much you thinking about that problem. how fascinating is that problem.

how difficult does it how much. opportunity is there to revolutionize. the space of logistics in autonomous. trucking in trucking period it's pretty. fascinating it's uh this is one of the. most motivating aspects of all this. where like yes there's like a mountain. of problems that are like you wanna you. have to solve to get to like the first. checkpoints and first drive list and so. forth and inevitably like in a space. like this you plug in initially into the. existing kind of system and start to. kind of you know learn and iterate but. um that opportunity is massive and so. you know a couple of the factors that um.

play into it so first of all um there's. obviously just the physical constraints. of driving time driver availability. some fleets have a 95 attrition rate you. know right now because of just. this demands and like you know kind of. gaps in competition and so forth and. then it's also incredibly fragmented. where. you would be shocked at like when you. when you look at industries like when. you think of the top 10 players like the. biggest fleets like the walmarts and. fedexes and so forth the percentage of. the overall trucking market that's.

captured by the top 10 or 50 fleets is. surprisingly small um the average kind. of uh truck operation is like a one to. five truck you know family business um. and so and so there's just like a huge. amount of like. fragmentation which makes for um. really interesting challenges in kind of. stitching together through like bulletin. boards and brokerages and some people. run their own fleets and and this. world's kind of like evolving um but. it is one of the less digitized and.

optimized worlds that there is. and the part that is optimized is. optimized to the constraints of today. and even within the constraints of today. this is the 900 billion dollar industry. in the u.s. and it's continuing to grow it feels. like from a business perspective. if i were to predict. that while trying to solve the. autonomous trucking problem waymo might. solve first the logistics problem. like because that that would already be. a huge impact yeah so on the way to. solving autonomous trucking.

the human driven like there's so much. opportunity to. significantly improve the human driven. trucking the timing the logistics so you. use humans optimally the handoffs the. like you know well even that you i mean. you get really ambitious you start to. expand this beyond like how does the uh. fulfillment center work and like how. does the transfer hub work how does a. warehouse work to. i mean there's a lot of opportunities to. start to automate these chains and um. a lot of the inefficiency today is.

because like you have a delay like port. of la has a bunch of ships right now. waiting outside of it because they can't. dock because there's not enough. labor inside of the port of la that. means there's a big backlog of trucks. which means there's a big backlog of. deliveries which means the drivers. aren't where they need to be and so you. have this like huge chain reaction and. your feasibility of readjusting in this. network is low because everything's tied. to humans and manual kind of processes. uh or distributed processes across a. whole bunch of players.

and so. one of the biggest enablers is um yes we. have to solve autonomous trucking first. and that by the way that's not like an. overnight thing that's decades of. continued kind of expansion and work. but um the first checkpoint in the first. route is like is not that far off but. once you start enabling and you start to. learn about how. the. constraints of autonomous trucking which. are very different in the constraints of. human trucking and again strengths and. weaknesses. how do you then start to leverage that.

and. rethink a flow of goods uh more broadly. and this is where like the learnings of. like really partnering with some of the. largest fleets in the us. and the sort of. learnings that they have about the. industry and the sort of needs that they. have and. what would change if you just. like really broke this one constraint. that like holds up the whole network or. what if you enabled this other. constraint. that actually drives the roadmap in a. lot of ways because um this is not like. an all or nothing problem it's uh you. know you start to kind of unlock more. and more functionality over time which.

functionality most enables this. optimization ends up being kind of part. of the discussion but you're totally. right like you fast forward to like you. know five years ten years uh 15 years. and. you think about like very generalized. capability of automation and logistics. as well as the ability to like poke into. how those handoffs work. the efficiency goes far beyond just. direct cost of today's like unit. economics of a truck they go towards. reinventing the entire system um in the.

same way that uh you know you see you. know these other industries that uh like. when you get to enough scale you can. really rethink um how you build around. your new set of capabilities not. the old set of capabilities yeah use the. analogy metaphor or whatever that. autonomous trucking is like email versus. mail and then with email you're still. doing the communication but it opens up. all kinds of comm. varieties of communication that. you didn't anticipate that's right. constraints are just completely. different um and yeah there's definitely.

a property of that here um and we're. also still learning about it because. there there is a lot of really um. fascinating and sometimes really elegant. things that the industry has done where. there's companies whose entire existence. is around despite the constraints. optimizing as much as they can out of it. and those lessons do carry over but it's. an interesting kind of merger of worlds. to think about like well what if this. was completely different how would we. approach it. and the interesting thing is that. for a really really really long time.

it's actually going to be the merger. between how to use autonomy and how to. use humans that leans into each each of. their strengths. yeah and then we're back to cosmo. human robot interaction so and the. interesting thing about waymo is because. there's the passenger vehicle the the. human the transportation of humans and. transportation of goods. you could see over time they might kind. of. meld together more. because you you'll probably have like. zero occupancy vehicles moving around so. you have transportation goods for short. distances and then.

for slightly longer distances and then. slightly longer and then there'll be. this then you just see the difference. between a passenger vehicle and a truck. is just size and you can have different. sizes and all that kind of stuff and at. the core you can have a way more driver. that doesn't as long as you have the. same that's sweet you can just think of. it as one problem and that's why over. time these do come kind of converge. where in a lot of ways a lot of the. challenges we're solving are freeway. driving which are going to carry over. very well to the vehicles to the car. side. um but there are like then unique.

challenges like uh you have a very. different dynamics in your vehicle where. you have to see much further out in. order to have the proper like response. time because you have an 80 000 pound. fully loaded truck um that's a very very. different type of braking profile than a. than a car you have uh. really interesting kind of dynamic. limits because of the trailer where you. actually it's very very hard to like. physically like flip a car or do. something like physically like most risk. in a car is from just collisions um. it's very hard to like in any normal.

operation to do something other than. like you know unless you hit something. it's actually kind of like roll over or. something on a truck you actually have. to drive much closer to the physical. bounds of the safety limits um. but you actually have like. real constraints because you could uh. you know you could have a really. interesting interactions between the. cabin and the trailer yeah there's. something called jackknifing if you turn. you know too quickly. you have roll risks and so forth and so. we spend a huge amount of time. understanding those boundaries and those. boundaries change based on the load that.

you have which is also an interesting. difference you have to propagate through. the out that through the algorithm so. that you're leveraging your dynamic. range but always staying within the. safety balance but understanding what. those safety bonds are and so we have. this like really cool test facility. where we like take it to the max and. actually imagine a truck with these. giant training wheels on the back of the. trailer and you're pushing it past the. safety limits uh in order to like try to. actually see where it rolls and so you. you you define this high dimensional. boundary which then gets captured in. software to stay safe and actually do.

the right thing but uh it's kind of. fascinating the sort of uh you know kind. of challenges you have there um but then. all of these things drive really. interesting challenges from perception. to um. unique behavior prediction challenges. and obviously in planner where you have. to think about merging and creating gaps. with a 53 foot trailer and so forth and. then obviously the platform itself is. very different where you have different. numbers of sensors sometimes types of. sensors and you also have unique blind. spots that you have because of the. trailer which you have to think about. and so it's a really interesting.

spectrum and in the end. you try to capture these special cases. in a way that is cleanly augmentations. of the existing tech stack. because a majority of what we're solving. is actually generalizable to freeway. driving um and different platforms and. over time. they all start to kind of merge ideally. where the things that are unique are as. as minimal as possible and that's where. you get the most leverage and that's why. waymo can do. you know take on two trillion dollar. opportunities um and.

have been nowhere near 2x the cost or. investment or size in fact it's much. much smaller than that. because of the high degree of leverage. so what kind of sensor suite. they can speak to that uh that a long. haul truck needs to have lidar vision. how many what are we talking about here. yeah so it's um more than the cars so. very loosely you can think of as like 2x. but it varies depending on the sensor. and so we have like dozens of cameras.

radar and then multiple lidar as well. you'll see one difference where the cars. have a central main sensor pod on the. roof in the middle and then a some kind. of hood sensors for blind spots the. truck moves to two main sensor pods on. the outsides where you would typically. have the mirrors next to the driver. they effectively go as far out as. possible um kind of up to. the understanding of the front. kind of on the cabin not all the way in. the front but like kind of where the. mirrors for the driver would be and so. those are the main sensor pods and the.

reason they're there is because if you. had one in the middle the trailer is. higher than the cabin and you would be. included with this like awkward wedge. too much occlusion too much occlusion. and so then you would add a lot of. complexity to the software yeah to make. up for that and and just unnecessary. components so many probably fascinating. design choices really cool because you. can probably bring up light or higher. and have it in the center or something. you could have all kinds of choices you. have to make the decisions here yeah. that ultimately probably will define the. industry right but by having two on the. side there's actually multiple benefits.

so one is like um you're just beyond the. trailer so you can see fully flush with. the trailer and so you eliminate most of. your blind spot except for right behind. the trailer um which is which is great. because now the software carries over. really well and the same perception. system you use on the car side largely. that architecture can carry over and you. can retrain some models and so forth but. you leverage it a lot it also actually. helps with redundancy where. there's a really nice built-in. redundancy for all the lidar cameras and. radar where you can afford to have any. one of them fail and you're still okay.

and at scale every one of them will fail. um. and you will be able to detect when one. of them fails because they don't uh. because the redundancy. they're giving you the data that's. inconsistent with the rest of that's. right and it's not just like they no. longer give data it could be like. they're fouled or they stop giving data. where the. some electrical thing gets cut or you. know part of your compute goes down so. what's neat is that like you have way. more sensors part of his field of view. and occlusions part of its redundancy. and part of it is new use cases so.

there's um uh new types of sensors uh to. optimize for long range and uh kind of. the the the sensing horizon that we look. for on our vehicles um that is unique to. trucks because it actually is like kind. of much like further out than um than a. car but a majority are actually used. across both cars and trucks and so we. use the same compute the same uh. fundamental baseline sensors cameras uh. radar um imus and so you get a great. leverage from all of the infrastructure.

and the hardware development as a result. so what about cameras what role does so. lidar is this rich set of information. has its strengths um has some weaknesses. camera is this rich source of. information that has some strengths has. its weaknesses. what role does lidar play what role does. vision. cameras play. in this in this beautiful. problem of autonomous trucking ah it is. beautiful there's like so much that. comes together. and how much yeah at which point do they.

come together yeah. so let's start with lidar so lidar has. been like waymo's um uh one of waymo's. big strengths and advantages where uh we. developed our own lidar uh in-house. where many generations in both in cost. and functionality it is um uh the best. and you know in this in the space which. generation because i know there's this. there's uh this cool i mean i love. versions that are increasing uh which. version of the hardware stack is at. currently uh officially publicly uh so.

uh so some parts iterate more than. others i'm trying to remember on the. sensor side so this the entire. self-driving system which includes. sensors and compute is fifth generation. yes um i can't wait until there's like. iphone style like announcements yeah for. like new versions of the weymouth. hardware yeah well we try to be careful. because man when you change the hardware. it takes a lot to like retrain the. models and uh and everything so we just. went through that and going from the. pacificas to the jaguars and so the. jaguars and then the trucks are you know. have the same generation now um but yeah.

the lidar is uh it's incredible and so. waymo has um leaned into that as a. strength and so a lot of the near-range. perception system. that obviously kind of carries over a. lot from the car side uh uses lidar as a. very prominent kind of like primary. sensor but then obviously. everything has its strengths and. weaknesses and so in the near range. lidar is a gigantic advantage um and it. has its weaknesses on you know when it. comes to occlusions in certain areas. rain and weather like you know things.

like that but it's an incredible sensor. and it gives you incredible density. perfect location precision and. consistency which is a very valuable. property um to be able to uh to kind of. apply a mel approach can you elaborate. consistency yeah when you have a camera. the position of the sun the time of the. day uh um various of the properties can. have a big impact uh whether there's. glare the field of view things like that. um. so consistent the signal with uh.

in the face of a changing external. environment the signal yeah daytime. night time. it's about 3d um. physical existence in effect like you're. you're seeing. beams of light that bounce physically. bounce off of something and come back. and so whatever the conditional. conditions are like the shape of a. human. sensor reading from a human or from a. car or from an animal like you have um a. reliability there which ends up being. valuable for kind of like the long tail. of challenges yeah now.

lidar is the first sensor to drop off in. terms of range and ours has a really. good range but at the end of the day um. it drops off and so particularly for um. for trucks on top of the general. redundancy that you want for near range. with and complements through cameras and. radar for occlusions and for. complementary information and so forth. when you get to long range you have to. be radar and camera primary because your. lidar data will fundamentally drop off. after a period of time and you have to. be able to see um kind of objects. further out now uh cameras have uh.

the the incredible range um where you. get a high density high resolution. camera you can get data you know well. past a kilometer and it's like really um. potentially a huge value now the signal. drops off the noise is higher detecting. is harder classifying is harder and one. that you might think about localizing. it's harder because you can be off by. like. two meters and where something's located. a kilometer away and that's the. difference between being on the shoulder. and being in your lane and so you have. like interesting challenges there that. you have to solve which have a bunch of. approaches that come into it um radar is.

interesting because um. uh. uh because it also has longer range than. um than lidar uh and it gives you speed. information so it becomes very very. useful for dynamic information of. traffic flow uh vehicle motions animals. pedestrians like uh just things that. might be um useful signals um and uh. it helps with weather conditions where. radar actually penetrates weather. conditions in a better way than um other.

sensors and so it's just it's kind of. interesting where we've kind of started. to converge towards not thinking about a. problem as a lidar problem or a camera. problem or radar problem but it's a. fusion problem where. these are all like large scale ml. problems where you put data into the. system and in many cases you just look. for the signals that might be present in. the union of all of these and. leave it to the system as much as. possible to start to really identify how. to um how to extract that and then. there's places we have to intervene and. actually.

include more but um. no single sensor is in a great position. to like really solve this problem and. then without a huge extra challenge. that's fascinating um. there's a question that's probably still. an open question is at which point do. you fuse them. do you. do. do you solve the perception problem for. each sensor suite individually the. lighter suite and the camera suite or do. you. do some kind of heterogeneous fusion or. do you fuse at the very beginning.

is there a good answer or at least an. inkling of intuitions you can accomplish. yeah so people refer to this as like um. early fusion or late fusion so late. fusion might be that you have like the. the camera pipeline the lidar pipeline. and then you like fuse them and like. when it gets to like final. you know semantics and classification. and tracking you like kind of fuse them. together and and figure out which one's. best um there's more and more evidence. that um uh that early fusion is. important um and that is because uh.

weight fusion does not allow you to pick. up on the complementary strengths and. weaknesses of the sensors um weather is. a great example where um if you do early. fusion you have an incredibly hard. problem for any single sensor in rain to. solve that problem um because you have. reflections from the lidar um you have. uh. you know weird kind of noise from the. camera blah blah blah right but the. combination of all of them can help you. filter and help you get to the real. signal that then gets you as close as.

possible to the original stack. and be much more fluid about the. strengths and weaknesses where. um you know your camera is much more. susceptible to like kind of uh fouling. on the on the actual lens from. you know like rain or random stuff. whereas like you might be a little bit. more resilient than other sensors and so. there's an element of. logic that always happens late in the. game but that fusion early on actually. especially as you move towards ml and. large-scale data-driven approaches just. maximizes your ability to pull out the. best signal you can out of each modality.

before you start making constraining. decisions that end up being hard to. unwind late in the stack so how much. of this is a machine learning problem. what role does ml machine learning. playing this whole. problem of autonomous driving autonomous. trucking. it's um. massive and it's increasing over time. you know if you go back to um. you know the grand challenge days in the. early days of kind of av development. there was ml but it was not in like kind.

of the mass scale data style of ml it. was like. learning models but in a more structured. kind of way and it was a lot of. heuristic and search-based approaches. and planning and so forth you can make a. lot of progress. with these types of approaches kind of. across the board an almost deceptive. amount of progress we can get pretty far. but then you re you start to really. grind the further you get in some parts. of stack. if you don't have an ability to absorb a. massive amount of experience in a way. that scales very sublinearly in terms of. human labor and human attention and so. when you look at the stack.

the perception side is probably the. first to get really revolutionized by ml. and it goes back many years because. ml for like computer vision and these. types of approaches has. kind of took off um was a lot of the. like early kind of push and um and deep. learning and so there's always a debate. on you know the spectrum between kind of. like end to end ml which. you know is a little bit kind of like. too far to how you architect it to where. you have modules but enough ability to. think about long tail problems and so.

forth but at the end of the day um you. have. big parts of system that are very ml and. data driven and we're increasingly. moving that direction all the way across. the board including. behavior. where. even when it's not like. a gigantic ml problem that covers like a. giant swath end to end more and more. parts of the system have this property. where you want to be able to put more. data into it and it gets better. and that has been one of the. realizations as you drive tens of. millions of miles and try to like solve.

new expansions of domains without. regressing in your old ones it becomes. intractable for a human to approach that. in the way that traditionally robotics. has kind of approached some elements of. the of the tech stack so are you trying. to um. create a data pipeline specifically for. the trucking problem this is it like how. much leveraging of the autonomous. driving is there in terms of data. collection yeah and. how unique. is the data required for the trucking. problem so we uh we we use all the same.

infrastructure um so labeling workflows. ml workflows everything so that actually. carries over quite well um. we heavily reuse the data even where. almost every model that we have on a. truck we started with the latest car. model cool and um so it's almost like a. good background model yeah it's like you. can think of like you despite the. different domain and different numbers. of sensors and position of sensors. there's a lot of signals that carry over. across driving and so it's almost like. pre-training and getting a big boost out. of the gate where you can reduce the.

amount of data you need by a lot. and it goes both ways actually and so. we're increasingly thinking about our. data. strategy on how we leverage both of. these. so you think about um you know how other. agents react to a truck yeah it's a. little bit different but the. fundamentals are actually like what will. other vehicles in the road do there's a. lot of carryover that's possible and in. fact. just to give you an example uh we're. constantly kind of like adding more data. from the trucking side but as of right. now. when we think of our like one of our. models behavior prediction for other.

agents on the road like vehicles. 85 percent of that data comes from cars. and a lot of that 85 comes from surface. streets. because we just had so much of it and it. was really valuable and so we're adding. in more and more particularly in the. areas where we need more data but you. get a huge boost out of the gate just. all different visual characteristics of. roads lane markings pedestrians all that. that's still relevant it's all still. relevant and then just the fundamentals. of how you know you detect the car.

does it really change that much whether. you're detecting it from a car or a. truck um the fundamentals of how a. person will walk around your vehicle is. it it'll change a little bit but the. basics like there's a lot of signal in. there that as a starting point to a. network can actually be very valuable. now we do have some very unique. challenges where there's a sparsity of. events on a freeway um the frequency of. events happening on a freeway whether. it's you know interesting. you know objects in the road or. incidents or or even like from a human. benchmark like how often does a human.

have an accident on a freeway is far. more sparse than on a surface street and. so that leads to really interesting data. problems where. you can't just drive infinitely to. encounter all the different permutations. of things you might encounter and so. there you get into interesting. tools like structured testing and data. collection data augmentation and so. forth and so there's really interesting. kind of technical challenges that. push some of the research um that. enables um these new suites of. approaches what role does simulation.

play. really good question so waymo simulates. about a thousand miles for every mile it. drives um so you think of in both so. across the board across the board yeah. uh so you think of for example well if. we've driven you know over 20 million. miles that's over 20 billion miles in. simulation now how do you use simulation. um. it's a multi-purpose so. uh you use it for basic development so. you want to do make sure you have. regression prevention and protection of. everything you're doing right um that.

that's an easy one. when you encounter something interesting. in the world let's say there was an. issue with how the vehicle behaved. versus an ideal human um you can play. that back in simulation and start. augmenting your system and seeing how. you would have reacted to that scenario. with this improvement or this new area. you can create scenarios that become. part of your regression set after that. point right um then you start getting. into like really really high kind of. hill climbing where um you say hey i. need to improve this system i have these. metrics that are really correlated with. final performance how do i know how well.

i'm doing. uh operation the actual physical driving. is the least efficient form of testing. and it's expensive it's time consuming. so grabbing a large scale. batch of historical data and simulating. it to get a signal of over these last or. just random sample of 100 000 miles how. has this metric changed versus where we. are today you can do that far more. efficiently in simulation than just. driving with that new system on board. right. and then you go all the way to the. validation phase where to actually see.

your human relative safety of like how. well you're performing on the car side. or the trucking side relative to a human. um. a lot of that safety case is actually. driven by. uh taking all of the physical. operational driving which probably. includes a lot of interventions where. like where the operate the driver took. over just in case um. and then you simulate those forward. and see if would anything have happened. and in most cases the answer is no but. you you can simulate it forward and you.

can even start to do really interesting. things where you. add virtual agents to create harder. environments you can fuzz the locations. of physical agents you can muck with the. scene and stress test the scenario from. a whole bunch of different dimensions. and effectively you're trying to like. more efficiently sample this like. infinite dimensional space but try to. encounter the problems as fast as. possible because what most people don't. realize is the hardest problem in. autonomous driving is actually the. evaluation problem in many ways not the.

actual autonomy problem and so if you. could in theory evaluate perfectly and. instantaneously. you can solve that problem in a really. fast feedback loop. quite well but the hardest part is being. really smart about this suite of. approaches on how can you get an. accurate signal on how well you're doing. as quickly as possible in a way that. correlates to physical driving that's in. the evaluation problem which metric are. you evaluating towards we're talking. about safety and some. what are the performance metrics that. we're talking about so in the end you.

care about and safety like that's in the. end what keeps you. like um that's what's deceptive where uh. there's a lot of companies that have. like a great demo. the path from like a really great demo. to being able to go driverless. can be deceptively long even when that. demo looks like it's driverless quality. and the difference is is that. the thing that keeps you from going. driverless is not the stuff you. encounter on a demo it's the stuff that. you encounter once in a hundred thousand. miles or 500 000 miles and so. that is at the root of what it what is.

most challenging about going driverless. because. any issue you encounter you can go and. fix it but how do you know you didn't. create five other issues that you. haven't that encountered yet so. those learnings like those were painful. earnings in waymo's history that waymo. went through and. led to us then finally being able to go. driverless in phoenix and now are at the. heart of how we develop. evaluation is simultaneously evaluating. final kind of end safety of how ready. are you to go driverless.

which may be as. you know direct as what is your. collision. human relative kind of collision rate uh. for all these types of scenarios and and. uh uh and severities to make sure that. you're better than a human bar you know. by by a good amount um but that's not. actually the most useful for development. for development it's much more kind of. analog metrics that. are part of the art of finding. how. what what are the properties of driving.

that give you a way quicker signal. that's more sensitive than a collision. that can correlate to qual the quality. you care about and push the feedback. loop to all of your development a lot of. these are for example comparisons to. human drivers like manual drivers how do. you how do you do relative to human. driver in various dimensions of various. um circumstances. can ask a tricky question so. if i brought you a truck how would you. test it. okay alan turing came along and you said. this one's can't tell if it's a human. driver or yeah exactly.

yeah but. not the human because. because you know humans are flawed but. yeah how do you actually know you're. ready basically how do you know it's. good enough um. yeah and by the way this is the reason. why like um. weymouth released the safety framework. for the car side because like. one it sets the bar so nobody cuts below. it um and does something bad for the. field that and that causes an accident. two it's to start the conversation on. like framing what does this need to look. like same thing we'll end up doing for. the trucking side um there it ends up.

being um. different demand different. portfolio of approaches there's easy. things like are you compliant with all. these like fundamental rules of the road. like you never drive above the speed. limit that's actually pretty easy like. you can fundamentally prove that it's. either impossible to violate that rule. or that in these like you can um. itemize the scenarios where that comes. up and you can do a test and show that. you you know you pass that test and. therefore you can handle that scenario. and so those are like traditional.

structure testing kind of system. engineering approaches where you can. just quant like. fault rates is another example where. when something fails how do you deal. with it you're not going to drive and. randomly wait for it to fail you're. going to force a failure and make sure. that you can handle it and close courses. and simulation or on the road. and. and run through all the permutations of. failures which you can often times for. some parts of system itemize like. hardware. the hardest part is behavioral where. you have. just infinite.

situations that could in theory happen. and you want to figure out the the. combinations of approaches that you know. that can work there you can probably. pass the turing test pretty quickly even. if you're not like completely ready for. driverless because the events that are. really. kind of like hard will not happen that. often just to give you a perspective. uh a human has a serious accident on a. freeway uh like a truck driver on a. freeway has uh there's a serious event. happens once every 1.3 million miles and.

something that actually has like a. really serious injury is 28 million. miles and so those are really rare and. so you could have a driver that looks. like it's ready to go but you have no. signal on on what happens there and so. that's where you start to get creative. on combinations of. sampling and statistical arguments. focused structured arguments where you. can kind of. simulate those scenarios and show that. you can handle them and. metrics that are correlated with what. you care about but you can measure much. more quickly and get to a right answer.

and that's what makes it pretty hard and. in the end um you end up borrowing a lot. of properties um from. uh aerospace and like space shuttles and. so forth where you don't get the chance. to launch it a million times just to say. you're ready because it's too expensive. to fail um and so you go through. a huge amount of kind of structured. approaches in order to validate it and. then by. by thoroughness you can make a strong. argument that you're ready to go. this is actually a harder problem in a. lot of ways though because you can think.

of a space shuttle as um. getting to a fixed point and then you. kind of like or an airplane and you like. freeze the software and then you like. prove it and you're good to go here you. have to get to a driver's quality bar. but then continue to aggressively change. the software even while you're. driverless and so and also the full. range of environment that you there's. there's an external environment where. the shuttle is you're basically testing. the. like the systems the internal stuff yeah. uh and you have a lot of control on the. external stuff yeah and the hard part is. how do you know you didn't get worse in.

something that you just changed yes. and so uh so in a lot of ways like. the turing test starts to fail pretty. quickly because you start to feel. driverless quality pretty early in that. curve. if you think about it right like in most. um. most uh kind of you know really good av. demos maybe you'll sit there for 30. minutes right yeah so you've driven you. know 15 miles or something like that um. to go driverless uh like what's the sort. of rate of issues that you need to have.

you won't even encounter so let's try. something different then let's try. a different version of the touring test. which is like an iq test. so there's these. difficult questions of increasing. difficulty they're very they're they're. designed you don't know them ahead of. time. nobody knows the answer to them right. and so is it possible to in the future. orchestrate yeah basically really. obstacle course almost of like yeah that. maybe change every year. and that represent if you can pass these.

it they don't necessarily represent the. full spectrum that's it yeah they won't. be conclusive but you can at least get a. really quick read and filter yeah like. you're able to yeah because you didn't. know them ahead of time like i don't. know. probably. like construction zones uh failures or. driving anywhere in russia yeah like. yeah. weather um cut-ins uh dense traffic kind. of merging lane closures. uh animal foreign objects on a road that. pop out on short notice mechanical. failures sensor braking tire popped.

weird behaviors by other vehicles like a. hard brake something reckless that. they've done fouling of sensors like. bugs or birds. you know poop or something so but yeah. like you have these like kind of like. extreme uh. conditions where like you have a nasty. construction zone where everything shuts. down and you have to like you know get. pulled to the other side of the freeway. with a temporary lane like that right. those are sort of conditions where we do. that to ourselves right we itemize. everything that could possibly happen to.

give you a starting point to how to. think about. what you need to develop and at the end. of the day there's no substitute for. real miles like if you think of. traditional ml like you know how there's. like a validation set where you hold out. some data and uh like real-world driving. is the ultimate validation set that's. the in the end like the cleanest signal. but you can do a really good job on. creating an obstacle course and you're. absolutely right like at the end um. if there was such a thing as automating. uh and kind of a readiness. um it would be these extreme conditions. like a red light runner right a um.

really reckless pedestrian that's. jaywalking a cyclist that you know makes. like a really awkward maneuver that's. actually what keeps you from going. driverless like in the end that is the. long tail. yeah and it's interesting to think about. the that to me is the touring test. stereotest means a lot of things but to. me in driving the touring test. is exactly this validation set that is. handcrafted there's a i don't know if. you know. him there's a guy named francoise he um. he decides he thinks about like how.

designed to test for general. intelligence he designs these iq tests. for machines. and the validation set for him is. handcrafted yeah and that it requires. like human genius or ingenuity to create. a really good test yeah and you hold you. truly hold it up it's an interesting. perspective on the validation set which. is like. make that as hard as possible right not. a generic representation of the data but. this is the hardest the hardest stuff.

yeah you know it's like go like you'll. never fully itemize like all the world. states that you'll you'll expand and so. you have to come up with different. approaches and this is where you start. hitting the struggles of ml where ml is. fantastic at optimizing the average case. it's a really unique craft to think. about how you deal with the worst case. which is what we care about in in av. space um when using an ml system on. something that that occurs like super. infrequently. so like you don't care about the worst. case really on ads because if you miss a. few it's not a big deal but you do care. about it on the driving side and so um.

and so typically like you'll never fully. enumerate the world and so you have to. take a step back and abstract away what. are the signals that you care about and. the properties of a driver. that correlate to defensive driving and. avoiding nasty situations that um. even though you'll always be surprised. by things you'll encounter you feel good. about your ability to generalize from. what you've learned. all right let me ask you a tricky. question.

so to me. the two companies. that. are building at scale some of the most. incredible robots ever built. is waymo and tesla. so. there's very distinct approaches. technically philosophically in these two. systems. let me ask you to play sort of devil's. advocate. and then the devil's advocate to the. devil's advocate.

it's it's a bit of a race of course. everyone can win. but. if waymo wins this race to level four. uh which. why would they win what aspect of the. approach do you think would be the. winning aspect and if tesla wins. why would they win and uh which aspect. of their approach would be the reason. just just building some intuition almost. not from a business perspective from any. of that just technically.

yeah yeah and we could summarize. i think maybe you can correct me what. one of the more. distinct. aspects is. uh waymo has a richer suite of sensors. as lidar and vision. tesla now removed radar they do vision. only tesla has a larger fleet of. vehicles operated by humans so it's. already deployed on the field in its uh. larger what do you call it operational.

domain. and then waymo is more focused on a. specific domain and growing it. with fewer vehicles so that's the both. are fascinating approaches both are i. think there's a lot of brilliant ideas. nobody knows the answer so i'd love to. get your comments on this lay of the. land yeah for sure so maybe i'll um i'll. start with waymo. and you're right like both incredible. companies and just a gigantic respect to. like everything tesla's accomplished and. uh how they push the field forward as. well so on the weymouth side there is a.

fundamental advantage in the fact that. it is focused and geared towards l4 from. the very beginning we've customized the. sensor suite for it the hardware the. compute the infrastructure the tech. stack and all of the investment inside. the company um. that's deceptively important because. there's like a giant spectrum of. problems you have to solve in order to. like really do this from. infrastructure to hardware to autonomy. stack to the safety framework and that's. an advantage because there's a reason.

why it's the fifth generation hardware. and. why all of those learnings went into the. dymor program um it becomes such an. advantage because you learn a lot as you. drive and you optimize for the best. information you have but fundamentally. like there's a big big jump um uh. like every order of magnitude that you. drive uh in numbers of miles and what. you earn and the gap from really kind of. like decent progress or l2 and so forth. to what it takes to actually go all for. and at the end of the day um.

there's a feeling that waymo has. uh there's a long way to go uh nobody's. won um but. there's a lot of advantages um in all of. these buckets where it's the only. company that has shipped a fully. driverless service we can go and you can. use it and it's at a decently like uh. you know sizeable scale um and those. learnings can feed forward and to solve. how to solve the more general problems. you see this process you've deployed in. chandler. you don't know the timeline exactly but. you could see the steps.

they they seem almost incremental. the steps it's become more engineering. than totally bind r d because it works. in one place and then you move yeah. another place and you grow it this way. and just to give you an example like we. fundamentally changed our hardware and. our software stack almost entirely from. what when driverless in phoenix to what. is the current generation of the system. on both sides. because the things that got us to. driverless even though it got to. driveway way like way beyond human. relative safety um it is.

fundamentally not well set up to scale. in an exponential fashion without like. getting into like huge kind of scaling. pains and so those learnings you just. can't shortcut and so that's an. advantage and so uh there's a lot of. open challenges to kind of get through. technical organizational like how do you. solve problems that are increasingly. broad and complex like this work on. multiple products but. there's the feeling that okay like balls. in our court. there's a head start there now we got to. go and solve it and i think that focus. on l4 it's a fundamentally different. problem if you think about it like um.

let's say we were designing an l2 truck. that was meant to be safer and help a. human you could do that with far less. sensors. far less complexity and provide value. very quickly arguably with what we. already have today just packaged up in a. good product but. you would take a huge risk in having a. gap from even like compute and sensors. not not to mention the software to then. jump from that system to an l4 system so. it's a huge risk basically so i can let. me allow me to be the person that plays. the devil's advocate and let's argue for.

the tesla approach so. that the what you just laid out makes. perfect sense and is exactly right there. are some open questions here which is. it's possible. that. investing more in faster data collection. which is essentially what tesla's doing. will get us there faster. if. the sensor suite doesn't matter. yeah as much and machine learning can do. a lot of the work this is the open. question is.

how much is is the thing you mentioned. before how much of driving can be end to. and learned. that's the open question obviously. the waymo and the vision only machine. learning approach. will. solve driving eventually both yeah the. question is of timeline what's faster. that's right and what you mentioned like. if i were to make the opposite argument. like what what puts tesla uh in the. strongest position it's data that is. their like superpower where they have an. access to real-world.

data effectively with like. a safety driver uh and uh you know like. they've they found a way to like um get. paid by safety drivers versus paper. safety drivers. it's uh it's brilliant right yeah but. you know all joking aside like um one it. is incredible that they've built a. business that's incredibly successful. they can now be a foundation and. bootstrap kind of like really aggressive. investment in autonomy space uh if you. can do it that's always like an. incredible kind of advantage and then. the data aspect of it um it is a giant.

amount of data if you can use it the. right way to then solve the problem but. the ability to collect um. and filter through the things that to. the things that matter at real-world. scale like a large distribution that is. a that is huge like it's a big advantage. um and so then the question becomes can. you use it in our right way and do you. have the right software systems and. hardware systems in order to solve the. problem and. you're right that in the long term. there's no reason to believe that pure. camera systems can't solve the problem. that humans obviously are solving with.

you know with vision systems. but. the question is when it's a risk it's a. big so there's no argument that it's not. a risk right like and it's already such. a hard problem and so much of that. problem by the way is um. uh you know even beyond the perception. side some of the hardest elements of the. problem on behavioral side and decision. making and the long tail safety case. if you are adding risk and complexity. on the input side from perception you're. now making a really really hard problem. like.

which is on its own is still like almost. insurmountably hard even harder and so. the question is just how much and this. is where like you can. easily get into a little bit of a. kind of a trap where similar to how you. how do you evaluate how good an av. company's product is like you go and you. do a. trial kind of a test run with them a. demo run which they've kind of optimized. like crazy and so forth and like and it. feels good do you do you put any weight. in that right you know that that gap is. kind of like you know pretty large still. um same thing on the like perception.

case like the long tail of computer. vision is really really hard and there's. a lot of ways that that can come up and. even if it doesn't happen that often at. all when you think about the safety bar. and what it takes to actually go full. driverless not like incredible. assistance driverless but full. driverless. that bar gets crazy high and not only do. you have to solve it on the behavioral. side but now you have to. push computer vision beyond arguably. where it's ever been pushed and so you. now on top of the broader av challenge.

you have a really hard perception. challenge as well so there's perception. there's planning there's human robot. interaction to me what's fascinating. about. what tesla is doing. is in this march towards level four. because it's in the hands of so many. humans. you get to see video you get to see. humans. i mean forget forget companies forget. businesses. it's fascinating for humans to be. interacting with robots that's. incredible and they're actually helping. kind of push it forward and yeah and. that is valuable by the way where even.

for us a decent percentage of our data. is human driving yes um we intentionally. have humans drive higher percentage than. you might expect because that creates. some of the best signals to train the. autonomy and so. that is uh on its own value so together. we're kind of learning about this. problem in an applied sense just like. you had with cosmo like once when when. you're chasing an actual product that. people are going to use. robot based product that people are. going to use you have to contend with. the reality of.

what it takes to build a robot that. successfully perceives the world and. operates in the world and what it takes. to have a robot that interacts with. other humans in the world and that. that's like to me one of the most. interesting problems humans have ever. undertaken because. you're. in trying to create an intelligent agent. that operates in a human world you're. also. understanding the nature of intelligence. itself. like. how hard is driving it's still not. answered to me yeah i still don't.

understand. like all the subtle cues like even. little things like um. your interaction with a pedestrian where. you look at each other and just go okay. go right like that's hard to do without. a human driver right and you're missing. that dimension how do you communicate. that so there's like really really. interesting kind of like elements here. now here's what's beautiful can you. imagine that like when autonomous. driving is solved. how much of the technology foundation of. that like space can go and have like. tremendous just transformative impacts.

on on other problem areas and other. other spaces that have. subsets of the these same problems like. it's just incredible. it's it's both a pro and a con is uh. with autonomous driving is so. um safety critical. it's so so once you solve it it's. beautiful because there's so many. applications that are a lot less safety. critical. but it's also the the con of that is. it's so safe it's so hard to solve and. the same journalists that you mentioned. and get excited for a demo are the ones.

who. who. write long articles about the failure of. your company if there's. one accident. that's based on a robot and it's it's. it's just society's so tense and. waiting for failure robots you're in. such a high stake environment failure. has such a high cost and it slows down. development it slows down development. yeah like the team like definitely. noticed that like once you go driverless. like we're driving from phoenix and you. continue to iterate.

your iteration pace slows down um. because your fear. of regression. forces. so much more rigor that you know. obviously you know you have to find a. compromise on like okay well how often. do we release driverless builds because. every time you release a driver's build. you have to go through this like. validation process which is very. expensive so far so um it is interesting. it's like it is just one of the hardest. things there's no other industry where. like uh you would not like you wouldn't. release products way way quicker when. you start to kind of provide.

even portions of the value that you. provide healthcare maybe is the other. one. that's right but at the same time right. like we've gotten there where you think. of like surgery right like you have. surgery there's always a risk but like. it's really really bounded you know that. there's an accident rate when you go out. and drive your car today right like. and you know what the fatality rate in. the u.s is per year we're not banning. driving because there was a car accident. but the bar for us is way higher and we. hold ourselves very serious to it where. you have to not only be better than a. human but you probably have to like at.

scale be far better than a human by a. big margin and you have to be able to. like really really thoughtfully explain. um all of the ways that we validate that. becomes very comfortable for humans to. understand because a bunch of jargon. that we use internally just doesn't. compute at the end of the day we have to. be able to explain to society how do we. quantify the risk. and acknowledge that there is some. non-zero risk but it's far above a human. you know relative safety here's the. thing. to push back a little bit.

uh and bring cosmo back in the. conversation he said something quite. brilliant at the beginning of this. conversation that i think probably. applies for. autonomous driving which is. you know there's this desire to make. autonomous cars much safer than human. driven cars. but. if you create a product that's really. compelling and is able to explain both. the leadership and the engineers and the. product itself. can communicate intent. then i think people may be able to be. willing to put up with the thing that.

might be even riskier than humans. because. they understand the value of taking. risks you mentioned the speed limit. humans understand the value of going. over the speed limit yeah humans. understand the value of like. going fast through a for through a. yellow light yeah to take in when you're. in manhattan streets pushing through uh. uh crossing pedestrians they understand. that i mean this is a much more tense. topic of discussion so this is just me. talking so in with cosmo's case there.

was something about the way. this particular robot communicated the. energy it brought the intent it was able. to communicate to the humans that you. understood. that of course he needs to have a camera. yeah of course he needs to have this. information and in that same way to me. of course a car needs to take risks of. course there's going to be accidents. that's what like that's. you know if you want a car that never. has an accident. have a car that just doesn't go anywhere.

yeah. and so that. but that's tricky because that's not a. robotics problem. like are not even under like due to you. right obviously. so there's a big difference though um. yeah you are. that's not a personal decision you're. also impacting obviously kind of the. rest of the road um and we're. facilitating it right and so there's a. higher kind of you know kind of ethical. and moral bar which obviously then you. know translates into as a society and.

from a regulatory standpoint kind of. like what what comes out of it where. it's hard for us to ever see this even. being a. debate in the sense that like. you have to be beyond reproach from a. safety standpoint because if you're. wrong about this you could set the. entire field back a decade right see i i. this is me speaking i think if we look. into the future. there will be. i personally believe. this is me speaking yeah. that there will be less and less focus. on safety still very very high yeah.

meaning like after autonomy is very. common and accepted it's not not not so. common as everywhere but there has to be. a transition because. i think for innovation just like you. were saying to explore ideas you have to. take risks and i think if autonomy. in the near term is to become prevalent. in society. i think people need to be more willing. to understand the nature of risk. the value of risk.

it's very difficult you're right of. course with driving. but that that's the fascinating nature. of it this. it's a. it's a life-and-death situation that. brings value to millions of people so. you have to figure out what what do we. value about this world how much do we. value. how deeply do we want to avoid. hurting other humans that's right and. there is a point where like. you can imagine a scenario where waymo. has a system. that is uh even when it's like uh kind.

of beyond a you know human relative. safety um and. provably statistically. will save lives. there is a. thoughtful navigation of. you know the that fact versus just. kind of. society readiness and perception. and. education of um. society and regulators and everything. else where like. it's it's multi-dimensional um and it's.

not a purely logical uh argument but um. ironically the logic can actually help. with the emotions. and just like any technology there's. early adopters and then there's kind of. like a curve that um happens after it. but eventually celebrities you get the. rock in a way more vehicle and then. everybody just comes and everybody calms. down because the rock likes it yeah. if you post uh yeah and it's like it's. an open question on how this plays out i. mean maybe we're presently surprised and. it just like people just realize that.

this is such a enabler of life and like. efficiency and cost and everything that. um there's a pull like at some point i. should fully believe that this will go. from a. thoughtful kind of you know. you know movement and tiptoeing and like. kind of like a push to society realizes. how. wonderful of an enabler this could. become and it becomes more of a pull and. um hard to know exactly how that will. play out but at the end of the day like. both. the goods transportation and the people. transportation side of it has that. property where it's not easy there's a.

lot of open questions and challenges to. navigate and there's obviously the. technical problems to solve uh as a you. know kind of prerequisite but um. they they have such an opportunity that. is um. on a scale that very few industries in. the last 20 30 years have even had a. chance to tackle that. i. maybe were pleasantly surprised by how. much how much that tipping point like in. a really short amount of time actually. turns into a societal pull to kind of. embrace the benefits of this yeah i i. hope so it seems like in the recent few.

decades there's been tipping points for. technologies where like overnight things. change it's uh like uh. from taxis to ride sharing services all. that that shift i mean there's just. shift after shift after shift that. requires digitization and technology i'm. i hope we're pleasantly surprised in. this so there's millions of long-haul. trucks now in the united states. do you see a future where. there's millions of waymo trucks and. maybe just broadly speaking way more. vehicles just.

like like ants running around the united. states uh. freeways and local roads yeah in other. countries too like uh. you look back decades from now and. it might be one of those things that. just feels so natural and then it. becomes almost like a kind of. interesting kind of oddity that we had. none of it like uh you know kind of. decades earlier. and. it'll take a long time to grow and scale. very different challenges appear at. every stage but. over time like.

this is one of the most enabling. technologies that um that we have in the. world uh today um it'll feel like. you know how was the world before the. internet how's the world before mobile. phones like it's gonna have that sort of. a feeling to it on both sides it's hard. to predict the future but. do you sometimes. uh think about weird ways it might. change the world like surprising ways so. obviously. there's more direct ways where like. there's increases efficiency it'll. enable a lot of kind of logistics. optimizations kind of things.

it will change. our uh probably our roadways and all. that kind of stuff but it could also. change society in some kind of. interesting ways do you ever think about. how might change cities how might change. their lives all that kind of yeah. you can imagine city uh where people. live versus work becoming more. distributed because the pain of. commuting becomes different just easier. uh and i don't know there's a lot of. options that open up. the way out of cities themselves and how. you think about.

car storage and parking obviously uh. just enables a completely different type. of uh uh. type of experience in urban environments. i i think there was like a statistic. that uh. something like. 30 of the traffic uh in cities. during rush hour is caused by a pursuit. of parking uh or some like some really. high stats so. those obviously kind of open up a lot of. options um. flexibility on goods will enable. new industries and businesses that never. existed before because now the.

efficiency becomes. more palatable good delivery timing. consistency and flexibility is going to. change. the way we distribute the logistics. network will change the way we then can. integrate with warehousing with. shipping ports you can start to think. about greater automation through the. whole kind of stack. and how that supply chain the ripples. become much more uh agile versus like. very. grindy the way they are.

today where just the adaptation is like. very tough and there's a lot of. constraints that we have i think it'll. be great for the environment it'll be. great for safety where like probably. about 95 of accidents today um. statistically are due to just uh. attention or things that are preventable. with uh. with the strengths of automation. yeah and it'll be one of those things. where like industries will shift but the. net creation is going to be massively. positive and then we just have to be. thoughtful about the negative. implications that will happen in local.

area places um and adjust for those but. i'm an optimist in general for the. technology where you could argue a. negative on any new technology but you. start to kind of. see that if there is a big demand for. something like this the in almost all. cases the like it's an enabling factor. that's gonna kind of propagate through. the um you know through society and. particularly as life expectancies get. longer and you know and so forth like. there's a just a lot more need for um a. greater percentage of the population to. kind of just be serviced with a high.

level of efficiency. because otherwise we can have a really. hard time kind of scaling to what's. ahead in the next 50 years um. yeah and you're absolutely right every. technology has uh negative consequences. of positive consequences and we tend to. focus on the negative a little bit too. much. in fact autonomous trucks are. often brought up as an example of uh. artificial intelligence and robots in. general taking our jobs. and as we've talked about briefly here. we talk a lot with steve you know.

that's. it is a concern that automation. will take away certain jobs it will. create other jobs so there's temporary. pain. uh hopefully temporary but pain is pain. and all people suffer and that human. suffering is really important to think. about. how uh but. trucking is. ver i mean there's a lot written on this. is i would say far from the the thing. that that would cause the most pain yeah. there's even more positive properties.

about trucking where not only is there. just a you know huge shortage which is. going to increase the average age of. truck drivers is getting closer to 50. because the younger people aren't. wanting to come into it they're trying. to like incentivize lower the age limit. like all these sort of things um and the. demand is just going to increase and the. least favorable like it depends on the. person but in most cases the least. favorable types of routes are the. massive long-haul routes where you're on. the road away from your family 300 plus. stations steve talked about the pain of. those kind of routes from a family. perspective you're.

you're basically away from family it's. not just hours you work insane hours but. it's also just time away from family. right and just obesity rate is through. the roof because you're just sitting all. day like. it's really really tough and um. and that's also where like the biggest. kind of safety risk is because of. fatigue and um. and so when you think of the gradual. evolution of how trucking comes in first. of all it's not overnight it's gonna. take decades to kind of phase in all the. like there's just a long long long road. ahead. but.

the the routes and the portions of. trucking that are going to require. humans the longest and benefit the most. from humans are the short-haul and most. complicated kind of more urban routes. which are also the more more pleasant. ones which are um you know less. continual driving time more um uh. more flexibility on like you know. geography and location and you get to. kind of sleep with the at home with you. at your own home and very importantly if. you optimize the logistics you're going. to use human. you're going to use humans much better.

that's right and and thereby pay them. much better because like one one of the. biggest problems is truck drivers. currently are paid by like how much they. drive so you they really feel the pain. of it inefficient logistics yeah because. like if they're just sitting around for. hours which they often do not driving. waiting yeah they're not getting paid. for that time that's right and that so. like logistics has a significant impact. on the quality of life of a truck driver. and high percentage of trucks are like. uh empty because of inefficiencies in.

the system um yeah it's one of those. things where like um. and the other thing is when you increase. the efficiency of a system like this the. overall net like volume of the system. tends to increase right like the. the entire market cap of trucking is. going to go up um when the efficiency. improves uh and facilitates both growth. and industries and better utilization of. trucking um and so that on its own just. creates more and more demand which um. uh of all the places where ai comes in. and starts to really um. uh.

kind of reshape an industry this is one. of those where like there's just a lot. of positives that for at least any time. in the foreseeable future seem really. lined up in a good way um to um. kind of come in and help with the. shortage and start to kind of optimize. for the routes that are most dangerous. and most painful. yeah so. this is true for trucking but if we zoom. out broader you know automation and ai. does. technology broadly i would say but you. know automation. is a thing that.

has a potential in the next couple of. decades to shift the kind of jobs. available to humans yes and. so that results in. like i said human suffering because. people lose their jobs there's economic. pain there. and there's also a pain of meaning so. for a lot of people. work is a. source of uh meaning it's a source of. identity of. of pride. of you know. pride in getting good at the job pride.

in craftsmanship and excellence which is. what truck drivers talk about yeah but. but that this is true for a lot of jobs. and is that something you think about as. a sort of a roboticist zooming out from. the trucking thing um. like where. do you think it would be harder. to find activity and work that's a. source of identity and source of meaning. in the future. like i do think about it because you. want to make sure that you you worry. about the entire system like not just.

like the party economy plays in it but. what are the ripple effects of it down. the road and um. on enough of a time window there's a lot. of opportunity to put in the right. policies and the right opportunities to. kind of reshape and retrain and find. those openings and so just to give you a. few examples both trucking and cars. we have remote assistance facilities. that. are there to interface with customers. and monitor vehicles and. provide like very focused kind of. assistance on uh kind of areas where the.

vehicle may want to request help uh in. understanding an environment so those. are jobs that kind of get created and. supported. i remember like taking a tour of one of. the amazon facilities where you've. probably seen the kiva systems robots uh. where you have these orange robots that. have automated um. the warehouse like kind of picking and. collecting of items in this like really. elegant and beautiful way um it's. actually one of my favorite applications. of robotics of all time um. uh you know like i think it kind of came. across a company like 2006 was just. amazing and.

what was the warehouse or was the. transport little thing so basically. instead of a person going and walking. around and picking the seven items in. your order um. these robots go and pick up a shelf. and move it over in a row where like the. seven shelves that contain the seven. items are lined up and a. you know laser or whatever points to. what you need to get and you go and pick. it and you place it to fill the order. and so the people were fulfilling the. final orders what was interesting about. that is that when i was asking them. about like kind of the impact on labor. when they transitioned that warehouse. the throughput increased so much that.

the jobs shifted towards the final. fulfillment. even though the robots took over. entirely the the search of the items. themselves and the labor. the job stayed like nobody like that was. actually the same amount of jobs uh. roughly they were necessary but the. throughput increased by i think over 2x. or some some amount right like so. um you have these situations that are. not zero-sum games in this really. interesting way and the optimist to me. thinks that there's these types of. solutions in almost any industry where. the growth that's enabled creates.

opportunities that you can then leverage. but you got to be intentional about. finding those and really helping make. those links because. any even if you make the argument that. like there's a net positive. locally there's always tough hits that. you got to be very careful about that's. right you have to have an understanding. of that link because there's a short. period of time. whether training is required or just. mental transition or physical or. whatever is required. that's still going to be short-term pain. the uncertainty of it there's families.

involved you know it. it's i mean it's exceptionally. it's difficult on a human level and you. have to really think about that even you. can't just look at economic metrics. always it's human beings that's right. and you can't even just uh take it as. like okay well we need to like subsidize. or whatever because like there is an. element of just personal pride where. right. majority of people like people don't. want to just be okay but like they want. to actually like have a craft like you. said and have a mission and. feel like they're having a really. positive impact and so um.

my personal belief is that there's a lot. of transferability and skill set um. that is possible especially if you. create a bridge and an investment um to. enable it um and to some degree that's. our responsibility as well. this process. uh you mentioned kiva robots amazon. let me ask you about. the astro robot which is i don't know if. you've seen it it's amazon's announced. that.

it's a home robot that they have a. screen. looks awfully a lot like. cosmo has. i think different vision probably. what are your thoughts about like home. robotics in this kind of space there's. been a. quite a bunch of home robots social. robots that very unfortunately have. closed their doors that um for various. reasons perhaps they were too expensive. there's manufacturing challenges all. that kind of stuff what are your. thoughts about amazon getting into this. space.

yeah we had some signs that they were. getting into like long long long long. ago maybe they're a little. too interested in cosmo and uh yeah. during our conversations but they're. also very good partners actually for us. as we kind of disintegrated a lot of. shared technology but if i could also. get your thoughts on. you know you could think of. alexa as a robot as well yeah echo. do you see those as fundamentally. different just because you can move and. look around is that fundamentally. different than the thing that just sits. in place uh it opens up options um but.

uh. you know my first reaction is i think. like. i have my doubts that this one's going. to hit the mark because i think for the. price point that it's at. and the like kind of functionality and. value propositions that they're i'm. trying to put out it's uh uh it's still. searching for like the killer. application that like justifies i think. it was like a 1500 price point or kind. of somewhere around there that's a. really high bar so. there's enthusiasts an early adopters. will obviously kind of pursue it but you. have to like really really hit a high.

mark at that price point which we always. tried to we were always very cautious. about jumping too quickly to the more. advanced systems that we really wanted. to make but would have. raised the bar so much you have to be. able to hit it. in today's cost structures and. technologies. the mobility is an angle that hasn't. been utilized. but. it has to be utilized in the right way. um and so that's going to be the biggest. challenge is like can you. meet the bar of what a con what the mass. market consumer like you know think like.

you know our. uh our neighbors our friend parents like. would they find a deep deep value like. in you know fi in this at a mass scale. that you know that justifies the price. point i think that's in the end one of. the biggest challenges for robotics. especially consumer robotics. where you have to kind of meet that bar. uh it becomes very very hard um and. there's also the higher bar just like. you were saying with cosmo of. you know a thing that can. look one way and then turn around and. look at you.

there's that's either a super desirable. quality or super undesirable quality. depending on how much you trust the. thing that's right and so there's uh. there's a problem of trust to solve. there there's a problem of personalities. the thing is the quote-unquote problem. that cosmos solved so well yeah is that. there you trust the thing yeah and that. has to do with the company with the. leadership with the intent that's. communicated by the device and the. company and everything together yeah. exactly right uh and so um and i think. they also have to retrace some of the.

like learnings on the character side. where like as usual i think that's the. place where it's uh a lot of companies. are great at the hardware side of it and. can you know think about those elements. and then there's like you know the. thinking about the ai challenges. particularly the advantage of alexa is a. pretty huge boost for them um the. character side of it for technology. companies is pretty new novel territory. and so that will take some iterations. but um yeah i mean i hope. i hope there's continued progress in the. space and that threat doesn't kind of go. dormant for too long.

and it's not you know it's going to take. a while to kind of. evolve into like the ideal applications. but you know. this is one of um amazon's i guess like. you could call it it's definitely like. part of their dna but in many cases it's. also strength where they're very willing. to like iterate uh kind of aggressively. and um and move quickly not take risks. and take risks you have deep pockets so. you can yeah and they'll maybe have more. misfires than an apple would um but uh. you know it's different styles and. different approaches and um you know.

at the end of the day it's like there's. a few familiar uh kind of elements there. for sure which was uh you know kind of. you know homage. is one way to put it yeah uh so why is. it so hard. at a high level. um to build a robotics company a. robotics company that. lives for a long time so if you look at. so i thought cosmo for sure would live. for a very long time that to me was.

exceptionally successful vision and idea. and implementation. irobot is an example of a company that. has. pivoted in all the right ways to survive. and. arguably thrive. by focusing on the. having like a. have a driver that constantly provides. profit which is the vacuum cleaner. and of course there's like amazon what. they're what they're doing is they're. almost like taking risks so they can. afford it because they have other.

sources of revenue right but outside of. those examples. most robotics companies fail yeah. why why do they fail why is it so hard. to run a robotics company our robot's. impressive because they found a really. really great. fit of where the technology could. satisfy a really clear used case in need. and. they did it well and they didn't try to. overshoot from a cost-to-benefit. standpoint. robotics is hard because it like tends.

to be more expensive it combines way. more technologies than a lot of other. types of companies do if i were to like. say one thing that is maybe the biggest. risk and like a robotics company failing. is that. it can be either a. technology in search of a application. or. they try to bite off a. kind of an offering that has. a mismatch and kind of price to function. um and uh. just the mass market appeal isn't there. and um consumer products are just hard.

it's just i mean after all the years and. it like definitely kind of feel a lot of. the battle scars because you have um you. know you not only do you have to like. hit the function but you have to educate. and explain get awareness up deal with. different conductive consumers like uh. you know there's um there's a reason why. a lot of technology sometimes start in. the enterprise space and then kind of. continue forward in the consumer space. even like you know you see ar like. starting to kind of make that shift with. hololens and so forth in some ways. consumers and price points that they're.

willing to kind of uh be attracted in a. mass market way and i don't mean like. you know 10 000 enthusiasts bought it. but i mean like. you know 2 million 10 million 50 million. like mass market kind of interest uh you. know have bought it. that bar is very very high and typically. robotics is novel enough and. non-standardized enough to where pushes. on price points so much. you can easily get out of range where. the capabilities and today's technology. or just a function that was picked just. doesn't line up um and so that product.

market fit is very important. so the space of killer apps or. a rather super compelling apps is much. smaller because it's easy to get outside. the price range yeah and most consumers. and it's not constant right like yeah. that's why like we picked off. entertainment because. the quality was just so low in physical. entertainment that we could we felt we. could leapfrog that and still create a. really compelling offering at a price. point that was defensible and and we. like that proved out to be true um.

and. over time that same opportunity opens up. in healthcare in home applications and. you know commercial. applications and kind of broader more. generalized interface. but. there's missing pieces in order for that. to happen and all of those have to be. present um for it to line up and we see. these sort of trends in technology where. um you know kind of technologies that. start in one place. evolve and kind of grow to another. something starting gaming some things. start in. uh. in space uh or aerospace and then kind.

of move into the consumer market. and sometimes it's just a timing thing. right where how many. stabs at what became the iphone were. there over the 20 years before that just. weren't quite ready in the function um. relative to the kind of price point and. complexity and sometimes it's a small. detail of the implementation that makes. all the difference which is uh design uh. design is so important well something. yeah like the the you the new generation. ux right yeah it's um and uh and that's. uh.

um it's tough and oftentimes all of them. have to be there and it has to be like a. perfect storm and um. but yeah history repeats itself in a lot. of ways uh in a lot of these trends. which is pretty fascinating well let me. ask you about the humanoid form what do. you think about the tesla bot and. humanoid robotics in general. so obviously to me autonomous driving. waymo and the other companies working in. the space. that seems to be a great place to invest. in potential revolutionary application. robotics application focused application.

what's the role of humanoid robotics do. you think. teslabot is ridiculous do you think it's. super promising do you think it's. interesting full of mystery nobody knows. what do you think about this thing yeah. i think today humanoid form robotics is. research there's very few situations. where you actually need a humanoid form. to solve a problem uh if you think about. it right like wheels are more efficient. than legs there's. joints and degrees of freedom beyond a. certain point just add a lot of. complexity and cost right so if you're.

doing a humanoid robot oftentimes it's. in the pursuit of a humanoid robot not. in a pursuit of an application for the. time being. um. especially when you have like kind of. the gaps and interface and you know kind. of ai that we kind of talk about today. so anything you want does i'm interested. in following so there's there's an. element of that world no matter how. crazy how crazy it is i just like you. know i'll pay attention i'm curious to. see what comes out of it so it's like. you can't you can't ever you know ignore. it but you know it's uh definitely far. afield from their kind of core business. um uh obviously and um what was.

interesting to me is i've. i've disagreed with you know elon a lot. about this. is. to me the in the compelling aspect of. the humanoid form. and a lot of kind of robots cosmo for. example. is a human robot interaction. part. from. elon musk's perspective the tesla bot. has nothing to do with the human. it's a form. that's effective for the factory because. the factory is designed for humans.

but to me the reason you might want to. argue for the humanoid form is because. you know at a party. yeah it's a nice way to fit into the. party the humanoid form has a compelling. notion to it in the same way that cosmo. is compelling. i you i would argue if we were. arguing about this. that it's cheaper to build a cosmo like. that form but if you wanted to make an. argument which i have with jim keller. about you know you could actually make a. humanoid robot for pretty cheap.

it's possible. and then the question is all right if if. you're using an application where it can. be flawed. it could it can have a personality and. be flawed in the same way that cosmo is. that maybe it's interesting for. integration to human society. that's that's to me is an interesting. application of a humanoid form because. humans are drawn like i mentioned to you. legged robots we're drawn to legs and. limbs and body language and all that. kind of stuff. and even a face even if you don't have.

the facial features which you might not. want to have for the. uh. to reduce the creepiness factor all that. kind of stuff but yeah that to me the. humanoid form is compelling but in terms. of. that being the right form for the. factory environment i'm not so sure yeah. for the factory environment like right. off the bat um what are you optimizing. for is it strength is it mobility is it. versatility right like that changes. completely the look and feel of the. robot that you create you know and uh. almost certainly the human form is over.

designed for some asp dimensions and. constrained for some dimensions and so. like like what are you grasping is it. big is it little right so you would. customize it and make it um. customizable um for the different needs. if that was the optimization right and. then you know for the other one uh. i could totally be wrong you know i. still feel that the closer you try to. get to a human the more you're subject. to the um biases of what a human should. be and you lose flexibility to shift. away from your weaknesses uh and towards.

your strengths. and that changes over time but there's. ways to make. really. approachable. and natural interfaces for. robotic kind of characters and. you know and uh. you know and kind of deployments in. these applications. that. do not at all look like a human directly. but that actually creates way more. flexibility and capability and role and.

forgiveness and interface and everything. else yeah it's interesting but i'm still. confused by the magic i see in legged. robots yeah so there is a magic so i i'm. uh. absolutely amazed at it from a. technical curiosity standpoint and like. the. the magic that like the boston dynamics. team can do from uh you know like from. walking and jumping and so forth now. like there's been a long journey to try. to find an application for that sort of. um technology but wow that's incredible.

technology right yes so. then you kind of go towards okay are you. working back from a goal of what you're. trying to solve or are you working. forward from a technology and then. looking for a solution and i think. that's where um it's a kind of a. bi-directional search oftentimes but you. gotta you the two have to meet and that. that's where. humanoid robots is kind of close to that. and that like it is a decision about a. form factor and a. technology that it forces um. that. doesn't have a clear justification on. why that's the killer app or you know. from the other end but i think the core.

fascinating idea with the tesla bot is. the one that's carried by waymo as well. is when you're solving the general. robotics problem. of perception control where this there's. the very clear applications of driving. it's. as you get better and better at it when. you have like way more driver yeah. the whole world starts to kind of start. to look like a robotics problem so it's. very interesting for now detection. classification. segmentation tracking. planning like it's carrie yeah so.

there's no reason i mean i'm not i'm not. speaking for way more here but. you know. um. moving goods. there's no reason. transformer like this thing couldn't you. know uh take the goods up an elevator. you know yeah like that like uh slowly. expand yeah what it means to move goods. and. expand more and more of the world uh. into a robotics problem well that's. right and you start to like think of it.

as an end robotics problem from like. loading from you know from everything. yes and even like the truck itself um. you know today's generation is. integrating into. today's understanding of what a vehicle. is right the pacifica jaguar uh the. freightliners from daimler there's. nothing that stops these us from like. down the road after like starting to get. to scale to like. expand these partnerships to really. rethink what would the next generation. of a truck look like um that is actually.

optimized for autonomy not for today's. world. um. and maybe that means a very different. type of trailer maybe that like there's. a lot of things you could rethink on. that front which is on its own very very. exciting. let me ask you like i said you went to. the mecca of robotics which is cmu. carnegie mellon university you got a phd. there. so maybe by way of advice. and maybe by way of. story and memories what does it take to. get a phd.

in robotics at cmu. and maybe. you can throw in there some advice for. people who are thinking about. doing work in artificial intelligence. and robotics and are thinking about. whether to get a phd it's like i. actually went i was a cmu for undergrad. as well and didn't know anything about. robotics coming in and was doing you. know electrical computer engineering. computer science and really got more and. more into kind of ai and then fell in. love with autonomous driving and at that. point like that was just by a big margin.

like such a incredible like central spot. of. develop of investment in that area and. so what i would say is that like. robotics like for. all the progress that's happened is. still a really young field there's a. huge amount of opportunity now that. opportunity shifted where something like. autonomous driving has moved from being. very research and academics driven to. being commercial driven where you see. the investments happening. in commercial now there's other areas. that are much younger. and you see like kind of grasping and. manipulation making kind of the same.

sort of journey that like autonomy made. and there's other areas as well what i. would say is the space moves very. quickly anything you do a phd in like it. is in most areas will evolve and change. as technology changes and constraints. change and hardware changes and the. world changes. um and so. the beautiful thing about robotics is. it's super broad it's not a narrow space. at all and it can be a million different. things in a million different industries. and so. uh it's a great opportunity to come in. and get a broad foundation on ai machine.

learning computer vision systems. hardware sensors all these separate. things. you do need to like go deep and find. something that you're like really really. passionate. about obviously like just like any phd. this is like a. five six year kind of uh endeavor. and you have to. love it enough to go super deep to learn. all the things necessary to be super. deeply functioning in that area and then. contribute to it in a way that hasn't. been done before and in robotics that. probably means um more breadth because.

robotics is rarely kind of like one. particular kind of narrow technology. and it means being able to collaborate. with teams where like one of the coolest. aspects of like. my the exp the experience that i kind of. cherish in our phd is that we actually. had a pretty large av project that for. that time was like a pretty serious. initiative where you got to like partner. with a larger team and you had the. experts in perception and the experts in. planning and the staff and the. mechanical challenge um so i was working. on the a project called upi back then uh.

which was basically the off-road version. of the darpa challenge it was a darpa. funded project for. basically like a large off-road vehicle. that you would like. drop and then give it a waypoint 10. kilometers away and it would have to. navigate a complete structure in an. office environment yeah so like forest. ditches rocks vegetation and so it was. like a really really interesting kind of. a hard problem where like wheels would. be up to my shoulders it's like gigantic. right yeah by the way av for people. stands for autonomous vehicles house. vehicles yeah sorry um and so what i. think is like the beauty of robotics but.

also kind of like the expectation is. that um. there's um spaces in computer science. where you can be very very narrow and. deep. robotics one of the the necessity but. also the beauty of it is that it forces. you to be excited about that breadth and. that partnership across different. disciplines that enable it but that also. opens up so many more doors where you. can go and you can do robotics and. almost any category where robotics isn't. a in isn't really an industry it's like. it's like ai right it's like the. application of physical automation to uh.

you know to all these other worlds and. so you can do robotic surgery you can do. vehicles you can do factory automation. you can do healthcare or you can do like. uh. leverage the ai around the sensing to. think about static sensors and scene. understanding so um so i think that's. got to be the expectation and the. excitement and it. breeds people they're probably a little. bit more collaborative and more excited. about um. working in teams uh if i could briefly. comment.

on the fact that the robotics people. i've met in my life. from cmu and mit. they're really happy people yeah because. i think it's the collaborative thing. yeah i think i think you don't. you're not like a sitting in like the. fourth basement uh exactly. which when you're doing machine learning. purely software it's very tempting to. just disappear into your own hole yeah. and never collaborate and and there that.

breeds. a little bit more of the silo mentality. of like. i have a problem it's almost like. negative to talk to somebody else or. something like that but robotics folks. are just very collaborative very. friendly just and there's also an energy. of like you get to confront the physics. of reality often which is. humbling. and also exciting so it's humbling when. it it fails and exciting when it finally. it's like the purity of the passion you. got to remember that like right now like.

robotics and ais like just all the rage. and autonomous vehicles and all this. like 15 years ago and 20 years ago. like. it wasn't that deeply lucrative people. went into robotics they did it because. they were like thought it was just the. coolest thing in the world to like make. physical things intelligent in the real. world and so there's like a raw passion. where they went into it for the right. reasons and so forth and so it's really. great space and that organizational. challenge by the way like um when you. think about the challenges in av we talk. a lot about the technical challenges the.

organizational challenge is through the. roof where. um you think about the challenge the. what it takes to build an av system and. you have companies that are now. thousands of people. and um you know you look at other really. hard technical problems like an. operating system it's pretty well. established like you kind of know that. there's a file system there's virtual. memory there's this there's that there's. like. caching and like and there's like a. really reasonably well established. modularity and apis and so forth and so.

you can kind of like scale it in an. efficient fashion that doesn't exist. anywhere near to that level of maturity. in autonomous driving right now. and tech stacks are being reinvented. organizational structures are being. reinvented you have problems like. pedestrians that are not isolated. problems they're part sensing part. behavior prediction part planning part. evaluation and. like one of the biggest challenges is. actually how do you solve these problems. where the mental capacity of a human is. starting to get strained on how do you. organize it and think about it where.

you know you have this like. multi-dimensional matrix that needs to. all work together and so. that makes it kind of cool as well. because it's not like solved at all uh. from you know like what what is what. does it take to actually scale this. right and then you look at like other. gigantic challenges that have you know. that have been success successful and. are way more mature. there's a stability to it and like maybe. the autonomous vehicle space will get. there but right now just as many uh. technical challenges as they are they're.

like organizational challenges and how. do you like solve these problems that. touch on so many different areas and. efficiently tackle them. while. like maintaining progress among all. these constraints um while scaling. by way of advice. what advice would you give to uh. somebody thinking about doing a robotics. startup you mentioned cosmo somebody. that wanted to carry the cosmo flag. forward the anki flag forward. looking back at your experience.

looking forward to the future that will. obviously have such robots what advice. would you give to that person yeah it. was the greatest experience ever and. it's like there's something you there's. things you learn. navigating a startup that you'll never. like you you it was very hard to. encounter that in like a typical kind of. work environment and um. and it's just it's wonderful you got to. be ready for it it's not as good like. you know the the glamour of a startup. there's just like just brutal emotional. swings up and down and so um having. co-founders actually helps a ton like i.

would not cannot imagine doing it solo. but having at least somebody where on. your darkest days you can kind of like. really openly just like have that. conversation and you know lean on to. somebody that's that's in the thick of. it with you helps a lot what i would say. what was the nature of darkest days and. the emotional swings is it worried about. the funding is it worried about whether. any of your ideas. are any good or ever were good is it. like the self-doubt uh is it like. facing new challenges that have nothing.

to do with the technology like. organizational human resources that kind. of stuff what yeah you come from a world. in school where. you feel that uh you put in a lot of. effort and you'll get the right result. and input translates proportional to. output and. you know you need to solve the set or do. whatever and just kind of get it done. now phd tests out a little bit but at. the end of the day you put in the effort. you tend to like kind of come out with. your enough results to you kind of get a. phd. in the startup space like.

you know like you could talk to 50. investors and they just don't see your. vision and it doesn't matter how hard. you kind of tried and pitched you could. uh work incredibly hard and you have a. manufacturing defect and if you don't. fix it you're gonna you're out of. business um you need to raise money by a. certain date and there's a you got to. have this milestone in order to like. have a good pitch and you do it you have. to have this talent and you just don't. have it inside the company or um. you know you have to get 200 people or. however many people kind of like along.

with you and kind of buy in the journey. um you're like disagreeing with an. investor and they're your investors so. it's just like you know it's like you. there's no walking away from it right so. um and it tends to be like those things. where you just kind of get clobbered in. so many different ways that like things. end up being harder than you expect and. it's like such a gauntlet. but you learn so much in the process and. there's a lot of people that actually. end up rooting for you and helping you. like from the outside and you get good. great mentors and you like get find. fantastic people that step up in the. company and you have this like magical.

period where everybody's like. it's life or death for the company but. like you're all fighting for the same. thing and it's the most satisfying kind. of journey ever um the things that make. it easier and that i would recommend is. like be really really thoughtful about. the. the application like there's a there's a. saying of like kind of you know team and. execution and market and like kind of. how important are each of those um and. oftentimes the market wins and you come. at it thinking that if you're smart. enough and you work hard enough and. you're like have the right talented team.

and so forth like you'll always kind of. find a way through and um it's. surprising how much dynamics are driven. by the industry you're in and the timing. of you entering that industry um and so. just uh waymo is a great example of it. there is. i don't know if there'll ever be another. company or suite of companies that. has raised and continues to spend so. much money at such an early uh phase of. revenue generation and product and. productization um the you know from a p.

l standpoint uh like it's it's a. anomaly like by any measure of any. industry that's ever existed um except. for maybe the u.s space program uh like. right uh like but. it's like uh. multiple trillion dollar opportunities. which is so unusual to find that size of. a market that. just the progress that shows the. de-risking of it you could apply. whatever discounts you want off of that. trillion-dollar market and it still. justifies the investment that is. happening because like being successful.

in that space makes all the investments. feel trivial now by the same consequence. like. the size of the market the size of the. target audience the ability to capture. that market share how hard that's going. to be who the incumbent's like that's. probably one of the lessons i appreciate. like more than anything else where like. those things really really do matter. and um oftentimes can dominate the. quality of the team or execution because. if you. miss the timing or you do it in the. wrong space you run into like the. institutional kind of headwinds of a.

particular environment like let's say. you have the greatest idea in the world. but you barrel into healthcare but it. takes 10 years to innovate in healthcare. because of a lot of challenges right. like there's fundamental. uh laws of physics that you have to. think about and so um the combination of. like anki waymo kind of drives that. point home for me where you can do a ton. if you have the right market the right. opportunity the right way to explain it. and you show the progress in the right. sequence. it actually can really significantly. change the course of your. journey and startup how much of it is.

understanding the market and how much of. it's creating a new market so how do you. think about. like space robotics is really. interesting you said exactly right the. space of applications is small. yeah. you know relative to the cost involved. so how much is like truly revolutionary. thinking. about like what is the application. and then. yeah but so creating something that. didn't exist it didn't really exist like.

this is pretty obvious to me the whole. space of home robotics just every. everything that cosmo did i guess you. could talk to it as a toy and people. will understand it picazo is much more. than a toy yeah. and. i don't think people fully understand. the value of that you have to create it. and the product will communicate it like. just like the iphone. nobody understood the value. of of no keyboard and a thing that's. that can do web browsing i don't think.

they understand the value of that until. you create it. yeah having a foot and a door in an. entry point still helps because at the. end of the day like an iphone replaced. your phone and so it had a fundamental. purpose and all these things that it did. better right sure and so then you could. do abc on top of it and uh and then like. you even remember the early commercials. where it's always like one application. of what he could do and then you get a. phone call right and so that was. intentionally sending a message. something familiar but then like yes you. can send a text message you can listen. to music you can surf the web right and.

so. you know autonomous driving obviously. anchors on that as well you don't have. to explain to somebody the functionality. of an autonomous truck right like. there's nuances around it but the. functionality makes sense um. in the home. you have a fundamental advantage like we. always thought about this because it was. so painful to explain to people what our. products did and how like how to. communicate that super cleanly. especially when something was so. experiential and so you compare like. anki to nest. nest um. had some beautiful products.

where they. started scaling and like actually find. like really great success and they had. like really clean and beautiful. marketing messaging because they. anchored on reinventing existing. categories where it was a smart. thermostat right and uh like and so you. you kind of are able to. um take what's familiar anchor that. understanding and then explain what's. what's better about it that's funny. you're right cosmo is like totally new. thing like what what is this thing. because we struggle we spent like a lot.

of money on marketing we had a heart. like we fought we actually had far. greater efficiency on cosmo than um. anything else because we found a way to. capture the emotion in some little. shorts to kind of lean into the. personality in our marketing and it. became viral where like we had these. kind of videos that would like go and. get like hundreds of thousands of views. and like kind of like get spread and. sometimes millions of views and so um. but it was like really really hard um. and so finding a way to kind of like. anchor on something that's familiar but. then grow into something that's not um.

is an advantage but then again like you. don't have like there's successes. otherwise like alexa never had a comp. right uh. you could argue that that's very novel. and very new and um. and there's a lot of other examples that. kind of created a. kind of a category out of like kiva. systems i mean they like came in and. they like. uh enterprise is a little easier because. if you can uh it's less susceptible to. this because if you can argue a clear. value proposition it's a more logical. conversation that you can have um with. customers it's not it's a little bit.

less emotional and um kind of subjective. but yeah in the home you have to. yeah so like a home robot it's like what. does that mean yeah and so then you. really have to be crisp about the value. proposition and what like. really makes it worth it like and and we. by the way went to that same order we. almost like. we almost hit a wall coming out of 2013. where. we were so big on explaining why our. stuff was so high-tech and all the kind. of like great technology in it and how. cool it is and so forth um to having to. make a super hard pivot on why is it fun.

and why did like does the random kind of. family of. four need this right like so it's. learnings but. that's that's the challenge and i think. like robotics tends to sometimes fall. into the new category problem but then. you gotta be really crisp about why. it needs to exist well i think some of. robotics depending on the category. depending on the application. is a little bit of a marketing. this uh challenge and i don't i don't.

mean. i mean. it's it's the kind of marketing that. weimo is doing that tesla is doing is. like. showing off incredible. engineering incredible technology. but convincing like you said a family of. four that this this will this is like. this is transformative for your life. this is this is this is fun this is you. don't care about tech isn't your thing. they don't they really don't like they. need to know why they want it so some of. that is just marketing yeah that's why. like roomba like um yes they didn't you.

know like. go and you know have this like. you know huge huge con you know ramp. into like the entirety of like kind of a. robotics and so forth but like they. built a really great business and um uh. in a vacuum cleaner world and like. everybody understands what a vacuum. cleaner is um most people are annoyed by. doing it um and now you have one that. like kind of does it itself. uh yeah various degrees of quality but. that is so compelling that like it's. easier to understand and like uh and. they had a very kind of and i think they.

have like 15 of the vacuum cleaner. market so it's like pretty successful. right. i think we need more of those um types. of thoughtful stepping stones in. robotics but the opportunities are. becoming bigger because. hardware's cheaper computes cheaper. clouds cheaper and ai's better so. there's a lot of opportunity if we zoom. out from specifically startups and. robotics what advice do you have. to uh high school students college. students about. career. and living a life that you can be proud. of you lived one heck of a life you're.

very successful in several domains. um. if you can convert that into a. generalizable potion what advice would. you give yeah it's a very good question. so it's very hard to. go into a space that you're not. passionate about and push. like push hard enough to be you know to. like. maximize your potential uh in it and so. there's a um. there's always kind of like the saying. of like okay follow your passion great.

try to find the overlap of where your. passion overlaps with like a growing. opportunity and need in the world where. it's not too different than the startup. kind of argument that we talked about. where um if you are where your passion. meets the market right you know i mean. like because it's like uh um it's a you. know that's a beautiful thing where like. you can do what you love but it's also. just opens up tons of opportunities. because the world's ready for it right. like and so um and so like if you're. interested in technology um that might. point to like go and study machine. learning because you don't have to. decide what career you're going to go.

into but it's going to be such a. versatile space that's going to be at. the root of like everything that's going. to be in front of us that. you can have. eight different careers in different. industries. and be an absolute expert in this like. kind of tool set that you wield that can. go and be applied um and that by the way. that doesn't apply to just technology. right it's uh it could be the exact same. thing if you want to um you know the. same thought process apprised to design. to marketing to um you know to sales to. anything but um that versatility where.

you like um. when you're in a space that's gonna. continue to grow um it's just like what. company do you join one that just is. going to grow and the growth creates. opportunities where the surface area is. just going to increase and the problems. will never get stale and you can have. you know many like and so you go into a. career where you have that sort of. growth in the in the world that you're. in. you end up having. so much more opportunity that. organically just appears and you can. then have more shots on goal to find. like that killer overlap of timing and.

passion and skill set and point in life. where you can like. you know just really be motivated and. fall in love with something um. and then at the same time like uh find a. balance like there's been times in my. life where i worked like a little bit. too obsessively and you know and crazy. and uh and i you know think we kind of. like tried to correct that you know kind. of the right opportunities but you know. i think i probably appreciate a lot more. now. friendships that go way back um you know. family and things like that and um. and i i'm kind of have the personality. where i could use like i have like so.

much desire to really try to optimize. like you know what i'm working on that i. can easily go to kind of an extreme and. now i'm trying to like kind of find that. balance and make sure that i have. the friendships the family like. relationship with the kids everything. that like i don't. uh i push really really hard but it kind. of find a balance and. and i think. people can be happy on. actually many kind of extremes on that. spectrum but it's easy to kind of. inadvertently make a choice. by how how you approach it that then.

becomes really hard to unwind um. and so being very thoughtful about. kind of all of those dimensions makes a. lot of sense and so. um to come those are all interrelated um. but at the end of the day oh love. passion and love yeah love towards you. said uh yeah family friends family and. hopefully. one day. if your work pans out. boris. is love towards robots. not the creepy kind of good guy that's a. good kind. just just friendship and yeah and fun.

just yeah it's like another dimension to. just how we interface with the world. yeah. of course you're one of my favorite. human beings roboticists you've created. some incredible robots and i think. inspired countless people. and like i said. i hope cosmo i hope you work with anki. lives on and um i can't wait. to see what you do with waymo i mean. that's if we're talking about artificial. intelligence technology that has the. potential to revolutionize.

so much of our world. that's it right there so thank you so. much for the work you've done and thank. you for spending your valuable time. talking with me thanks alex. thanks for listening to this. conversation with boris sofman to. support this podcast please check out. our sponsors in the description and now. let me leave you some words from isaac. asimov. if you were to insist i was a robot. you might not consider me capable of. love. in some mystic human sense.

thank you for listening and hope to see. you next time. you.