Jitendra Malik: Computer Vision | Lex Fridman Podcast #110

the following is a conversation with. jitendra malik. a professor at berkeley and one of the. seminal figures in the field of computer. vision. the kind before the deep learning. revolution and. the kind after he has been cited. over 180 thousand times and has mentored. many world-class researchers in computer. science. quick summary of the ads two sponsors. one new one. which is better help and an old goody. expressvpn please consider supporting.

this podcast by going to betterhelp.com. lex and signing up at expressvpn.com. lexpod click the links buy the stuff. it really is the best way to support. this podcast and the journey i'm on. if you enjoy this thing subscribe on. youtube review it with 5 stars on apple. podcast. support it on patreon or connect with me. on twitter. at lex friedman however the heck you. spell that. as usual i'll do a few minutes of ads.

now and never neons in the middle that. can break the flow of the conversation. this show is sponsored by better help. spelled h-e-l-p help. check it out at betterhelp.com lex. they figure out what you need and match. you with a licensed professional. therapist. in under 48 hours it's not a crisis line. it's not self-help it's professional. counseling done securely. online i'm a bit from the david goggins. line of creatures.

as you may know and so have some demons. to contend with. usually on long runs or all nights. working. forever and possibly full of self-doubt. it may be because i'm russian. but i think suffering is essential for. creation. but i also think you can suffer. beautifully in a way that doesn't. destroy you. for most people i think a good therapist. can help in this. so it's at least worth a try check out. their reviews. they're good it's easy private.

affordable. available worldwide you can communicate. by text anytime. and schedule weekly audio and video. sessions. i highly recommend that you check them. out at betterhelp.com. lex this show is also sponsored by. expressvpn get it at expressvpn.com. to support this podcast and to get an. extra three months. free on a one-year package i've been. using expressvpn for many years.

i love it i think expressvpn is the best. vpn out there they told me to say it. but it happens to be true it doesn't log. your data. it's crazy fast and it's easy to use. literally just. one big sexy power on button. again for obvious reasons it's really. important that they don't log your data. it works on linux and everywhere else. too but really. why use anything else shout out to my. favorite flavor of linux ubuntu mate.

2004 once again get it at. expressvpn.comlexpod. to support this podcast and to get an. extra three months free and a one year. package. and now here's my conversation with. jitendra. in 1966 seymour papper. at mit wrote up a proposal called the. summer vision project to be given. as far as we know to 10 students to work. on and solve that summer.

so that proposal outlined many of the. computer vision tasks we still work on. today. why do you think we underestimate and. perhaps we did underestimate and perhaps. still underestimate. how hard computer vision is because. most of what we do in vision we do. unconsciously or subconsciously. in human vision in human vision so that. gives us. this that effortlessness gives us the. sense that oh. this must be very easy to implement on a.

computer. now this is why. the early researchers in ai got it so. wrong. however if you go into neuroscience or. psychology. of human vision then the complexity. becomes very clear. the fact is that a very large part of. the. the cerebral cortex is devoted to visual. processing. i mean and this is true in other. primates as well. so once we looked at it from a.

neuroscience or psychology perspective. it it becomes quite clear that the. problem is very challenging and it will. take some time. you said the higher level parts are the. harder parts. i think vision appears to to be easy. because. uh most of what visual processing. is subconscious or unconscious right. so we underestimate the difficulty. whereas. uh when you are.

like proving a mathematical theorem or. playing chess. the difficulty is much more evident so. because it is your conscious brain which. is processing. uh various aspects of the. problem-solving. behavior whereas in vision all this is. happening but it's not in your. awareness it's in your it's operating. below that. but it's it still seems strange yes. that's true but it seems strange that.

as computer vision researchers for. example. the community broadly is time and time. again makes the mistake of um. thinking the problem is easier than it. is or maybe it's not a mistake. we'll talk a little bit about autonomous. driving for example how hard of a vision. task that is. it do do you think i mean what. is it just human nature or is there. something fundamental to the vision. problem that we we underestimate.

we're still not able to be cognizant of. how hard the problem is. yeah i think in the early days it could. have been excused because. in the early days all aspects of ai were. regarded as too easy. but i think today it is much less. excusable. and i think why people. fall for this is because of what i call. the fallacy of the successful first. step there are many problems in.

vision where getting 50. of the solution you can get in one. minute getting to 90 percent. can take you a day getting to 99 percent. may take you five years and. 99.99 may be not in your lifetime. i wonder if that's a unique division. that it seems that language people are. not so. confident about so natural language. processing people are a little bit more. cautious about our ability.

to to solve that problem. i think for language people intuit that. we have to be able to do. natural language understanding for. vision. it seems that we're not cognizant or we. don't think about how much. understanding is required it's probably. still an open problem. but in your sense how much understanding. is required to solve vision like. this put another way how much.

something called common sense reasoning. is. required to really be able to interpret. even static scenes yeah so vision. operates at. uh at all levels and there are parts. which are. which can be solved with what we could. call maybe peripheral processing. so in the in the human vision literature. there used to be these terms. sensation perception and cognition. which roughly speaking referred to like.

the front end of processing. middle stages of processing and higher. level of processing. and i think they made a big deal out of. out of this and they wanted to just. study only perception and then. dismiss certain certain problems as. being quote cognitive. but really i think these are artificial. divides. the problem is continuous at all level. and there are challenges at all levels. the techniques that we have today.

they work better at the lower and mid. levels of the problem. i think the higher levels of the problem. quote the cognitive levels of the. problem. are there and we. in many real applications we have to. confront them. now how much that is necessary will. depend on the application. for some problems it doesn't matter for. some problems it matters a lot. so i am for example. a pessimist on fully autonomous driving.

in the near future. and the reason is because i think there. will be. that 0.01 percent of the cases. where quite sophisticated cognitive. reasoning is called for. however there are tasks where you can. first of all they are much more they are. robust so in the sense that. error rates error is not so much of a. problem. for example uh uh let's say we are.

you're doing uh. image search you're trying to get images. based on some. some some description some visual. description. we are very tolerant of errors there. right i mean when google image search. gives you some images back and a few of. them are. wrong it's okay it doesn't hurt anybody. there's no. there's not a matter of life and death. but. making mistakes when you're. driving at 60 miles per hour and you.

could potentially kill somebody. is much more important so just for the. for the fun of it since you mentioned. let's go there briefly. about autonomous vehicles so one of the. companies in the space tesla. is work with andre karpathy and elon. musk are working on. a system called autopilot which is. primarily a vision-based system with. eight cameras. and uh basically a single neural network. a multi-task neural network.

they they call it hydro net multiple. heads. so it does multiple tasks but is forming. the same representation. at the core do you think driving can be. converted. in this way to uh purely a vision. problem. and then solved within you with learning. or even more specifically in the current. approach. what do you think about what tesla. autopilot team is doing. so the way i think about it is that. there are certainly.

subset subsets of the visual based. driving problem which are quite solvable. so for example driving in freeway. conditions. is quite a solvable problem i think. there were demonstrations of that going. back to the 1980s by. someone called ernst stickmans in munich. in the 90s there were approaches from. carnegie mellon there were approaches. from. our team at berkeley in the 2000s there. were approaches from stanford.

and so on so autonomous driving in. certain settings is very doable. the challenge is to have an autopilot. work. under all kinds of driving conditions. at that point it's not just a question. of vision. or perception but really also of control. and dealing with all the edge cases. so where do you think most of the. difficult cases. to me even the highway driving is an. open problem because.

uh it applies the same 50 90 95 99 rule. or the first step the fallacy of the. first step i forget how you put. it we fall victim to i think even. highway driving has a lot of elements. because to solve autonomous driving you. have to completely relinquish. the the fat help of a human being. you're always in control so that you're. really going to feel the edge cases. so i i think even highway driving is. really difficult but. in terms of the general driving task do.

you think. vision is the fundamental problem or is. it. also your action the the interaction. with the environment. the ability to uh and then like the. middle ground i don't know if you put. that under vision which is. trying to predict the behavior of others. which is a little bit. in the world of understanding the scene. but it's also trying to form a model of. the actors in the scene. and predict their behavior yeah i.

include that in vision because. to me perception blends into cognition. and building predictive models of other. agents in the world. which could be other agents could be. people other agents could be other cars. that is part of the task of perception. because. perception always has to uh not tell us. what is now but what will happen. because what's now is boring it's done. it's over with. okay yeah we care about the future.

because we. act in the future and we care about the. past. and as much as it informs what's going. to happen in the future. so i think we have to build predictive. models of of. of behaviors of people and and those can. get quite. complicated so uh uh. i mean uh i i've seen examples of this. in. uh actually i mean i own a tesla. and it has various safety features built.

in. and uh what i see are these examples. where. let's say there is some uh skateboarder. i mean. this i and i i don't want to be too. critical because. obviously this is these are the systems. are always being improved. and any specific criticism i have. maybe the system six months from now. will not have that. that that particular failure mode. so uh it. it had it it had the wrong response and.

it's because it couldn't predict. what what this skateboarder was going to. do. okay and because it really required that. higher level cognitive understanding. of what skateboarders typically do as. opposed to a normal pedestrian. so what might have been the correct. behavior for a pedestrian. a typical behavior for pedestrian was. not the typical behavior for a. skateboarder right yeah. and uh so so therefore.

to do a good job there you need to have. enough data where. you have pedestrians you also have. skateboarders. you've seen enough skateboarders to see. what. uh what kinds of patterns or behavior. they have. so it is it is in principle with enough. data that problem could be solved. but uh i think our current. systems computer vision systems they. need far. far more data than humans do for.

learning those. same capabilities so say that there is. going to be a system that solves. autonomous driving. do you think it will look similar to. what we have today. but have a lot more data perhaps more. compute but the fundamental. architectures involved. like neuro well in the case of tesla. autopilot is. neural networks do you think it will. look similar. in that regard and we'll just have more. data that's a. scientific hypothesis as which way is it.

going to go. uh i will tell you what i would bet on. uh. so and this is at my general. philosophical position on how these. uh learning systems have been. uh what we have found currently very. effective in. computer vision uh with in in the deep. learning paradigm is. sort of tabula rasa learning and tabular. us are learning. in a supervised way with lots and lots. of what's going on.

in the sense that blank slate we just. have the system which is. given a series of experiences in this. setting and then it learns there. now if let's think about human driving. it is not tabular assad learning. so at the age of 16 in high school. uh a teenager goes into uh. goes into driver ed class right and now. at that point they learn but at the age. of 16 they are already visual geniuses.

because from 0 to 16 they have built a. certain repertoire of vision. in fact most of it has probably been. achieved by. age 2 right in in this period of age. up to age 2 they know that the world is. three-dimensional they know how. objects look like from different. perspectives. they know about occlusion they. know about common dynamics of humans and. other bodies. they have some notion of intuitive.

physics so they. they built that up from their. observations and interactions. in early childhood and of course. reinforced through. their their growing up to age 16. so. then. at age 16 when they go into driver ed. what are they learning they're not. learning afresh the visual world. they have a mastery of the visual world. what they are learning. is control okay they are learning how to. be smooth.

about control about steering and brakes. and so forth. they're learning a sense of typical. traffic situations. now the the that education process. can be quite short because they are. coming in as visual geniuses. and of course in their future they're. going to encounter situations which are. very novel. right so during my driver ed class. that i may not have had to deal with a. skateboarder i may not have had to deal.

with a truck. driving in front of me who's from. who's where the back opens up and some. junk gets dropped from the truck. and i have to deal with it right but i. can deal with this. as a driver even though i did not. encounter this in my driver at. class and the reason i can deal with it. is because i have all this. general visual knowledge and expertise. and uh do you think the learning. mechanisms we have today. can do that kind of long-term.

accumulation of knowledge. or do we have to uh do some kind of. you know in the the the work that led up. to expert systems with knowledge. representation. you know the broader field of what of. artificial intelligence. uh worked on this kind of accumulation. of knowledge. do you think neural networks can do the. same i think uh. i don't see any in principle problem. with neural networks doing it. but i think the learning techniques.

would need to evolve significantly. so the current uh the current. learning techniques that we have yeah is. our supervised learning you're given. lots of examples. xiy pairs and you you learn the. functional mapping between them. i think that human learning is far. richer than that. it includes many different components. there are. there is a a child explores the world. and sees as for example a child.

takes an object and manipulates it. in his or her hand and therefore gets to. see the object from different points of. view. and the child has commanded the movement. so that's a kind of learning data but. the learning data has been. arranged by the child and this is a very. rich. kind of data the child can do various. experiments with the world so.

so there are many aspects of sort of. human learning and these have been. studied in. in child development by psychologists. and they what they tell us is that. supervised learning is a very small part. of it. there are many different aspects of. learning. and what we would need to do is to. develop models of. all of these and then. train our systems in that with that kind. of.

uh protocol so new new methods of. learning. yes some of which might imitate the. human brain but you also. in your talks have mentioned some of the. compute side of things. the in terms of the difference in the. human brain or referencing marvik. hans marvel the so. do you do you think there's something. interesting valuable to consider about. the difference. in the computational power of the human. brain versus. the computers of today in terms of.

instructions. per second yes so if we go back. uh so so this is a point i've been. making for 20 years now. and i think once upon a time the way i. used to. argue this was that we just didn't have. the computing power of the human brain. our computers were uh were not. quite there and i mean there is a. well well-known trade-off which we know. that. the that neurons are slow compared to.

transistors but uh but we have a lot of. them and they have a very high. connectivity. whereas in silicon you have much faster. devices transistors switch at. on the order of nanoseconds but the. connectivity is usually smaller. right at this point in time i mean we. are now talking about. 2020 we do have if you consider the. latest gpus and so on. amazing computing power and if we look.

back at enhanced modex type of. calculations which he did in the 1990s. we may be there today in terms of. computing power comparable to the brain. but it's not in the of the same style. it's of a very different style. so i mean for example the the style of. computing that we have in our gpus. is far far more power hungry than. the style of computing that is there in.

the human brain or other. biological uh entities. yeah and that the efficiency part is uh. we're gonna have to solve that in order. to build actual real world systems. of large scale let me ask sort of. the high level question step taking a. step back. how would you articulate the general. problem of computer vision. does such a thing exist so if you look. at the computer vision conferences and. the work that's been going on. it's often separated into different.

little segments. breaking the problem of vision apart. into whether segmentation. 3d reconstruction object detection. i don't know image capturing whatever uh. there's benchmarks for each. but if you were to sort of. philosophically say what is. the big problem of computer vision does. such a thing exist. yes but it's not in isolation so. if we have to so for all.

intelligence tasks i. always go back to sort of biology or. humans and if we think about. vision or perception in that setting we. realize that. perception is always to guide action. perception. in a for a biological system does not. give any benefits. unless it is coupled with action so we. can go back. and think about the first multicellular. animals.

which arose in the cambrian era you know. 500 million years ago. and uh these animals could move. and they could see in some ways and. their two. activities helped each other because uh. uh how does movement help movement. helps that because you can get food in. different places. but you need to know where to go and. that's really about. perception or seeing i mean i mean. vision is. perhaps the single most perception sense.

but. all the others are equally are also. important so. uh so perception and action kind of grow. go together. so earlier it was in these very simple. feedback loops. which were about uh finding food. or avoiding becoming food if there's a. predator running uh. trying to you know eat you up. and and so forth so so we must at the. fundamental level connect. perception to action then.

as we evolved uh perception became more. and more sophisticated. because it served many more purposes and. uh so today we have what seems like a. fairly general purpose capability. which can look at the external world and. build and. a model of the external world inside the. head. we do have that capability that model is. not perfect. and psychologists have great fun in. pointing out the ways in which.

the model in your head is not a perfect. model of the external world. and they have create various illusions. to. show the ways in which it is imperfect. but. it's amazing how far it has come from a. very simple. perception action loop that you exists. in you know. an animal 500 million years ago once we. have this. these very sophisticated visual systems. we can then. impose a structure on them it's we as.

scientists who are imposing that. structure. where we have chosen to characterize. this part of the system as this. code module of object detection or quote. this module of 3d reconstruction. what's going on is really all of these. processes are running. simultaneously and uh. and and they are running simultaneously. because originally their purpose was. in fact to help guide action so.

as a guiding general statement of a. problem do you think. we can say that the the general problem. of computer vision. you said in humans it was tied to action. do you think we should also say that. ultimately the the goal. the problem of computer vision is to. sense the world. in the way that helps you act in the. world. yes i think that's the most fundamental. uh.

that's the most fundamental purpose. we have by now hyper evolved. so we have this visual system which can. be used for other things. for example judging the aesthetic value. of a painting. and this is not guiding action maybe. it's guiding action in terms of how much. money you will put in your auction bid. but that's a bit stretched. but the basics are in fact in terms of. action. but we have we've evolved.

really this hyper uh we have hyper. evolved our visual system. actually just too uh sorry to interrupt. but perhaps it is. fundamentally about action you kind of. jokingly said about spending. but perhaps the capitalistic. uh drive that drives a lot of the. development in this world. is is about to exchange your money and. the fundamental action is money if you. watch. netflix if you enjoy watching movies. you're using your perception system to.

interpret the movie. ultimately your enjoyment of that movie. means you'll subscribe to netflix. so the action is this uh. this extra layer that we've developed in. modern society perhaps this is. fundamentally tied to the action of. spending money. well certainly with respect to uh. you know interactions with firms so so. in this homo economics role. when you're interacting with firms it. does become. uh it does become that that's what else.

is there. uh that was a rhetorical question okay. so. to to linger on the division between the. static and the dynamic. so much of the work in computer vision. so many of the breakthroughs that you've. been a part of. have been in the static world in. looking at static images and then you've. also. worked on starting but it's a much. smaller degree the community is looking. at dynamic and video.

at dynamic scenes and then there is. robotic vision. which is dynamic but also where you. actually have a robot in the physical. world. interacting based on that vision. which problem is harder. the the the intuit sort of the the. trivial first answers. well of course one image is harder but. so if you look at a deeper question. there. are we um what's the term cutting.

ourselves. cutting ourselves at the knees or like. making the problem harder by focusing on. the images that's a fair question i. think. sometimes we we can simplify our problem. so much. that we essentially lose. part of the juice that could enable us. to solve the problem. and one could reasonably argue that to. some extent this happens when we go from. video to single images.

now historically uh you have to consider. the limits of. imposed by the competition capabilities. we had. so if we many of the choices made in the. computer vision community. uh through the 70s 80s 90s. can be understood as. choices which were forced upon us by. the fact that we just didn't have access.

to compute. enough compute not enough memory none of. hard drives not. exactly not enough not enough compute. not enough storage. so so think of these choices so one of. the choices is. focusing on single images rather than. video okay. clear questions storage and compute. we had to focus on we did we. used to detect edges and throw away the. image right so you have an image. which i say 256 by 256 pixels and. instead of keeping around the grayscale.

value what we did was we detected edges. find the places where the brightness. changes a lot. so now that and now and then throw away. the rest. so this was a major compression device. and the hope was that this makes it. that you can still work with it and the. logic was humans can interpret a line. drawing. and uh and yes and this will save us a. competition so many of the choices were. dictated by that.

i think uh today. we are no longer detecting edges right. we. process images with convnets because we. don't need to we don't have that. those compute restrictions anymore now. video is still. under studied because video compute is. still quite challenging. if you are a university researcher i. think. video computing is not so challenging if. you are at. google or facebook or amazon still super. challenging i've.

just spoke with the vp of engineering. google head of. the youtube search and discovery and. they still struggle doing stuff on. video it's very difficult except doing. except using techniques that are. essentially the techniques you used in. in the 90s some very basic computer. vision techniques. no that's when you want to do things at. scale so if. you want to operate at the scale of all. the content of youtube it's very. challenging and there's similar issues. in. facebook but as a researcher you.

you have you have more uh you know. opportunities. you can train large you know that works. with relatively large. uh video data sets yeah yes so i think. that. this is part of the reason why we have. so emphasized static images. i think that this is changing and over. the next few years. i see a lot more progress happening in. in video so i have this generic. statement that. to me video recognition feels like 10.

years behind. object recognition and you can quantify. that because. you can take some of the challenging. video data sets and. their performance on action. classification is like say 30. which is kind of what we used to have. around. 2009 in object detection you know so. it's like about 10 years behind. and uh whether it'll take 10 years to. catch up is a different question. hopefully it will take less than that.

let me ask a similar question i've. already asked but once again so for. dynamic scenes. do you think do you think some kind of. injection. of knowledge basis and reasoning is. required. to help improve like action recognition. like if if if um. if we solve the general action. recognition problem. what do you think the solution would. look like it's another way yeah.

so i i completely. agree that knowledge is called for and. that knowledge can be. quite sophisticated so the way i would. say it is that. perception blends into cognition and. cognition brings in. issues of memory and. this notion of a schema from psychology. which is. uh let me use the classic example which. is. you go to a restaurant right now the. things that. happen in a certain order you walk in.

somebody takes you to a table. a waiter comes gives you a menu. takes the order food arrives eventually. a. bill arrives etc etc this is a classic. example of ai from the 1970s. uh it was called there was the term. frames and. scripts and schemas these are all quite. similar ideas. okay in the 70s the way.

the ai of the time dealt with it was by. build hand coding this. so they hand coded in this notion of a. script and the various. stages and the actors and so on and so. forth. and use that to interpret for example. language. i mean if there's a description of a of. a story involving. some people eating at a restaurant there. are way all these. inferences you can make because you know. what happens typically at a restaurant.

so i think this kind of uh. this kind of knowledge is absolutely. essential so i think. that when we are going to do long-form. video understanding. we are going to need to do this i think. the kinds of technology that we have. right now with. 3d convolutions over a couple of seconds. of clip or video. it's very much tailored towards. short-term video understanding. not that long-term understanding. long-term understanding. requires a notion of.

this notion of schemas that i talked. about perhaps some notions of. goals intentionality functionality. and so on and so forth now. how will we bring that in so we could. either revert back to the 70s and say. okay i'm going to hand code in. a script or we might. try to learn it so i. tend to believe that we have to find.

learning ways of doing this. because i think learning ways to land up. being more robust. and there must be a learning version of. the story because. uh children acquire a lot of this. knowledge. by uh sort of just observation so. at no moment in a child's life there's a. it's possible but i think it's not so. typical that. somebody that a mother coaches a child. through all the stages of what happens. in a restaurant.

they just go as a family they they. they go to the restaurant they eat come. back and the child goes through 10 such. experiences. and the child has has got a schema of. what happens when you go to a restaurant. so we somehow need to we need to provide. that capability to our systems. you mentioned the following line from. the end of the alan turing paper. uh computing machinery and intelligence. that many people. like you said many people know and very. few have read.

where he proposes the turing test this. is this is how you know because it's. towards the end of the paper. instead of trying to produce a program. to simulate the adult mind. why not rather try to produce one which. simulates the child's. so that's a really interesting point if. i think about the benchmarks we have. before us the the tests. of our computer vision systems they're. often kind of trying to. get to the adult so what kind of. benchmarks should we have.

what kind of tests for computer vision. do you think we should have. that mimic the child's in computer. vision yeah. i think we should have those and we. don't have those today. and i think uh the part of that. the challenge is that we should really. be collecting data. of the type that a child uh that the. child experiences. right so that gets into issues of you. know privacy and so on and so forth. but there are attempts in this direction.

to. sort of try to collect the kind of data. that a child. encounters growing up so what's the. child's linguistic. environment what's the child's visual. environment. so if we could collect that kind of data. and then develop learning schemes based. on that data. that would be one way to do it i. i think that's a very promising. direction myself there might be people. who would argue.

that we could just short circuit this in. some way and. uh sometimes we have. imitated uh we have not. we have had success by not imitating. nature in detail so. the usual example is airplanes right we. don't build flapping winds. flapping wings so uh. yes that's uh that's one of the points. of debate. uh in my mind i i i would i would bet on.

this this learning like a child approach. so one of the fundamental aspects of. learning like a child is the. interactivity. so the child gets to play with the data. set it's learning from. yes it's against the select i mean you. can call that active learning you can. you know in the machine learning world. you can call it a lot of terms. what are your thoughts about this whole. space of being able to play with the. data set or select what you're learning. yeah so i think that uh i.

i believe in that and i think that we. could achieve it in in two ways and i. think we should use both. so one is uh actually. real robotics right so real uh. you know physical embodiments of agents. who are interacting with the world and. they have a physical body with. dynamics and mass and moment of inertia. and friction and all the rest and you.

learn your body the robot learns its. body by. doing a series of. actions the second is that simulation. environments. so i think simulation environments are. getting much much better. in my in my life in. facebook ai research our group has. worked on something called habitat. which is a simulation environment. which is a visually photorealistic.

environment of. you know places like houses or interiors. of. various urban spaces and so forth and as. you move. you get a picture which is a pretty. accurate picture. so uh i i can now uh you can imagine. that. subsequent generations of these. simulators will be accurate not just. visually but with respect to. you know forces and masses and. haptic interactions and so on.

and uh then then we have that. environment to play with. i think that let me state one reason why. i think. this active being able to act in the. world is. important i think that this is one way. to break. the correlation versus causation barrier. so this is something which is of a great. deal of interest these days i mean. people like judea pearl have. talked a lot about uh why.

that we are neglecting causality and he. describes the entire set of successes of. deep learning as just curve fitting. right because it's uh but i i don't. quite agree. about as a troublemaker he is but uh. causality. is important but causality is not. is not like a single silver bullet it's. not like one single principle there are. many different aspects here. and one of the ways in which uh.

one of our most reliable ways of. establishing causal links and this is. the way. for example the the medical community. does this is. randomized control trials so you have. you. you pick some situation and now in some. situation you perform an action and. for certain others you don't. right so so you have a control. experiment well the child is in fact. performing controlled experiments all. the time. right right right okay small scale and.

in a small scale and. but but that is a way that the child. gets to. build and refine its causal models of. the world. and my colleague alison gopnik has. together with a couple of authors. co-authors has this book called the. scientist in the crib. referring to children so i like the part. that i like about that is. the scientist wants to do wants to build. causal models. and the scientist does control.

experiments and i think the child is. doing that. so to enable that we will need to. have these these active experiments. and i think this could be done some in. the real world and some in simulation so. you have hope for simulation. i have a hopeless solution that's an. exciting possibility if we can get to. not just photo realistic but what's that. called. life realistic yeah uh simulation. so you don't see any fundamental.

blocks to why we can't eventually. simulate. the the principles of what it means to. exist in the world. as a physical i i don't see any. fundamental problems there i mean. and look the computer graphics community. has come a long way. right so the in the early days back. going back to the 80s and 90s they were. they were focusing on visual realism. right and then they could do the easy. stuff but they couldn't do stuff like. hair or fur and so on. okay well they managed to do that then.

they couldn't do physical. actions right like there's a bowl of. glass and it falls down and it shatters. but then they could start to do pretty. realistic models of that. and so on and so forth so the graphics. people have shown that they can do. this forward direction not just for. optical interactions but also for. physical interactions. so i think uh of course some of that is. very computer intensive but. i think by and by we will find ways of.

making our models ever more realistic. you break vision apart into in one of. your presentations. early vision static scene understanding. dynamics and understanding. and raise a few interesting questions i. thought i could just throw some. some at you just to see if you want to. talk about them. so early vision so it's what is it. you said um sensation. perception and cognition so is this a. sensation yes.

what can we learn from image statistics. that we don't already know. so at the lowest level what um. what can we make from just this the the. statistic the basics so there were the. variations in the rock pixels the. textures and so on. yeah so what we seem to have learned is. uh. uh uh is that there's a lot of. redundancy in these images and. as a result we are able to do a lot of.

compression. and and this compression is very. important in biological settings right. so you might have ten to the eight. photoreceptors and only ten to the six. fibers in the optic nerve so you have to. do this compression by. a factor of hundreds to one and. uh and uh so there are analogs of that. which are happening in. in our neural net artificial neural. network that's the early layer so you. think. there's a lot of compression that can be. done in the beginning.

yeah just just the statistics yeah. um how much. how much well so i mean the the way to. think about it is. just how successful is image compression. right and we we and there are and that's. been done with. older technologies but it can be done. with there are. several companies which are trying to. use. sort of these more advanced neural. network type techniques for compression.

both for static images as well as for. for video. one of my former students has a company. which is trying to do. stuff like this and. i think i think that they are showing. quite. interesting results and i think that. that's all. the success of that's really about image. statistics and video statistics but. that's still not doing. compression of the kind when i see a. picture of a cat. all i have to say is it's a cat that's.

another semantic kind of complication. yeah so this is this is at the lower. level right so we are we are we as i. said yeah. that's focusing on low level statistics. so to linger on that for a little bit. uh you mentioned how far can bottom-up. image segmentation go. and in general what you mentioned. that the central question for scene. understanding is the interplay of. bottom-up and top-down information maybe. this is a good time. to elaborate on that maybe define what.

is. what is up what is top down. in the comments yes the computer vision. uh right that's uh. so today what we have are a are very. interesting systems because they work. completely bottom up. how are they what does bottom bottom-up. mean sorry so bottom-up means in this. case means a feed-forward net neural. network. so starting from the raw pixels yeah. they start from the raw pixels and they. they end up with some something like cat. or not a cat. right so our our systems are running.

totally feed forward. they're trained in a very top-down way. so they're trained by saying okay this. is a cat there's a cat there's a dog. there's a zebra etc. and i'm not happy with either of these. choices fully. we have gone into uh because we have. completely separated these processes. right so there is a so i would like the. uh the process uh.

so what do we know compared to biology. so in biology what we know is that the. processes. in at test time at run time. those processes are not purely feed. forward but they involve feedback. so and they involve much shallower. neural networks. so the kinds of neural networks we are. using in computer vision say a resnet 50. has 50 layers. well in in the brain in the visual. cortex. going from the retina to it maybe we.

have like seven. right so they're far shallower but we. have the possibility of feedback so. there are backward connections. and this might enable us to uh. to deal with the more ambiguous stimuli. for example. so the the biological solution seems to. involve feedback. the solution in in artificial. vision seems to be just feed forward but. with a much deeper network.

and the two are functionally equivalent. because if you have a feedback network. which just has like three rounds of. feedback. you can just unroll it and make it three. times the depth. and create it in a totally feed forward. way. so this is something which i mean we. have written some papers on this. theme but i really feel that this should. this theme should be pursued further. have some kind of recurrence mechanism. yeah. okay the other uh so that so that's uh.

so i. so i want to have a little bit more top. down in the. at test time okay then at training time. we make use of a lot of top-down. knowledge right now. so basically to learn to segment an. object we have to have all these. examples of this is the boundary of a. cat and this is the boundary of a chair. and this is the boundary of a horse and. so on and this is. too much top-down knowledge how do. humans do this we manage to we manage.

with far less supervision. and we do it in a sort of bottom-up way. because for example. we're looking at a video stream and the. horse moves. and that enables me to say that all. these pixels are together. yeah so the gestural psychologists used. to call this. the principle of common fate so there. was a bottom-up. process by which we were able to segment. out these objects. and we have totally focused on this.

top-down training signal. so in my view we have currently solved. it. in machine vision this top-down. bottom-up interaction. but i don't find the solution fully. satisfactory. and i would rather have a bit of both in. at both stages. for all computer vision problems which. is not just segmentation. and and and and the question that you. can ask is. so for me i'm inspired a lot by human. vision and i care about that.

you could be a just a hard-boiled. engineer not give a damn. so to you i would then argue that uh you. would need far less training data. if you could make my uh research agenda. you know fruitful okay so. maybe taking a step into uh segmentation. static scene understanding. what is the interaction between. segmentation and recognition. you mentioned the movement of objects.

so for people who don't know computer. vision. segmentation is this weird activity that. we. that computer vision folks have all. agreed is very important. uh of drawing outlines around objects. versus a bounding box or. and then classifying that object. what's what's the value of segmentation. what is it. as a problem in computer vision how is. it fundamentally different from. detection recognition any other problems.

yeah so i think. uh so so segmentation. enables us to say that. some set of pixels are an object without. necessarily even being able to name that. object or knowing properties of that. object. oh so you mean segmentation purely as. as as the act of separating an object. from its background a blob of uh. of that's united in some way from his.

background yeah so identification if you. were. making an entity out of it and. justification yeah beautifully. so so i think that we have that. capability. and that is that enables us. to uh as we are growing up to. acquire uh names of objects. with very little supervision so suppose. the child. lets posit that the child has this. ability to separate out. objects in the world then when the.

there's a. the mother says pick up your bottle or. the cat's behaving funny today. [Laughter]. the word cat suggests some object and. then the child sort of does the mapping. right right the mother doesn't have to. teach. a specific object labels by pointing to. them. weak supervision works in the context. that you have. the ability to create objects so.

i think that uh so to me that's that's a. very fundamental capability. uh there are applications where this is. very important uh. for example medical diagnosis so in. medical diagnosis. uh you have some uh brain scan i mean. some. this is some work that we did in my. group where you have ct scans of people. who have. had traumatic brain injury and what uh. what the radiologist needs to do is to. precisely delineate various.

places where there might be bleeds for. example. and there's there are clear needs like. that. so they're certainly very practical. applications of computer vision where. segmentation is necessary. but philosophically segmentation. enables the task of recognition. to proceed with much weaker supervision. than we require today. and you think of segmentation as this. kind of task that takes on.

a visual scene and breaks it apart. into into interesting entities yeah. that might be useful for whatever the. task is yeah. and and it is not semantics free so i. think i. i mean it it blends into it involves. perception and cognition it is not it is. not. i i think the mistake that we used to. make in the early days of computer. vision. was to treat it as a purely bottom-up.

perceptual task it is not just that. because we do revise our notion of. segmentation with more experience right. because. for example there are objects which are. non-rigid like animals. or humans and uh i think. understanding that all the pixels of a. human are one entity is actually quite a. challenge. because the parts of the human they can. move independently. and the human wears clothes so they.

might be differently colored. so it's all sort of a challenge you. mentioned the three hours of computer. vision. are recognition reconstruction. reorganization. can you describe these three r's sure. how they interact. yeah so uh so recognition is the easiest. one. because that's uh what i think. people generally think of as computer. vision. achieving these days which is uh labels.

so is this a cat is this a dog is this a. chihuahua i mean you know it could be. very fine grain like. you know specific breed of a dog or a. specific species or bird. or it could be very abstract like animal. but given a part of an image or a whole. image say. put a label on that yeah so that's. that's recognition. reconstruction is uh. essentially it you can think of it as.

inverse. graphics i mean that's one way. to think about it so graphics is your. you have some internal computer. representation. and uh you have a computer. representation of some objects arranged. in a scene. and what you do is you produce a picture. you produce the pixels corresponding to. a rendering of that scene. so uh so let's. do the inverse of this we are given an. image and we try to.

we we we say oh this image. arises from some objects in a scene. looked at with a camera from this. viewpoint and we might have more. information about the objects like their. shape maybe their textures maybe. you know color et cetera et cetera so. that's the reconstruction problem in a. way. that you are in your head creating a. model of the external world.

okay reorganization is to do with. essentially finding these entities so. uh so it's uh organization or. the word organization implies structure. so uh that in in uh perception. in psychology we use the term perceptual. organization. that uh the the world is not just. an image is not just seen as is not. internally represented as just a.

collection of pixels but we. make these entities we create these. entities. objects whatever you want to call in the. relationship between the entities as. well or is it purely about the entities. it could be about the relationships but. mainly we focus on the fact that there. are entities. sometimes i'm trying to pinpoint what. the organization means. so organization is that instead of like. a. uniform grid we have the structure of. objects.

so segmentation is a small part of that. so segmentation gets us going towards. that. yeah and you kind of have this triangle. where they all interact together. yes so how do you see that interaction. in uh sort of uh. reorganization is yes defining the. entities in the world. the recognition is labeling those. entities. and then reconstruction is what filling.

in the gaps. well to for example see. impute some 3d objects corresponding to. each of these. entities that would be part of adding. more information that's not. there in the raw data correct. i mean i started pushing this kind of a. view in the around 2010 or something. like that. because at that time in computer vision.

the distinction that. people were were just. working on many different problems but. they treated each of them as a separate. isolated problem with each with its own. data set and then you try to solve that. and get good numbers on it. so i wasn't i didn't like that approach. because i wanted to see. the connection between these and. if people divided up vision into. into various modules the way they would.

do it is as low level mid-level and. high-level vision. corresponding roughly to the. psychologist's notion of sensation. perception and cognition. and i didn't that didn't map to tasks. that people cared about. okay so therefore i tried to promote. this particular framework. as a way of considering the problems. that people in computer vision were. actually working on. and trying to be more explicit about the. fact that they actually. are connected to each other and i was at.

that time. just doing this on the basis of. information flow. now it turns out in the last five years. or so. in the post the deep learning revolution. that this this architecture has turned. out to be. very conducive to that. because basically in these neural. networks we are trying to. build multiple representations.

there can be multiple output heads. sharing common representations. so in a certain sense today given the. reality of what solutions people have to. these. i i i i do not need to preach this. anymore. it is it is just there it's part of the. solution space. so speaking of neural networks how much. of. this uh problem of computer vision.

of the organization recognition. can be um reconstruction. how much of it can be learned end to end. do you think. instead of uh set it and forget it just. plug and play. have a giant data set multiple perhaps. multi-modal. and then just learn the entirety of it. well so i i think that currently what. that end-to-end learning means nowadays.

is end-to-end supervised learning. and and that i would argue is too narrow. a view of the problem. i would i like this child development. view. this lifelong learning view one where. there are certain capabilities that are. built up and then there are certain. capabilities which are built up. on top of that so uh. that's that's what i i believe in. so i think uh.

end-to-end learning in the supervised. setting. for a very precise task to me is. a kind of is uh. it's sort of a limited view of the of. the learning process. got it so if we think about beyond. purely supervised look at back to. children you mentioned six lessons. that we can learn from children uh of. be multimodal be incremental be physical.

explore be social use language can you. speak to these perhaps picking one. that you find most fundamental toward. yeah time today. yeah so i mean i should say to give due. credit this is from a paper by. smith and gasser and it reflects. essentially i would say common wisdom. among. child development people it's just that.

these are this is not common wisdom. among people. in computer vision and ai and machine. learning so. i view my role as uh trying to. bridge the worlds bridge the two worlds. so uh so let's take an example of a. multi-modal i like that. so multi-modal canonical example is uh. a child interacting with uh with an. object. so then the child so the child holds a.

ball and plays with it. so at that point it's getting a touch. signal. so the touch signal is. is getting as the notion of 3d shape but. it is sparse. and then the child is also seeing a. visual signal right. and and these two so imagine these are. two in totally different spaces. right so one is the space of receptors. on the skin. of the fingers and the thumb and the. palm. right and then these map on to these.

neuronal fibers are. getting activated somewhere right these. lead to some activation in somatosensory. cortex. i mean a similar thing will happen if we. have a robot. hand okay and then we have the pixels. corresponding to the. visual view but we know that they. correspond to the same. object right so that's. a very very strong cross calibration. signal. and it is self-supervisory which is.

beautiful right. there's nobody assigning a label the. mother doesn't have to. come and assign a label the child. doesn't even have to. know that this object is called a ball. okay but the obj the child is learning. something about the three-dimensional. world. from this signal uh. i think tactile and visual there is some. work on. there is a lot of work currently on. audio and visual. okay an audio visual so there is some. event that happens in the world.

and that event has a visual signature. and it has a. auditory signature so there is this. glass bowl on the table and it falls and. breaks and i hear the. smashing sound and i see the pieces of. glass. okay i've built that connection between. the two. right we have people uh i mean this has. become a hot topic in computer vision in. the last couple of years. there is there are problems like uh. separating out multiple speakers right.

which was a classic problem in in. audition they call this the problem of. source separation or the. cocktail party effect and so on but just. try to do it visually. when you also have it becomes so much. easier and so much more useful. so the the multimodal i mean there's so. much more. signal with multimodal and you can use. that. for some kind of weak supervision as. well yes.

because they are occurring at the same. time in time yeah so you have time. which links the two right so at a. certain moment t1. you've got a certain signal in the. auditory domain and a certain signal in. the visual domain. but they must be causally related yeah. it's an exciting area not well studied. yet. not yeah i mean we have a little bit of. work at this but uh but. but so much more needs to be done yeah. so so so. so this this is this is a good example. be physical. that's to do with uh like the one thing.

we talked about. earlier that that there's a embodied. world. to mention language use language so. no chomsky believes that language may be. at the core of cognition at the core of. everything in the human mind. what is the connection between language. and vision to you. like what's more fundamental are they. neighbors. is one the parent and the child the. chicken and the egg. oh it's very clear it is vision which is.

the appearance the fundament the. permission is the fundamental. ability okay well so. uh it comes before you think vision is. more fundamental than language. correct and and and it and yeah. you can think of it either in phylogeny. or in ontogeny. so phylogeny means if you look at. evolutionary time. right so you we have vision that. developed 500 million years ago. okay then something like when we get to. maybe like.

five million years ago you have the. first bipedal primate so when we started. to. walk then the hands became free and so. then. manipulation the ability to manipulate. objects and build tools and. so on and so forth so you said 500 000. years ago no no sorry. the the first multicellular animals. which you can say. had some intelligence arose 500 million. years ago. okay and now let's fast forward to say.

the last. seven million years which is the. development of the hominid line right. where from the other primates we have. the branch which leads on to modern. humans. now there are many of these hominids. but the the ones which. you know people talk about lucy because. that's like a skeleton from three. million years ago and we know that lucy. walked okay so at this stage you have.

that the hand is free for manipulating. objects. and then the ability to manipulate. objects build. tools and the brain size. grew in this era so okay so now you have. manipulation. now we don't know exactly when language. arrows. but after that but after that because no. apes have i mean so i mean chomsky is. correct in that that it is a uniquely. human capability. and we primates.

other primaries don't have that but so. it developed somewhere in this era. but it developed i would. i mean uh argue that it probably. developed after we had this stage of. uh uh humans or i mean the. human species already able to manipulate. and a hands-free much bigger brain size. and for that there's a lot of vision. has already had had to have developed.

yeah so. the sensation and the perception may be. some of the cognition. yeah so we we so those. so so that so the world so there. so so these ancestors of us. you know three four million years ago. they had. uh they had spatial intelligence so they. knew that the world consists of objects. they knew that the objects were in. certain relationships to each other. they had observed causal.

interactions among objects they could. move in space so they had space and time. and all. of that so language. builds on that substrate so language has. a lot of. i mean i mean the all human languages. have constructs which depend on. a notion of space and time where did. that notion of space and time come from. it had to come from perception and. action in the world we live in.

yeah what you refer to as the spatial. intelligence yeah yeah. to linger a little bit we mentioned. touring and his uh mention of. we should learn from children. nevertheless language is. the fundamental piece of the test of. intelligence that touring proposed. what do you think is a good test of. intelligence are you. what would impress the heck out of you. is it fundamentally. natural language or is there something.

in vision. i i think uh i i wouldn't i. i don't think we should have created a. single test of intelligence. so just like i don't believe in iq as a. single number. i think generally there can be many. capabilities. which are correlated perhaps. so i think that there will be. uh there will be accomplishments which. are visual accomplishments. accomplishments which are.

uh accomplishments in manipulation or. robotics and then accomplishments in. language. i do believe that language will be the. hardest not to crack. really yeah so what's what's harder to. pass. the spirit of the touring test or like. whatever formulation will make it. natural language convincingly in natural. language. like somebody you would want to have a. beer with hang out and have a chat with. or the general natural scene. understanding. you think language is the type i think.

i'm not a fan of the. i think i think turing test that turing. as he proposed the test in 1950. was trying to solve a certain problem. yeah imitation. yeah and and i think it made a lot of. sense then. where we are today 70 years later. i think i think we. we should not worry about that i mean i. think the turing test is no. longer the right way to uh to.

to channel research in in ai because. that it takes us down this path of this. chat bot which can fool us for five. minutes or whatever. okay i think i would rather have a list. of 10 different tasks i mean i think. their tasks which their tasks in the. manipulation domain tasks and navigation. tasks and visual scene understanding. tasks in under reading a story and. answering questions based on that i mean. so my favorite.

language understanding task would be. you know reading a novel and being able. to answer arbitrary questions from it. okay right i i think that to me. uh and this is not an exhausted list by. any means. so i would uh i think that that's what. we. where we need to be going to and each of. these. on each of these axes there's a fair. amount of work to be done. so on the visual understanding side in. this intelligence olympics that we've.

set up yeah what's a good. test for one of many. of visual scene understanding. uh do you think such benchmarks exist. sorry to interrupt no there. there aren't any i i think i think. essentially. to me a really uh good. aid to the blind so suppose there was a. blind person. and i needed to assist the blind person. so ultimately like we said vision that. aids in the action.

in the survival in this world yeah. maybe in a simulated world. maybe easier to to measure performance. in a simulated world. what we are ultimately after is. performance in the real world. so david hilbert in 1900 proposed 23. open problems in mathematics some of. which are still unsolved. most important famous of which is. probably the riemann hypothesis. you've thought about and presented about.

the hilbert problems of computer vision. so let me ask what to you today. i don't know when the last year you. presented that 2015 but versions of it. yeah you're kind of the the face and the. spokesperson for computer vision. yeah it's your job to just to state what. the problem. the open problems are for the field so. what today. are the hilbert problems of computer. vision do you think. let me pick pick one to which i regard. as.

uh clearly clearly unsolved. which is what i would call long-form. video understanding. so so we have a video clip and we want. to. understand the behavior. in there in terms of. agents their goals. intentionality and uh. make predictions about what might happen. you know so so that that kind of.

understanding which goes away from. atomic visual action so. so in the short range the question is. are you sitting are you standing are you. catching a ball. right that we can do now or we even if. we can't do it fully accurately. if we can do it at 50 percent maybe next. year we'll do it at 65 and so forth. but i think the long range video. understanding. i don't think we we we can do today well.

today and that means so long and it. blends into cognition that's the reason. why it's challenging. and so you have to track you have to. understand the entities. you have to understand the sds you have. to track them. and you have to have some kind of model. of their behavior. correct and their and if their behavior. might be. these are these are agents so they are. not just like passive. objects but the agent so therefore we. they might they would exhibit gold. directed behavior.

okay so this is this is one area then i. will talk about. say understanding the world in 3d now. this may seem. paradoxical because in a way we have. been able to do 3d understanding even. like. 30 years ago right but i don't think we. currently have the richness of. 3d understanding in our computer vision. system that we would like. because ah so let me elaborate on that a. bit. so currently we have two kinds of.

techniques which are. not fully unified so there are the kinds. of techniques from. multi-view geometry that you have. multiple pictures of a scene and you do. a. reconstruction using stereoscopic vision. or structure from motion. but these techniques do not. they totally fail if you just have a. single view because they are relying. on this this multiple geometry. okay then we have some techniques that. we have developed in the computer vision. community which try to.

guess 3d from single views and these. techniques are based. on on supervised learning. and they are based on having a training. time. 3d models of objects available. and this is completely unnatural. supervision. right that's not cad models are not. injected into your brain. okay so what would i like what i would. like would be a kind of. uh learning as you.

move around the world uh notion of 3d. so so we we have our. succession of visual experiences. and from those we. so in as part of that i might see a. chair from different viewpoints. or a table from viewpoint different. viewpoints and so on. now as part that enables me to build. some internal representation and then.

next time i just see. a single photograph and it may not even. be of that chair it's of some other. chair. and i have a guess of what its 3d shape. is like. so you're almost learning the cad model. kind of. yeah implicitly i mean implicitly i mean. the cad model need not be in the same. form as. used by computer graphics hidden in the. representation. it's hidden in the representation the. ability to predict new views. and what i would see if i.

went to such and such position by the. way and. on a small tangent on that are you. uncomforta are you. okay or comfortable with. neural networks that do achieve visual. understanding that do for example. achieve this kind of 3d understanding. and you don't know how they you don't. know. the rep you're not able to interest but. you're not able to. visualize or understand or interact with. the representation.

so the fact that they're not or may not. be explainable. yeah i think that's fine i to me that is. uh. so so let me put some caveats on that. so it depends on the setting so first of. all i think. uh uh the. uh humans are not explainable. so yeah that's a really good point yeah. so we we. one human to another human is not fully.

explainable. i think there are settings where. explainability matters. and these might these are these might be. for example questions on medical. diagnosis. so i'm in a setting where. maybe the doctor maybe a computer. program has made a certain diagnosis. and then depending on the diagnosis. perhaps i should have treatment day or. treatment b. right so now is the computer programs.

diagnosis based on data. which was data collected of. for american males who are in their 30s. and 40s. and maybe not so relevant to me. maybe it is relevant you know et cetera. et cetera and we i mean in. medical diagnosis we have major issues. to do with the reference class. so we may have acquired statistics from. one group of people and applying it to. a different group of people who may not.

share all the same characteristics. the data might have there might be error. bars in the prediction. so that prediction should really be. taken with. a huge grain of salt and but this has an. impact on what treatments. should be picked right so. so there are settings where i want to. know more than just. this is the answer but what i. acknowledge is that. so so so so i in that sense.

explainability and interpretability may. matter. it's about giving error bounds and a. better sense of the quality of the. decision. where what i where i'm willing to. sacrifice interpretability is that. i believe that there can be systems. which can be highly performant but which. are internally. black boxes and and that seems to be. words headed some of the best performing. systems are essentially black boxes yeah.

uh. fundamentally by their construction you. and i are. black boxes to each other yeah so the. nice thing about the black boxes we are. is so we ourselves are black boxes. but we're also those of us who are. charming. are able to convince others like explain. the black. what's going on inside the black box. with narratives with stories. so in some sense uh neural networks. don't have to actually. explain what's going on inside they just.

have to come up with stories real or. fake. that convince you that they know what's. going on. and i'm sure we can do that we can. create those nearer. those stories neural networks can create. those stories yeah. and the transformer will be involved do. you think we will ever. build a system of human level or. superhuman level intelligence. we've kind of defined what it takes to. try to approach that but do you think.

we'll. do you think that's within our reach the. thing that we thought we could do. what touring thought actually we could. do by a year 2000. right what do you think we'll ever be. able to do so. i think there are two answers here one. question one answer is. in principle can we do this at some time. and my answer is yes the second. answer is a pragmatic one do you think. we will be able to do it in the next 20. years. or whatever and to that man says no.

so and of course that's a wild guess i i. i i think that. you know donald trump's felt is not a. favorite person of mine but. one of his lines is very good which is. about. known knowns known unknowns and unknown. unknowns. so in the business we are in. there are known unknowns and we have. unknown unknowns. so i think with respect to. a lot of what the case in.

vision and robotics i feel like. we have known unknowns so i have a sense. of where we need to go. and what the problems that need to be. solved are. i feel with respect to natural language. understanding and high level cognition. it's not just known unknowns but also. unknown unknowns. so it is very difficult to put any kind. of uh. time frame to that uh do you think some.

of the. unknown unknowns might be positive in. that they'll surprise us and make the. job much easier. so fundamental breakthroughs i think. that is possible because certainly i. have. been very positively surprised by how. effective these deep learning systems. have been because. i certainly would not have believed that. in. 2010 i think.

what we knew from the mathematical. theory. was that convex optimization works when. there's a single global optima then. these gradient descent techniques would. work now these are. non-linear systems with non-convex. systems. huge number of variables so. over-parametrized over-parameterized. and the people who used to play with. them a lot. the ones who are totally immersed in the. lore and the. black magic they knew that they worked.

uh well even though they were really. i thought like everybody no the claim. that. i hear from my friends like yan lacoon. and so forth. now yeah that they feel that they were. comfortable with them. well he says but the community as a. whole. was certainly not and i think uh. we were to me that was the surprise that. they actually worked robustly. for a wide range of problems from a wide.

range of initializations and so on. and uh so that was that that was. certainly. more rapid progress than uh we expected. but then there are certainly lots of. times in fact. most of the history and fear is when we. have made less pro. progress at a slower rate than we. expected. so uh we just keep going. i think uh what i regard as.

uh really unwarranted are these. these fears of uh you know agi in 10. years and 20 years and. that kind of stuff because that's based. on completely unrealistic models of how. rapidly we will make progress in this. field so i agree with you but i've also. gotten a chance to interact with very. smart people who really worry about the. existential threats of ai. and i as an open-minded person and sort.

of taking. and taking it in do you think. if ai systems in some way the unknown. unknowns. not super intelligent ai but in ways we. don't quite understand. uh the nature of superintelligence will. have a detrimental effect on society. do you think this is something we should. be worried about. or we need to first allow the unknown. our nose to become. known unknowns i think we need to be.

worried about ai today. i think that it is not just a worry we. need to have when we get that. agi i think that ai is being used in. many systems today. and there might be settings for example. when it causes. biases or decisions which could. be harmful i mean decisions which could. be unfair to some people. or it could be a self-driving cars which. kills a pedestrian. so ai systems are being deployed today.

right and they're being deployed in many. different settings maybe in medical. diagnosis maybe in a self-driving car. maybe. in selecting applicants for an interview. so. i would argue that when these systems. make mistakes there are consequences. and we are in a certain sense. responsible for those consequences. so i would argue that this is a. continuous effort. it is we and and this is something that.

in a way is not so surprising it's about. all. engineering and scientific progress. which uh. great power comes great responsibility. so as these systems are deployed we have. to worry about them and. it's a continuous problem i don't think. of it as something. which will suddenly happen on some day. in 2079. for which i need to design some clever. trick. i'm saying that these problems exist. today yeah. and we need to be continuously on the.

lookout for. worrying about safety biases risks. right i mean the self-driving car kills. are pedestrian. and they have right i mean the this uber. incident in arizona yeah right it has. happened. right this is not about agi it in fact. it's about a very dumb intelligence. which is also killing people the worry. people have with agi. is the scale and i but i think you're. 100. right is like the thing that worries me.

about ai. today and it's happening in a huge. skills recommend. recommender systems recommendation. systems so if you look at. twitter or facebook or youtube their. controlling the ideas that we have. access to. the news and so on and that's a. fundamentally machine learning algorithm. behind each of these recommendations. and they i mean my life would not be the. same without. these sources of information i'm a.

totally new human being and. the ideas that i know are very much. because of the internet. because of the algorithm that i. recommend those ideas and so. as they get smarter and smarter i mean. that is the agi. yeah is that's the the algorithm that's. recommending. the next youtube video you should watch. has control of millions of billions of. people. that that algorithm is already super. intelligent and. has complete control of the population.

not a complete but. very strong control for now we can turn. off youtube we can just. go have a normal life outside of that. but the more and more that. gets into our life it's that algorithm. we start. depending on it in the different. companies that are working on the. algorithm so i think it's. you're right it's already it's already. there. and youtube in particular is using. computer vision. doing their hardest to try to understand. the content of videos so they could.

be able to connect videos with the. people who would benefit from those. videos the most and so that development. could go in a bunch of different. directions some of which might be. harmful. so yeah you're right the the the threats. of ai are here already we should be. thinking about them. on a philosophical notion. if you could personal perhaps. if you could relive a moment in your. life outside of family.

because it made you truly happy or was a. profound moment that impacted the. direction of your life. what would you go to. i don't think of single moments but i. look over the long haul. i feel that i've been very lucky because. i feel that i think that in. scientific research a lot of it is about.

being at the right place at the right. time. and you can you can work on problems at. a time when. they're just too premature you know you. butt your head. against them and and nothing happens. because it's. the prerequisites for success are not. there and then there are times when you. are in a field which is all. pretty mature and you can only. solve curricules upon colloquius i've.

been lucky to have been in this field. which. for 34 years 35 well actually 34 years. as a professor at berkeley so. longer than that uh which. when i started in it was just. like some little crazy absolutely. useless field which couldn't really do. anything. to a time when it's really really. solving a lot of practical problems has. a lot.

has offered a lot of tools for. scientific research. right because computer vision is. impactful for. images in biology or astronomy and and. so on and so forth. and we have so we have made great. scientific progress which has had. real practical impact in the world and i. feel lucky that. i i got in at a time when the field was. very young and at a time when it is. it's now mature but not fully mature.

it's mature but not. done i mean it's really in still in a in. a productive phase yes. yeah yeah i think people 500 years from. now would laugh are you calling this. field mature. yeah that is very possible yeah so but. you're also. lest i forget to mention you've also. mentored. some of the biggest names of computer. vision computer science and ai. today uh there's so many questions i.

could ask but really is. what what is it how did you do it what. does it take to be. a good mentor what does it take to be a. good guide. yeah i i think what i feel i've been. lucky to have. had very very smart and hardworking and. creative students i think. some part of the credit just belongs to. being at berkeley. i think those of us who are at top. universities. are blessed because we have.

very very smart and capable students. coming on. knocking on our door so so i have to be. humble enough to acknowledge that. but what have i added i think i have. added something. what i have added is uh i think. what i've always tried to teach them is. a sense of picking the right problems. so i think that in science in the short. run.

success is always based on technical. competence. your you know you're quick with math or. you are. whatever i mean there's certain. technical capabilities which make for. short-range. progress long-range progress is really. determined. by asking the right questions and. focusing on the right problems. and i feel that. what i've been able to bring to the. table in terms of. advising these students is.

some sense of taste of what are good. problems. what are problems that are worth. attacking now as opposed to waiting. 10 years what's a good problem if you. could summarize. if is that possible to even summarize. like what what's your sense of a good. problem. i i think uh i think uh i have a sense. of what is a good problem which is. uh there is a british scientist uh. in fact he won a nobel prize peter. medover who has a.

a book on on this and uh basically he. calls. it the research is the art of the. soluble. so we need to sort of find problems. which are. which are not yet solved but which are. approachable. and he sort of refers to this. sense that there is this problem which. isn't quite solved yet but it has a soft. underbelly. there is some place where you can you.

know. spear the beast yes and having that. intuition that this problem is ripe is. is a good thing because otherwise you. can just beat your head and not make. progress. so i think that is that is important so. if. if i have that and if i can convey that. to students. it's not just that they do great. research while they're working with me. but that they continue to do great. research so in a sense i'm proud of my. students. and their achievements and their great.

research even. 20 years after they've seized being my. student. so it's in part developing helping them. develop that sense that a problem. is not yet solved but it's solvable. correct. the other thing which i have which i i. think i bring to the table. uh is i is a certain. intellectual breadth i i've. spent a fair amount of time studying. psychology. neuroscience relevant areas of applied. math and so forth.

so i can probably help them see some. connections. to disparate things which. they might not have otherwise so. so the smart students coming into. berkeley can be. very uh deep in the sense they can think. very deeply meaning very. hard down one particular path but. where i could help them is the the. shallow breadth. but uh whereas they would have the.

the narrow depth and uh but. that's that's of some value well it was. beautifully refreshing just to hear you. naturally jump to psychology back to. computer science and this conversation. back and forth. i mean that that's uh that's actually a. rare quality and i think it's. certainly for students empowering to. think about problems in a new way. so for that and for many other reasons i. really enjoyed this conversation thank. you so much it was a huge honor thanks.

for talking today. it's been my pleasure thanks for. listening to this conversation. with jitendra malik and thank you to our. sponsors. betterhelp and expressvpn. please consider supporting this podcast. by going to betterhelp.com. lex and signing up at expressvpn.com. lexpod click the links buy the stuff. it's how they know i sent you and it. really is the best way to support this. podcast.

and the journey i'm on if you enjoy this. thing. subscribe on youtube review 5 stars on. apple podcast. support it on patreon or connect with me. on twitter. at lex friedman don't ask me how to. spell that i don't remember. myself and now let me leave you with. some words from prince mishkin. and the idiot by dostoyevsky beauty. will save the world thank you for. listening. and hope to see you next time.

you.