Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398

- The following is a conversation. with Mark Zuckerberg inside the Metaverse. Mark and I are hundreds of miles apart. from each other in physical space, but it feels like we're in the same room. because we appear to each other. as photorealistic codec avatars in 3D. with spatial audio. This technology is incredible, and I think it's the future of how human beings. connect to each other. in a deeply meaningful way on the internet. These avatars can capture many.

of the nuances of facial expressions that we use, we humans use, to communicate emotion to each other. Now I just need to work on upgrading my emotion, expressing capabilities of the underlying human. This is "The Lex Fridman Podcast.". And now, dear friends, here's Mark Zuckerberg.

(both laughing). This is so great. Lighting change. Wow. - [Mark] Oh yeah, you can put the light anywhere. - And it doesn't feel awkward to be really close to you. - No, it does. I actually moved you, I moved you back a few feet before you got into headset. You were like right here. - I don't know if people can see this,

but this is incredible. The realism here is just incredible. Where am I? Where are you, Mark? Where are we? - [Mark] You're in Austin, right? - No, I mean this place. (both laughing). We're shrouded by darkness with ultra-realistic face, and it just feels like we're in the same room. This is really the most incredible thing I've ever seen. And sorry to be in your personal space. We have done jiujitsu before. - Yeah, no, I was commenting to the team before.

that even that I feel like we've choked each other. from further distances. than it feels like we are right now. (Lex laughing). - I mean, this is just really incredible. I don't know how to describe with words. It really feels like we're in the same room. - [Mark] Yeah. - It feels like the future. This is truly, truly incredible. I just wanted to take it in. I'm still getting used to it. It's like, it's you, it's really you, but you're not here with me, right?

You're there wearing a headset. and I'm wearing a headset. It's really, really incredible. So, can you describe what it takes currently. for us to appear so photorealistic to each other? - Yeah, so I mean, for background, we both did these scans for this research project. that we have at meta called Kodak Avatars. And the idea is that instead of our avatars being cartoony,

and instead of actually transmitting a video, what it does is we've sort of scanned ourselves. and a lot of different expressions. And we've built a computer model. of sort of each of our faces and bodies. and the different expressions that we make. and collapse that into a Kodak. that then when you have the headset on your head, it sees your face, it sees your expression, and it can basically send an encoded version.

of what you're supposed to look like over the wire. So, in addition to being photorealistic, it's also actually much more bandwidth efficient. than transmitting a full video. or especially a 3D immersive video. of a whole scene like this. - And it captures everything, like the flaws. Like to me, the subtleties of the human face, like even the flaws, that's all amazing. It makes it so much more immersive.

It makes you realize that like perfection. isn't the thing that leads to immersion. It's like the little subtle flaws like freckles. and like variations in color and just. - Yeah, wrinkles. - Asymmetry. - Yeah, asymmetry. - Yeah. And just the different, like the corners of the eyes, like what your eyes do when you smile, all that kind of stuff. - Yeah, eyes are a huge part of it. - It's just incredible. - Yeah, I mean, there's all the studies. that most of communication, even when people are speaking. is not actually the words that they're saying, right?

It's kind of the expression and all that. So, and we try to capture that with the kind of classical. expressive avatar system that we have. That's the kind of more cartoon designed one. You can kind of put those kind of expressions. on those faces as well. But there's obviously a certain realism. that comes with delivering kinda. this photo realistic experience. that I don't know. I just think it's really magical. I mean, this gets to kinda the core. of what the vision around virtual and augmented reality is, of like delivering a sense of presence,

as if you're there together. no matter where you actually are in the world. And I mean, this experience, I think is a good embodiment of that, where it's like, I mean, we're in two completely different states. halfway across the country, and it just like, you know, looks like you're just sitting right in front of me. It's pretty wild. - Yeah, I mean, I can't, I'm almost getting emotional. It's like, it feels like a totally, it's fundamentally new experience. Like for me to have this kind of conversation. with loved ones, it would just change everything.

Maybe just to elaborate, so I went to Pittsburgh. and went through the whole scanning procedure, which has so much incredible technology, so software and hardware going on, but it is a lengthy process. So what's your vision for the future of this. in terms of making this more accessible to people? - You know, it starts off with a small number. of people doing these very detailed scans, right? Which is, that's the version that you did and that I did.

And you know, before there were a lot of people. who we've done this kind of a scan for, we probably need to kind of over collect expressions. when we're doing the scanning. because we haven't figured out. how much we can reduce that down. to a really streamlined process. and extrapolate from the scans. that have already been done. But, you know, the goal, and we have a project that's working on this already, is just to do a very quick scan with your cell phone. where you just take your phone,

kind of wave it in front of your face. for a couple of minutes, you know, say a few sentences, make a bunch of expressions, but overall, have the whole process. just be two to three minutes. and then produce something that's of the quality. of what we have right now. So I think that that's one. of the big challenges that remains. And right now we have the ability. to do the scans if you, you know, have hours to sit for one. And with today's technology, I mean, you're using. a meta headset that exists. It's a product that's kind of for sale, now.

You can drive these with that, but the production of these scans. in a very efficient way is one of the last pieces. that we still need to really nail. And then obviously there's all the experiences around it. I mean, right now we're kind of sitting. in a dark room, which, you know, is, you know, familiar for your podcast. But I think part of the vision for this over time. is, you know, not just having this be like a video call. I mean, that's fine.

It's cool or it feels like it's immersive, but you know, you can do a video call on your phone. The thing that you can do in the metaverse. that is different. from what you can do on a phone is like doing stuff. where you're physically there together. and participating in things together. And we could play games like this. We could have meetings like this in the future. Once you get mixed reality and augmented reality, we could have Kodak avatars like this. and go into a meeting. and have some people physically there. and have some people show up. in this photorealistic form,

superimposed on the physical en environment. I think that stuff like that is gonna be super powerful. So we gotta still build out all those kinda applications. and the use cases around it. But I don't know, I think it's gonna be a pretty wild. next few years around this. - I mean, I just, I'm actually almost at a loss of words. This is just so incredible. This is truly incredible. I hope that people like watching this. can get a glimpse of like, how incredible it is. It really feels like we're in the same room. Like there is that,

I guess there's an uncanny valley. that seems to have been crossed here. Like it looks like you. - Yeah. - Like really you. I mean, I think there's still a bunch of tuning. that I think will want to do. where different people emote. to different extents, right? So I think one of the big questions is, you know, like when you smile, how wide is your smile? And how wide do you want your smile to be? And I think getting that to be tuned on a per person basis. is gonna be one of the things.

that we're gonna need to figure out. You know, it's like to what extent. do you wanna give people control over that? You know, some people might try. to, you know, might prefer a version. of themselves that's more emotive. in their avatar than their actual faces. You know, so for example, you know, I always get a lot of critique. and shit for having like a relatively stiff expression. But, you know, I mean, I might feel pretty happy, but just make a pretty small smile.

So, I mean, maybe, you know, for me, I would, it's actually, you know, it's like I'd wanna have my avatar really. be able to better express like, how I'm feeling than how I can do physically. So I think that there's a question. about how you want to tune that. But, overall, yeah, I mean you, we wanna start from the baseline. of capturing how people. actually emote and express themselves. And I mean, I think the initial version. of this has been pretty impressive. And like you said, I do think we're kind of beyond. the uncanny valley here.

And it does feel like you. It doesn't feel weird or anything like that. - I mean, that's gonna be the meme. that the two most monotone people. are in a metaverse together. But I think that actually makes it more difficult. Like, the amazing thing here. is that the subtleties. of the expression of the eyes, you know, people say I'm monotone and emotionless, but I'm not. It's just, maybe my expression of emotion. is more subtle, usually like with the eyes. And that's one of the things I've noticed.

is just how expressive the subtle movement. of the corners of the eyes are. in terms of displaying happiness or boredom. or all that kind of stuff. - I'm curious to see, just because I've never done one of these before, I've never done a podcast. as one of these Kodak avatars, and I'm curious to see what people think of it. Because you know, one of the issues that we've had in some of the VR. and mixed reality work is it tends. to feel a lot more profound. when you're in it than the 2D videos. capturing the experience.

So I think that this one, because it's photorealistic may look kind of as amazing. in 2D for people watching it. as it as it feels, I think, to be in it. But we've certainly had this issue. where a lot of the other things just, it's like you feel the sense of immersion. when you're in it, that doesn't quite translate to a 2D screen. But I dunno, I'm curious to see what people think. - Yeah, I'm curious to see if people could see that. Like, my heart is actually beating fast now. This is super interesting, like,

that such intimacy of conversation. could be achieved remotely. There's been, you know, I don't do remote podcast for this reason. and this is like, breaks all of that. This feels like just an incredible transition. to something else, a different kind of communication. Breaks all barriers, like geographic physical barriers. Do you have a sense of timeline. in terms of how many difficult things. have to be solved. to make this more accessible to like,

scanning with a smartphone? - Yeah, I mean, I think we'll probably roll this out. progressively over time. So it's not gonna be like we roll it out. and one day, everyone has a Kodak avatar. We want to get more people scanned and into the system. and then we wanna start integrating it. into each one of our apps, right? Making it so that, you know, I think that for a lot. of the work style things, productivity, I think that this is gonna make a ton of sense. and a lot of game environments, I mean, this could be fine, but games tend to have their own style, right? Where you almost want to fit more.

with the aesthetic style of the game. But I think for doing meetings, and one of the things. that we get a lot of feedback on workrooms, where, you know, people are pretty blown away. by the experience. and this feeling that you can like be remote, but feel like you're physically there. around a table with people. But then, you know, we get some feedback. that people have a hard time. with the fact that the avatars are so expressive. and don't feel, you know, as realistic in that environment. So I think something like this. could make a very big difference.

for those remote meetings. And especially with Quest 3 coming out, which is gonna be the first mainstream. mixed reality product, right? Where you're really taking digital, you know, expressions of either a person or objects. and overlaying them on the physical world, I think the ability to do kind of remote meetings. and things like that where you're like. just remote hang sessions with friends. I mean, I think that that's gonna be very exciting. So yeah, rolling it out over the next, over the next few years, it's not ready to be like a kind of mainstream product yet,

but we just want to, we'll keep tuning it and keep getting more scans in there. and keep, you know, and kind of rolling it out. into more of the features. But yeah, I mean, definitely in the next few years, you'll be seeing a bunch more experiences like this. - Yeah, I would love to see some celebrities. scanned and some non-celebrities. and just more people to experience this. I would love to see that. This is something. I mean my mind is blown. I'm literally at a loss of words. 'cause it's very difficult to just convey.

how incredible this is. How like how I feel the emotion, how I feel the presence, how I feel like the subtleties of the emotion. in terms of like work meetings. or in terms of podcasts. This is like, this is awesome. and I don't even need your arms or legs. - Well, we gotta get that. I mean that's its own challenge. And part of the question is also, so you have the scan, then it takes a certain amount of compute. to go drive that, both for the sensors on the headset.

and then rendering it. So one of the things that we're working through. is what is the level of fidelity that is optimal, right? You could do the full body in kind of a Kodak, and that can be quite intensive, but one of the things. that we're thinking about is like, all right, maybe you can kind of stitch. a somewhat lower fidelity version of your body, would still have the main kind of the major movements, but your face is really the thing.

that we have the most resolution on, right? In terms of being able to read and express emotions. I mean, like you said, if you move your, you know, eyebrows like a millimeter, I mean that really changes the expression. and what you're emoting, whereas, you know, I mean, moving your arm. like an inch probably doesn't matter quite as much. So yeah. So I think that we do wanna get all of that into here. and that'll be some of the work. over the next period as well. - So you mentioned Quest 3. That's coming out.

I've gotten a chance to try that too. That's awesome. So, how'd you pull off the mix? So it's not just virtual reality, it's mixed reality. - Yeah, I mean, I think it's gonna be. the first mainstream mixed reality device. I mean, obviously we shipped Quest Pro last year, but it was $1,500. And well, part of what I'm super proud of is, you know, we try to innovate not just on pushing the state-of-the-art. and delivering new capabilities, but making it so it can be available to everyone. And you know, we have this, and it's coming out,

it's $500 and in some ways I think the mixed reality. is actually better in Quest 3. than what we're using right now in Quest Pro. So, and I'm really proud of the team. for being able to deliver. that kind of an innovation and get it out. But, you know, some of this is just. software you tune over time and get to be better. Part of it is you put together a product. and you figure out what are the bottlenecks. in terms of making it a good experience. So we got the resolution for the mixed reality cameras. and sensors to be multiple times better in Quest 3.

And we just figured that that made a very big difference. when we saw the experience. that we were able to put together for Quest Pro. And part of it is also that, you know, Qualcomm just came out with their next generation chip set. for VR and MR that we worked with them. on a kind of custom version of it. But that was available this year for Quest 3. and it wasn't available in Quest Pro. So, you know, in a way in Quest 3, even though it's not, you know, the Pro product actually has a stronger chip set. in it than the Pro line at a third of the cost.

So, I'm really excited to get this in people's hands. It does all the VR stuff that Quest 2. and the others have done too. It does it better. because the display is better. and the chip is better. So you'll get better graphics. It's 40% thinner, so it's so just more comfortable as well. But the MR is really the big capability shift. And part of what's exciting. about the whole space right now is, you know, this isn't like smartphones. where, you know, companies.

put out a new smartphone every year. and you can almost barely tell the difference. between that and the one the year before it. You know, for this, each time we put out a new headset, it has like a major new capability. And the big one now is mixed reality. The ability to basically take digital representations. of people or objects and superimpose them on the world. And basically, you know, I mean there's a one version. of this is you're gonna kind. of have these augments or holograms.

and experiences that you can kind of bring. into your living room. or a meeting space or an office. Another thing that I just think. is gonna be a much kind of simpler innovation. is that there are a lot of VR experiences. today that don't need to be fully immersive. And, you know, if you're playing a shooter game. or you know, you're doing a fitness experience. and sometimes people get worried. about swinging their arms around, like, "Am I gonna hit a lamp or something?". You know, and "Am I gonna run into something?". So having that in mixed reality actually. is just a lot more comfortable for people, right?

You kind of still get the immersion. and the 3D experience and you can have an experience. that just wouldn't be possible. in the physical world alone, but by being anchored to, and being able to see the physical world around you, it's like, it just feels so much safer and more secure. And I think a lot of people. are really gonna enjoy that too. So yeah, I'm really excited. to see how people use it. But yeah, Quest 3 coming out later this fall. - Yeah. And I got to experience it. with other people sitting around. and there's a lot of furniture. And so you get to see that furniture, you get to see those people.

And you get to see those people. like enjoy the ridiculousness of you, like swinging your arms. I mean, presumably they're friends of yours. Even if they make fun of you, there's a lot of love behind that. And I got to experience that. So that's a really fundamentally different experience. than just pure VR. with zombies coming outta walls and you. - Yeah, it's like someone shooting at you. and you hide behind your real couch. in order to duck the fire. Yeah. - It's incredible how it's all integrated, but also like subtle stuff, like in a room with no windows,

you can add windows to it. and you can look outside. as the zombies run towards you, but like it's still nice view outside, you know? - Yeah. - It's just, it's really. And so that's pulled off by having cameras. on the outside of the headset. that do the pass through, that technology is incredible. to do that on a small headset. - Yeah, it's not just the cameras. You basically you need to be in multiple cameras. to capture the different angles. and sort of the three-dimensional space. And then it's a pretty complex compute problem,

an AI problem to map that to your perspective, right? Because the cameras aren't exactly. where your eyes are. because no two people's eyes, you're not gonna be in exactly the same place. You kind of need to get that to line up. and then do that basically in real time, and then generate something that looks, that kind of feels natural, and then superimpose whatever digital objects. you wanna put there. So, yeah, it's a very interesting technical challenge. and I think we'll continue tuning this.

for the years to come, as well. But, I'm pretty excited to get this out. because I think Quest 3. is gonna be the first device like this. that millions of people are gonna get. that's mixed reality. And it's only when you have millions of people. using something that you start getting. the whole developer community really starting. to experiment and build stuff. 'cause now there are gonna be people. who actually use it. So I think we'll get, you know, we got some of that flywheel going with Quest Pro, but I think it'll really get accelerated once Quest 3. gets out there. And so yeah, I'm pretty excited about this one.

- Plus, there's hand tracking without, you don't need to have a control. So the cameras aren't just in the pass through. of the entire physical reality around you. It's also tracking the details of your hands. in order to use that for like gesture recognition, this kind of stuff. - Yeah, we've been able to get way further. on hand recognition. in a shorter period of time than I expected. So that's been pretty cool. I don't know, did you see the demo experience. that we built around. - Piano?

- Yeah, the piano, learning to play piano? - Yeah, yeah. It's incredible. You're basically playing piano on a table. and that's without any controller, and like how well it matches. physical reality with no latency. And it's tracking your hands with no latency. And it's tracking all the people. around you with no latency. Integrating physical reality and digital reality, obviously that connects exactly to this Kodak avatar, which is in parallel allows us. to have ultra realistic copies of ourselves.

in this mixed reality. So, like, it is all converging. towards like an incredible digital. experience in the metaverse. To me, obviously I love the intimacy of conversations, so even this is awesome. But do you have other ideas of what this unlocks. of like something like Kodak avatar unlocks. in terms of applications, in terms of things we're able to do? - Well, there's what you can do.

with avatars overall. in terms of superimposing digital objects. on the physical world. And then there's kind of psychologically. what is having photorealistic do, you know? So, I think we're moving towards a world. where, you know, we're gonna have something. that looks like normal glasses. where you can just, you see the physical world, but you'll see holograms. And in that world, I think. that there are gonna be, you know, not too far off, you know, maybe, you know,

by the end of this decade. we'll be living in a world. where there are kinda as many holograms. when you walk into a room. as there are physical objects. And it really raises this interesting question. about, you know, a lot of people have this phrase. where they they call the physical world the real world. And you know, I kind of think increasingly, yeah, the physical world is super important, but I actually think the real world. is the combination of the physical world. and the digital worlds coming together. But until this technology,

they were sort of separate, right? It's like you access the digital world. through a screen, right? And, you know, maybe it's a small screen. that you carry around. or it's a bigger screen. when you sit down at your desk. and, you know, strap in for a long session. But they're kind of fundamentally divorced and disconnected. And I think part of what this technology is gonna do. is bring those together into a single coherent experience. of what the modern real world is, which is, it's gotta be physical. because we're physical beings. So the physical world. is always gonna be super important.

But increasingly, I think a lot of the things. that we kind of think of can be digital holograms. I mean, any screen that you have can be a hologram. You know, any media in any book art, you know, can basically be just as effective as a hologram, as a physical object. Any game that you're playing, a board game. or any kind of physical game cards, you know, ping pong, things like that. They're often a lot better as holograms. 'cause you could just kind of snap your fingers.

and instantiate them and have them show up. You know, it's like you have a ping pong table. show up in your living room, but then you can snap your fingers. and have it be gone. So that's super powerful. So I think that it's actually. an amazing thought experiment. of like how many physical things we have today. that could actually be better. as interactive holograms. But then beyond that, I think the most important thing obviously as people, so the ability to, you know, have these mixed hangouts,

whether they're social or meetings. where, you know, you show up to a conference room, you're wearing glasses or a headset in the very near term, but, you know, hopefully by, you know, for the next five years glasses or so, and you know, you're there physically. Some people are there physically, but other people are just there as holograms, and it feels like it's them who are right there. And also by the way, another thing that I think is gonna be fascinating. about being able to blend together. the digital and physical worlds.

in this way is we're also going. to be able to embody AIs as well. So I think you'll also have meetings. in the future. where you're basically, you know, maybe you're sitting there physically. and then you have, you know, a couple of other people who are there as holograms. And then you have like Bob the AI who's an engineer. on your team who's helping with things, and he can now be embodied. as a realistic avatar as well, and just join the meeting in that way. So I think that's gonna be pretty compelling as well.

So then, okay, so what can you do. with photorealistic avatars. compared to kind of the more expressive ones. that we have today? Well, I think a lot of this actually. comes down to acceptance of the technology. And because all of the stuff that we're doing, I mean the motion of your eyebrows, the motion of your eyes, the cheeks and all of that, there's actually no reason. why you couldn't do that. on an expressive avatar too.

I mean, it wouldn't look exactly like you, but I mean, you can make a cartoon version. of yourself and still have it. be almost as expressive. But I do think that there's this bridge. between the current state of most our interactions. in the physical world. and where we're getting in the future. with this kind of hybrid, physical and digital world. where I think it's gonna be a lot easier for people. to kind of take some of these experiences seriously. with the photorealistic avatars to start.

And then I'm actually really curious. to see where it goes longer term. I could see a world where people stick to the photorealistic. and maybe they modify them. to make them a little bit more interesting, but maybe fundamentally we like photorealistic things. But I can also see a world. that once people get used to the photorealistic avatars. and they get used to these experiences. that I actually think. that there could be a world. where people actually prefer. being able to express themselves.

in kind of non, you know, ways that aren't so tied to their physical reality. And so that's one of the things. that I'm really curious about. And I dunno, in a bunch of our internal experiments on this, one of the things. that has I thought was psychologically pretty interesting. is people have no issues. blending photorealistic stuff and not. So, you know, for this specific scene. that we're in now, we happen to sort of be in a dark room. I think part of that aesthetic decision,

I think was based on the way you like to do your podcast. But we've done experiences like this. where you have like a cartoony background, but photorealistic people. who you're talking to, and we seem to like, people just seem to just think. that that is completely normal, right? It doesn't bother you. It doesn't feel like it's weird. Another thing that we've experienced. is basically you have a photorealistic avatar. that you're talking to, and then right next to them. you have an expressive kind of cartoon avatar.

And that actually is pretty normal too, right? It's like, it's not that weird right? To basically being interacting. with different people in different modes like that. So I'm not sure. I think it'll be an interesting question. to what extent these photorealistic avatars. are like a key part of just transitioning. from being comfortable in the physical world. to this kind of new, modern, real world that that kind of includes. both the digital and physical. or if this is like the long-term way that it stays.

I mean, I think that there are gonna be uses. for both the expressive. and the photorealistic over time. I just dunno what the balance is gonna be. - Yeah, it's a really good, interesting philosophical question. But to me, in the short term, the photorealistic is amazing. to where I would prefer, like you said, the workroom, but like on a beach with a beer. to see a buddy of mine. remotely on a chair next to me drinking a beer. I mean, that as realistic as possible. is an incredible experience.

So I don't want any fake hats on him. I don't want any just chilling. with a friend drinking beer, looking at the ocean while not being. in the same place together. I mean that, yeah, that experience. is just, it's a fundamentally, it's just a high quality experience, a friendship. Whatever we seek in friendship, it seems to be present there. in the same kind of realism. I'm seeing right now. This is totally a game changer. So to me, I can see myself sticking with this. for a long time.

- Yeah, and long term, I mean, it's also, it's novel, and it's also a technological feat, right? It's like being able to pull this off is like, it's like a pretty impressive, and I think to some degree. it's just this kind of like awesome experience. - Yeah. But I'm already, sorry to interrupt, I'm already forgetting. that you're not real, like this really. - So I am real. - It's novel. - This is just an avatar version of me. - It's a deep philosophical question, yes. - But I mean, so I put this on this morning.

and I was like, all right, like I, it's like, okay, so my hair is a little shorter. in this than my physical hair is right now. I probably need to go get a haircut. And like, I dunno, I actually, I did happen to shave this morning, but if I hadn't, you know, I could still have this photorealistic avatar. that is more cleanly shaven, right? Even if I'm, you know, a few days in, physically. So I do think that there are gonna start. to be these subtle questions. that seep in where the avatar is realistic.

in the sense of this is kind of what you looked. like at the time of capture, but it's not necessarily temporarily accurate. to exactly what you look like in this moment. And that there're gonna end up being a bunch of questions. that come from that over time. that I think are gonna be fascinating too. - You mean just like the nature of identity. of who we are? Are we the people? You know, how people do. like summer Beachbody or the people.

will be for the scan, they'll try to lose some weight. and look their best. and sexiest with the nice hair. and everything like that. I mean, it does raise the question. of, you know, if a lot of people interacting. with the digital version of ourselves, who are we really? Are we the entity driving the avatar? Are we the avatar? - Well, I mean, I think our physical bodies also fluctuate. and change over time too. So I think there's a similar question of like, which version of that are we?

Right? I mean, and it's interesting identity question. because all right, it's like, I don't know, it's like weight fluctuates or things like that. It's like I think most people don't tend. to think of themselves as the, well, I don't know, it's an interesting psychological question. Maybe some people, maybe a lot of people do think about themselves. as the kind of worst version, but I think a lot of people probably think. about themselves as the best version. And then it's like what you are on a day-to-day basis. doesn't necessarily map to either of those.

So I think that that's, yeah, there will definitely be. a bunch of social scientists and folks. will have to, you know, and psychologists, really, there's gonna be a lot to understand. about how our perception of ourselves. and others has shifted from this. - Well, this might be a bit of a complicated. and a dark question, but one of the first feelings I had experiencing this. is I would love to talk to loved ones.

And the next question I have is. I would love to talk to people. who are no longer here that are loved ones. So like, if you look into the future, is that something you think about? People who pass away, but they could still exist in the metaverse. And you could still have, you know, talk to your father, talk to your grandfather and grandmother and a mother. once they pass away. The power of that experience. is one of the first things my mind jumped to. 'cause it's like, this is so real.

- Yeah, I think that there are a lot. of norms and things that people. have to figure out around that. There's probably some balance where, you know, if someone has lost a loved one and is grieving, there may be ways in which, you know, being able to interact or relive certain memories. could be helpful. But then there's also probably an extent. to which it could become unhealthy. And I mean, I'm not an expert in that, so I think we'd have to study that. and understand it in more detail.

We have, you know, a fair amount of experience. with how to handle death. and identity and people's digital content. through social media already, unfortunately. Right, where there's, you know, unfortunately, you know, people who use our services die every day. and their families, you know, often wanna have access to their profiles. And we have whole protocols that we go through. where, you know, there are certain parts of it. that we try to memorialize. so that way the family.

can get access to it. so that way the account doesn't. just go away immediately. But then there are other things that are, you know, important kind of private things that that person has. Like, we're not gonna give the family access. to someone's messages, you know, for example. So, yeah, I think that there's some best practices. I think from the current digital. world that will carry over. But, yeah, I think that this. will enable some different things. Another version of this. is how this intersects with AIs, right? Because, and one of the things.

that we're really focused on is, you know, we want the world to evolve in a way. where there isn't like a single AI super intelligence, but where, you know, a lot of people are empowered by having AI tools. to do their jobs and, you know, make their lives better. And if you're a creator, right? And if you run a, you know, podcast like you do, then you know, you have a big community of people. who are super interested to talk to you.

I know you'd love to, you know, cultivate that community, and you interact with them online. outside of the podcast as well. But I mean, there's way more demand. both to interact with you. and I'm sure you'd love to interact. with the community more, but you just are limited by the number of hours in the day. So, you know, at some point, I think making it so that you could build. an AI version of yourself. that could interact with people, you know, not after you die, but while you're here to, you know, help people kind of fulfill this desire.

to interact with you. and your desire to build a community. And there's a lot. of interesting questions around that. And, you know, that's obviously, it's not just in the metaverse. I think, you know, we'd wanna make that work, you know, across all the messaging platforms, you know, WhatsApp and Messenger and Instagram direct. But, you know, there's certainly, you know, a version of that where if you could have an avatar version. of yourself in the metaverse. that people can interact with, and you could define that sort of an AI version.

where, you know, people know. that they're interacting with an AI, that it's not, you know, the kind of physical version of you, but maybe that AI, even if they know it's an AI, is the next best thing. because they're probably not gonna, you know, necessarily all get to interact with you directly. I think that that could be. a really compelling experience. There's a lot of things that we need. to get right about it. that, you know, we're not ready. to release the version that a creator. can kind of build a version of themselves yet, but we're starting to experiment with it.

in terms of releasing a number of AIs. that people can interact with in different ways. And I think that that is also just gonna be. a very powerful, you know, set of capabilities that people have over time. - So you've made major strides. in developing these early AI personalities. with the idea where you can talk to them. across the meta apps and have like interesting, unique kind of conversations. Can you describe your vision there.

in these early strides. and what are some technical challenges there? - Yeah, so I mean, a lot of the vision comes from this idea. that, you know, I don't think. we necessarily want there. to be like one big super intelligence. We wanna empower everyone to both, you know, have more fun, accomplish their business goals, you know, it just everything. that they're trying to do. And, you know, we don't tend to have, you know, one person that we work with on everything. And I don't think in the future we're gonna have, you know, one AI that we work with. I think you're gonna want a variety of these.

So there are a bunch of different uses. You know, some will be kind of more assistant oriented. There's a sort of the kind of plain and simple one. that we're building. It's called just meta AI. It's simple, it, you know, you can chat with it in any of your threads. It doesn't have a face, right? It's just kind of more vanilla. and neutral and kind of factual, but it can help you with a bunch of stuff.

Then there are a bunch of cases. that are more kind of business oriented. So let's say you wanna contact a small business. You know, similarly, you know, that business. probably doesn't wanna have to staff someone. to man the phones and you probably don't wanna wait. on the phone to talk to someone. But, you know, having someone. who you can just like talk to. in a natural way who can, you know, help you if you're having an issue. with a product or you know, if you wanna make a reservation or if you wanna, buy something online, having the ability. to do that and have a natural conversation.

rather than navigate some website. or have to call someone and wait on hold, I think is gonna be really good, both for the businesses and for normal people. who want to interact with businesses. So I think stuff like that makes sense. Then there are gonna be a bunch of use cases. that I think are just fun, right? So I think people are gonna, I think that there will be AIs that like can tell jokes. so you can put them into chat thread with friends. I mean, I think a lot of this, because we're like a social company, right? I mean we're, you know, fundamentally around helping people connect.

in different ways. And part of what I'm excited about is, you know, how do you enable these kind of AIs. to facilitate connection. between two people or more, you know, put them in a group chat, you know, make the group chat more interesting. around whatever your interests are, sports, fashion, trivia. - Video games. I love the idea of playing, I think you mentioned Balder's Gate, an incredible game, just having an AI that you play together with.

and, I mean, that seems like a small thing, but it could deeply enrich the like, gaming experience. - Well I do think that AIs will be, will make the NPCs a lot better in games too. So that's a a separate thing that I'm pretty excited about. But, but yeah, I mean one of the AIs. that we've built that just in our internal. testing people have loved the most. is like a adventure text-based, get like a dungeon master. - Yeah, nice.

- And, I think, you know, part of what has been fun, and we talked about this a bit, but we've gotten some like real kind of cultural figures. to play a bunch of these folks. and be the embodiment in the avatar of them. So, Snoop Dogg is the dungeon master, which I think is just hilarious. - Yes, in terms of the next steps of, you know, you mentioned Snoop to create a Snoop AI, so basically AI personality replica, a copy or not a copy maybe inspired by Snoop.

What are some of the technical challenges of that? What does that experience look like for Snoop. to be able to create that AI person? - So starting off creating new personas is easier. because it doesn't need to stick exactly. to what, you know, that physical person would want, how they'd wanna be represented, right? It's like, it's just a new character. that we created. So even though, so Snoop in that case. you know, he's basically an actor, right?

He's playing the, the dungeon master, but it's not Snoop Dogg, right? It's, you know, whoever the dungeon master is. If you wanna actually make it. so that you have an AI embodying a real creator, there's a whole set of things that you need. to do to make sure that that AI. is not gonna say things that the creator. doesn't want, right? And that the AI is gonna, you know, know things and be able to represent things.

in the way that the creator would want, the way that the creator would know. So I think that it's less of a question. around like having the avatar express them. I mean that I think where, you know, it's like, well, we have our kind of V 1 of that. that we'll release soon is after connect, but, you know, that'll get better over time. But a lot of this is really just. about continuing to make. the models for these AIs, it's that they're just more and more,

I don't know, you could say like reliable or predictable. in terms of what they'll communicate. that way, you know, when you wanna create the Lex assistant AI. that your community can talk to, you know, you don't program them. like normal computers. You're training them. They're AI models, not kind of normal computer programs, but you want to get it to be predictable enough. so that way you can set some parameters for it. And even if it isn't perfect all the time,

you want it to generally be able to stay. within those bounds. So that's a lot of what I think we need. to nail for the creators. And that's why that one's actually. a much harder problem, I think, than starting with new characters. that you're creating from scratch. So that one I think will probably. start releasing sometime next year, not this year, but experimenting with existing characters. and the assistant and games.

and a bunch of different personalities. and experimenting with some small businesses. I think that that stuff we'll be ready. to do this year and we're rolling it out, you know, basically right after Connect. - Yeah, I'm deeply entertained. by the possibility of me sitting down. with myself and saying, "Hey man, like you need to stop the dad jokes,". or whatever. - I think the idea of a podcast. between you and AI assistant Lex podcast. (both laughing). - I mean, there is just.

even the experience of aquatic avatar, being able to freeze yourself, like to basically first mimic yourself. So everything you do, you get to see yourself do it. That's a surreal experience. That feels like if I was like an ape. looking in a mirror. for the first time, realizing like, oh, that's you. But then freezing that and being able. to look around like, I'm looking at you. I don't know how to put it into words, but it just feels like a fundamentally new experience. Like I'm seeing maybe color for the first time.

I'm seeing, I'm experiencing. a new way of seeing the world. for the first time. because it's physical reality, but it's digital like, and realizing that that's possible. It's just, it's blowing my mind. It's just really exciting. because like I lived most of my life, you know, before the internet. and experiencing the internet, experiencing voice communication, video communication, you think like, well there's a ceiling to this,

but this is making me feel like there might not be, there might be that blend of physical reality. and digital reality that's actually what the future is. - Yeah, I think so. - It's a weird experience. It's feels like the early days of, of like a totally new way of living. And like, there's a lot of people that kind of complain, "Well, you know, the internet, that's not reality. You need to turn all that off. and go, you know, in nature.".

But this feels like this will make those people happy, I feel like, because it feels real, the flaws in everything. - Yeah. Well, I mean, a big part of how we're trying to design this, these new computing products. is that they should be physical, right? I think part that's a big part of the issue. with computers and TVs and even phones is like, yeah, I mean maybe you can interact with 'em. in different places, but they're fundamentally like you're sitting, you're still, and I mean, people are just not meant to be that way. I mean, I think you, and I have this shared passion.

for sports and martial arts. and doing stuff like that. We're just moving around. It's like, so much of what makes us people. is like, you know, you move around. We're not just like a brain and a tank, right? You know, the human experience is a physical one. And so it's not just about. having the immersive expression. of the digital world. It's about being able. to really natively bring that together. And I do really think. that the real world is this mix. of the physical and the digital, right?

The digital is, there's too much digital at this point. for it to just be siloed to a small screen. But the physical is too important. So you don't wanna just sit down all day long at a desk. So I think that this is, yeah, I do think that this is the future. This is, I think, the kind of philosophical way. that I would want the world. to work in the future is a much more coherently blended, physical and digital world. - There might be some difficult philosophical. and even ethical questions we have.

to figure out as a society. Maybe you can comment on this. So the metaverse seems to enable, sort of unlock a lot of experiences. that we don't have in the physical world. And the question is like, what is and isn't allowed in the metaverse. You know, in video games, we allow all kinds of crazy stuff. And in physical reality, you know, a lot of that is illegal. So where's that line?

Where's that gray area between video game. and physical reality? Do you have a sense of that? - Well, I think, I mean there are content policies. and things like that, right? In terms of what people are allowed to create. But I mean, a lot of the rules around physical, I think we try to have a society that. is as free as possible, meaning that people can do as much. of what they want. unless you're gonna do damage. to other people and infringe on their rights. And the idea of damage is somewhat different.

in a digital environment. I mean, when I get into, in know some world with my friends, the first thing we start doing is shooting each other, which obviously we would not do in the physical world. 'cause you, you didn't need to hurt each other. But in a game that's like, it's almost, you know, it's like just fun, and even in like the lobby of a game, right? It's like, it's not even bearing on the game. It's kind of like a funny, sort of humorous thing to do. So it's like, is that problematic? I don't think so because it's fundamentally,

you're not causing harm in that world. So I think that the part of the question. that I think we need to figure out. is what are the ways. where things could have been harmful. in the physical world. that we'll now be freed from that? And therefore there should be fewer restrictions. in the digital world. And then there might be new ways. in which there could be harm. in the digital world. that there weren't the case before. So there's more anonymity, right? It's, you know, when you show up. to, you know, a restaurant or something, it's like all the norms.

where you pay the bill at the end, it's because, you know, you have one identity and you know, if you stiff them, then like, you know, life is a repeat game. and that's not gonna work out well for you. But, you know, in a digital world. where you can be anonymous and show up. and in different ways, I think the incentive to act like. a good citizen can be a lot less. And that causes a lot of issues. and toxic behavior so that needs to get sorted out. So I think in terms of what is allowed,

I think you wanna just look at what are the damages. But then there's also other things. that are not related to kind of harm, you know, less about what should be allowed. and more about what will be possible. that are more about the laws of physics, right? It's like if you wanted to travel to see me in person, you'd have to get on a plane and that would like, you know, take a few hours to get here. Whereas, you know, we could just jump in a conference room. and, you know, put on these headsets. and we're basically teleported.

into a space where, you know, it feels like we're together. So that's a very novel experience. that it breaks down some things. that previously would've defied. the laws of physics. for what it would take to get together. And I think that that will create. a lot of new opportunities, right? So, and one of the things. that I'm curious about is, you know, there are all these debates right now about, you know, remote work or people being together. and you know, I think this gets us. a lot closer to being able. to work physically in different places,

but actually have it feel like we're together. So, you know, I think that the dream. is that people will one day be able. to just work wherever they want, but we'll have all the same opportunities. because you'll be able. to feel like you're physically together. I think we're not there today. with just video conferencing. and the basic technologies that we have. But I think part of the idea. is that with something like this, over time, you could get closer to that. and that would open up. a lot of opportunities, right? Because then people could live physically. where they want while still being able.

to get the benefits of being physically. or kind of feeling like you're together with people at work. All the ways that that helps to build more culture. and build better relationships and build trust, which I think are real issues that. if you're not seeing people, you know, in person ever. So yeah, I don't know. I think it's gonna be, it's very hard from first principles. to think about all the implications. of a technology like this. and, you know, all the good. and the things that you need to mitigate.

So, you know, you try to do your best. to kind of envision what things are gonna be like. and accentuate the things that're gonna be awesome. and hopefully mitigate some. of the downside things. But, you know, the reality is. that we're gonna be building. this out one year at a time. It's gonna take a while. So we're gonna just get to see how it evolves. and what developers and different folks do with it. - If you could comment, this might be a bit of a very specific technical question, but Llama 2 is incredible.

You've released it recently. There's already been a lot. of exciting developments around it. What's your sense about its release. and is there a Llama 3 in the future? - Yeah, I mean, I think. on the last podcast that we did together, we were talking about the debate. that we were having around. open sourcing Llama 2, and I'm glad that we did, you know, I think at this point there's the value. of open sourcing, a foundation model like Llama 2.

is significantly greater than the risks. in my view. I mean, we spent a lot of time, took a very rigorous assessment of that. and red teaming it. But I'm very glad that we released Llama 2. I think the reception has been, it's just been really exciting to see how excited people. have have been about it. And it's gotten way more, you know, downloads and usage. than I would've even expected. And I was pretty optimistic about it.

So that's been great. Llama 3? I mean, there's always another model that we're training. So I mean it's, you know, for right now, you know, we built, we train Llama 2 and we released it. as an open source model. And right now the priority is building that. into a bunch of the consumer products, all the different AIs. and a bunch of different products. that we're basically building as consumer products.

'Cause Llama 2 by itself, it's not a consumer product, right? It's more of a piece of infrastructure. that people could build things with. So that's been the big priority, is kind of continuing to fine tune. and kind of just get Llama 2, and it's little, the branches. that we've built off of it, ready for consumer products. that hopefully, you know, hundreds of millions of people. will enjoy using those products and billions one day. But yeah, I mean we're also working.

on the future foundation models. and I don't have anything new or news on that. I don't know, you know. I don't know exactly when it's gonna be ready. I think just like we had a debate around Llama 2. and open sourcing it, I think we'll need to have a similar debate. and process to red team this. and make sure that this is safe. But, and my hope is that we'll be able. to open source this next version when it's ready too. But, we're not, you know, close to doing that this month.

I mean, it's a thing. that we're still somewhat early and working on. - Well, in general, thank you so much for open sourcing Llama 2. and for being transparent. about all the exciting developments around AI. I feel like that's contributing to a really awesome. conversation about where we go with AI. And obviously it's really interesting. to see all the same kind of technology integrated. into these personalized AI systems.

with the AI personas, which I think will, when you put in people's hands. and they get to have conversations. with these AI personas, you get to see like interesting failure cases, like where the things are dumb. or they go into weird directions. and we get to learn as a society together. what's too far, what's interesting, what's fun, How much personalization is good, how much generic is good? And we get to learn all of this. And you probably don't know this yourself. Like we have to all figure it out by using it, right?

- Yeah. I mean, part of what we're trying to do. with the initial AI's launch. is having a diversity. of different use cases. just so that people can try different things. 'cause I don't know what's gonna work. I mean, are are people gonna like playing, you know, the text-based adventure games. or are they going to, you know, like having a comedian. who can add jokes to threads. or they can wanna interact with historical figures. You know, we made one of Jane Austin. and one of Marcus Aurelius.

and I'm curious to see how that goes. - I'm excited for both. - Yeah? - As a big fan, I'm excited for both to have conversations with them. I mean, yeah. You know, and I am also excited. to see, you know, the internet, I dunno if you heard, can get kind of weird and I applaud them for it. - I've heard that, yeah. - Yeah, So it'd be nice to see. how weird they take it. What kind of memes are generated from this. And I think all of it is, especially in these early stages. of development as we progress.

towards AGI, it's good to learn. by playing with those systems and interacting. with them at like a large scale, like you said. - Yeah, totally. I mean that's why, well, so we're starting out with a set. and then we're also working on this platform. that we call AI Studio that's gonna make it. so that, you know, over time, anyone will be able to create, you know, one of these Ais, almost like they create. any other UGC content across the platform. So I'm excited about that. I think that to some degree,

we're not gonna see the full potential of this. until you just have the full creativity. of the whole community being able to build stuff. But there's a lot of stuff. that we need to get right. So, I'm excited to take this in stages. I don't think anyone out there is really doing. what we're doing here. I think that there are people. who are doing kind of like fictional. or consumer oriented character type stuff, but the extent to which we're building it out. with the, you know, avatars and expressiveness.

and making it so that they can interact across, you know, all of the different apps. and they'll have profiles and, you know, we'll be able to engage people on Instagram and Facebook. I think, it's gonna be really fun. - Well, I'm still, so we're talking about AI, but I'm still blown away this entire time. that I'm talking to Mark Zuckerberg. and you're not here, but you feel like you're here. I've done quite a few intimate conversations.

with people alone in a room, and this feels like that. So I keep forgetting for long stretches of time. that like, we're not in the same room. And for me to imagine a future. where I can with a snap of a finger, do that with anyone in my life, the way we can just call right now. and have this kind of shallow 2D experience. to have this experience. like we're sitting next to each other. is like, I don't think we can even imagine.

how that changes things. where you can immediately. have intimate one-on-one conversations with anyone. That might, like, in a way we might not even predict, change civilization. - Well, I mean this is a lot of the thesis. behind the whole metaverse is giving people. the ability to feel like you're present with someone. I mean, this is like the main thing. I talk about all the time, but I do think that there's a lot. to process about it. I mean, from my perspective, I mean, I'm definitely here. We're not physically in the same place.

It's not like, you know, you're not. talking to an AI, right? So, I think the thing that's novel. is the ability to convey, through technology, a sense of almost physical presence. So the thing that is not physically real. is us being in the same physical place, but kind of everything else is. And I think that that gets. to this somewhat philosophical.

question about what is the nature of kind. of the modern real world? And I just think it really is. this combination of physical world. and the presence that we feel, but also being able to combine that. with this increasingly rich and powerful. and capable digital world that we have. and all of the innovation that's getting created there. So I think it's super exciting. 'cause I mean, the digital world is just increasing. in its capability and our ability to do awesome things.

But the physical world is so profound, and that's a lot of what makes us human. is that we're physical beings. So I don't think we wanna run away. from that and just spend. all day on a screen. And that's like, you know, it's one of the reasons why I care so much. about helping to shape and accelerate. these future computing platforms. I just think this is so powerful. And, you know, even though the current version. of this is like you're wearing a headset, I just think this is gonna be by far the most human. and social computing platform that has ever existed.

And that's what makes me excited. - Yeah, I think just to linger. on this kind of changing nature of reality, like of what is real, maybe shifting it towards the sort of consciousness. So what is real is the subjective experience. of a thing that makes it feel real. versus necessarily being in the same physical space. 'cause it feels like we're in the same physical space.

- Yeah. - And that the conscious experience of it, that's probably what is real, not like that the space time, like the physics of it. Like you're basically breaking physics. and focusing on the consciousness. That's what's real. Just whatever's going on inside my head. - But there are a lot of social. and psychological things that go along with that experience. that was previously only physical presence, right? I think that there's like an intimacy, a trust.

You know, there's a level of communication. because so much of communication. is non-verbal and it's based on expressions. that you're kind of, you know, you're sharing. with someone when you're in this kind of environment. And before those things would've only been possible, you know, had, you know, I gotten on a plane. and flown to Austin and sat, you know, physically with you in the same place. So I think we're basically short-cutting those laws. of physics and delivering. the social and psychological benefits.

of being able to be present. and feel like you're there with another person, which I got real benefits to anyone in the world. And I think that like you said, I mean, I think that is gonna be a very profound thing. and that a lot of that is, you know, that's the promise of the metaverse. and well, I think that that's the next frontier. for what we're working on. You know, I started working on social networks. when they were primarily text. where the first version of Facebook,

your profile, you know, you had one photo and the rest of it was like. lists of things that you were interested in. And, then we kinda went through the period. where we were doing photos and you know, now we're kind of in the period. where most of the content is video, but there's a clear trend where, you know, over time the way that we want to express ourselves. and kind of get insight and content. about the world around us. gets increasingly just richer and more vivid. And I think the ability to be immersed.

and feel present with the people around you. or the people who you care about is, from my perspective, clearly the next frontier. It just so happens. that it's incredibly technologically difficult, right? It requires building up these new computing platforms. and completely new software stacks to deliver that. But I mean, I kind of feel like. that's what we're here to do as a company. - Well, I really love the connection you have. through conversation. And so for me, this photo realism. is really, really exciting. I'm really excited for this future.

and thank you for building it. Thanks to you, and thanks to the amazing Meta teams that I've met, the engineers and just everybody I've met here. Thank you for helping to build this future. And thank you Mark for talking to me inside the Metaverse. This is blowing my mind. I can't quite express. I would love to measure my heart rate this whole time. It would be hilarious if you're actually. like sitting on a beach right now.

- I'm not. I'm in a conference room. - Okay, well, I'm at a beach and not wearing any pants. I'm really sorry about that. for anyone else who was watching me. in physical space. Anyway, thank you so much for talking today. This really blew my mind. It's one of the most incredible experiences in my life. So thank you for giving that to me. - Awesome, awesome. Glad you got to check it out and it's always fun to talk. All right, I'll catch you soon. See you. - See you later. This is so, so amazing, man. This is so amazing.