Travis Oliphant: NumPy, SciPy, Anaconda, Python & Scientific Programming | Lex Fridman Podcast #224

the following is a conversation with. travis oliphant. one of the most impactful programmers. and data scientists ever. he created numpy. scipy and anaconda. numpai formed the foundation of. tensor-based machine learning in python. scipy formed the foundation of. scientific programming in python and. anaconda specifically with conda made. python more accessible to a much larger. audience. travis's life work across a large number.

of programming and entrepreneurial. efforts has and will continue to have. immeasurable impact on millions of lives. by empowering scientists and engineers. in big companies small companies and. open source communities to take on. difficult problems and solve them with. the power of programming. plus he's a truly kind human being which. is something that when combined with. vision and ambition makes for great. leader and a great person to chat with.

to support this podcast please check out. our sponsors in the description. this is the lex friedman podcast and. here is my conversation with travis. oliphant. what was the first computer program. you've ever written do you remember whoa. that's a good question i think it was in. fourth grade just a simple uh loop in. basic basic basic on an atari 800 atari. 400 i think or maybe there's an atari. 800 it was at a part of a class and we. just were just basic loops.

to print things out. did you use go to statements um yes yes. we used go to statements. i remember in the early days that's when. i first realized there's like principles. to programming when i was told that. don't use go to statements those are bad. software engineering prints like it goes. against. what great beautiful code is i was like. oh okay there's rules to this game. i didn't see that until high school when. i took an ap computer science course. right i did a lot of other kinds of just.

programming on ti but finally when i. took an ap computer science course in. pascal. wow that's that was pascal that's when i. oh there are these principles not c or c. plus plus no i didn't take c until uh. the next year in college i had a course. in c um but i haven't done much in. pascal just that ap computer science. course. now sorry for the romanticized question. but when did you first fall in love with. programming oh man good question i think. actually when i was 10 you know my dad.

got us a t a timex sinclair. and uh he was excited about the. spreadsheet capability and then but i. made him get the basic the add-on so we. could actually program in basic and just. being able to write. instructions and have the computer do. something then we got a ti-99. ti-994a when i was about 12 and i would. just it had sprites and graphics and. music you could actually program to do. music. that's when i really sort of fell in. love with programming so this is a full. like a real computer with like uh.

with memory and storage yeah processors. so what not the type of ti yeah the. timex sinclair was one of the very first. it was a cheap cheap like i think it was. well it was still expensive but it was. 2k of memory we got the 16k add-on pack. but yeah it had memory and you could. program it you had the in order to store. your programs you had to attach a tape. drive remember that old the sound that. would play when you invented the. converted the the modems would convert. digital bits to audio files on a tape.

drive still remember that sound. but that was the storage and what was. the programming language do you remember. it was basic it was basic and then they. had a visi calc and so a little bit of. spreadsheet programming busy but. mostly just some basic do you remember. what kind of things drew you to. programming was it uh. working with data was it video games and. video games. mathy stuff yeah i've i've always loved. math and. a lot of people think they don't like. math because i think when they're. exposed to it early they uh it's about.

memory. you know when you're exposed to math. earlier you have a good short term. memory members timetables. and i i do have a reasonably i mean not. perfect but a reasonably long um little. short-term memory buffer. and so i did great at times tables i. said oh you're good at math but i. started to really like math just the the. problem solving aspect and so computing. was. problem solving. applied. and so that's always kind of been the. draw kind of coupled with the. mathematics.

did you ever see the computer as like an. extension. of your mind like something able to. achieve not till later okay yeah no not. then it's just like a little set of. puzzles that you can play with and you. can you can play with math puzzles and. yeah it was it was too rudimentary early. on like it was sort of. yeah it was too it was a lot of work to. actually take. a thought you'd have and actually get it. implemented and that's still work but. it's getting easier and. so yeah i would say that's definitely. what's attracted me to python is that. that was more real.

right. i could think in python. speaking a foreign language i only speak. another language fluently besides. english which is spanish and i remember. the day when i would dream in spanish. and you start to think in that language. and then you actually i do definitely. believe that language. limits or expands your thinking. uh there are some languages that. actually lead you to certain. thought processes yeah like uh. so i speak russian fluently and that's. certainly uh. a language that leads you down certain.

thoughts. well yeah i mean there's a um. there's a history of. the two world wars right of the. of millions of people starving to death. or near to death throughout its history. of suffering of. injustice like this promise sold to the. people and then. the the carpet or whatever swept from. under them it's like broken promises and. all of that pain and melancholy is in. the language the sad songs the sad.

hopeful songs the over romanticized like. i love you i hate you the the sort of. the swings between all the various. uh spectrums of emotion so that's all. within the language the way it's twisted. uh poach there's a there's a there's a. strong culture of rhyming poetry so like. the bards like this this thing there's a. musicality to the language too did. dostoevsky write in russian. yeah so like. yes. all the uh.

[Laughter]. all the ones that i know about which are. translated and curious how the. translations so dostoevsky. did not use. the musicality of the language too much. so it actually translates pretty well. because it's so philosophically dense. that the story does a lot of the work. but there's a bunch of things that are. untranslatable certainly the poetry is. not translatable. i actually have a few. conversations coming up offline and also. in this podcast with people who've.

translated dusty esky and that's in for. people who worked. who work in this field know how. difficult that is sometimes you can. spend. you know months thinking about a single. sentence right in in the context like. because there's just the magic captured. by that sentence and how do you. translate it just in the right way. because those words can be um. can be really powerful there's a famous. line. beauty will save the world from. dostoyevsky. you know there's so many ways to. translate that and you're right the.

language gives you the tools with which. to tell the story but it also leads your. mind down certain trajectories and paths. to where over time as you think in that. language you become a different human. being yes yeah yeah that's a fascinating. reality i think that i know people have. explored that but it's just rediscovered. well we don't we live in our own like. little pockets like this is the sad. thing. is. i feel like unfortunately given time and. given getting older i'll never know.

the uh china the chinese world because i. don't truly know the language same with. japanese i don't truly know japanese and. portuguese and brazil that whole south. american continent like yeah i'll go to. brazil and argentina but will i truly. understand the people. if i don't understand the language it's. it's sad because um. i wonder how much how many geniuses were. missing because uh because so much of. the scientific world so much of the. technical world is in english.

and so much of it might be lost because. they're they just we don't have the. common language i completely agree i'm. very much in that vein of. there's a lot of genius out there that. we miss and it's sort of sort of. fortunate when it when it bubbles up. into something that we can understand or. process there's a lot we miss. so i tend to lean towards really loving. uh democratization or things that. empower people or you know i. very resistant to sort of authoritarian. structures.

fundamentally for that reason it well. several reasons but it just hurts us. yeah we're worse off. so speaking of languages that empower. you so. python was the first language for me. that um that i could i really enjoyed. thinking in yeah as you said sounds like. you shared my experience too so when did. you first do you remember when you first. kind of connected with python maybe even. fell in love with python it's a good. question it was a process it took about. a year i first encountered python in. 1997 i was a graduate student studying.

biomedical engineering at the mayo. clinic. and i had previously i've been involved. in. taking information from satellites i was. an electrical engineering student. used to taking information and trying to. get something out of it doing some data. processing information out of it and i'd. done that in matlab i'd done that in. perl i've done that in you know. scripting of on a vms there's actually a. vax vms system and they had their own. little scripting tools uh around fortran. done a lot of that and then. as a graduate student i was looking for.

something. and encounter python and because python. had an array had two things that made me. not filtered away because i was. filtering a bunch of stuff as yorick i. looked at yorick i looked at a few other. languages throughout there at the time. in 1997 but it it had arrays there's a. library called numeric that had just. been written in 95 like not very. not not too much earlier by an mit alum. uh jim huganin. you know and i went back and read the. mailing list to see the history of how. it grew and there was a very interesting.

it's fascinating to do that actually to. see how this emergent. cooperation unstructured cooperation. happens in the open source world that. led to a lot of. this collective uh programming which is. something maybe we might get into a. little later but what that looks like. what gap did numeric fill merrick fill. the gap of having an array object so. there was no array out there was no. array there was a one dimensional. byte concept but there was no. uh n-dimensional two three. four-dimensional tensor they call it now.

i'm still in the category that a tensor. is another thing and it's just an md. array we should call it but yeah kind of. lost that battle. there's many battles in this world some. of which will win some we lose that's. exactly right. so and but it was uh it had no math to. it so numeric had math and a basic way. to think in a race so i was looking for. that and it had complex numbers. a lot of programming languages and you. can see it because you know if you're. just a computer scientist you think ah.

complex numbers just two floats so you. can people can build that on. but in practice a complex number as a as. one of the significant algebras that. helps connect a lot of physical and. mathematical ideas particularly fft for. an electrical engineer. and and it's a really important concept. and not having it means you have to. develop it several times. and those times may not share an. approach one of the common things in. programming one things programming. enables is abstractions. but when you have shared abstractions.

it's even better it sort of gets the. level of language of actually we all. think of this the same way which is both. powerful and dangerous. right because. powerful in that we now can quickly make. bigger and higher level things on top of. those abstractions dangerous because it. also limits us as to the things we left. maybe left behind in producing that. abstraction which is at the heart of. programming today and actually building. around the programming world so i think. it's a fascinating philosophical topic. yeah it will continue for many years i. think. as it builds more and more and more.

abstractions yes i often think about you. know we have we have a world that's. built on these abstractions that were. they the only ones possible yeah. certainly not but they led to. now it's very hard to to do it. differently yeah like there's an inertia. that's very hard to you know. push out push away from there's that has. implications for things like you know. the julia language which you have heard. of i'm sure and. i've met the creators and i like julia. it's a really cool language but they've. struggled to kind of. against the just the tide of like this.

inertia of people using python and. and you know there's strategies to. approach that but nonetheless it's a. it's a phenomena and sometimes so i love. complex numbers and i love to raise so i. looked at python. and then i had the experience i did some. stuff in python and i was just doing my. phd so i was out my focus was on. i was actually doing a combination of. mri and ultrasound and looking at a. phenomena called elastography which is. you push waves into the body. and observe those waves like you can. actually measure them.

and then you. do mathematical inversion to see what. the elasticity is and so that's the. problem i was solving is how to do that. with both ultrasound and mri i needed. some tool to do that with so i was. starting to use python in 1997 in 98 i. went back looked at what i'd written and. realized i could still understand it. which is not the experience i'd had when. doing pearl in 95. right i'd done the. same thing and then i looked back and i. forgotten what it was even saying. now you know i'm not saying it so i that.

that means hey this may work i like this. this is something i can. retain without becoming an expert per se. and so that led me to go i'm going to. push more into this and. then that 98 was kind of the when i. started to fall in love with python i. would say. a few peculiar things about python so. maybe compared to pearl compared to some. of the other languages so there's no. braces yeah. so you. space is used indentation i should say.

is used as part of my language yeah. right uh so. did you. i mean that that's quite a leap uh were. you comfortable with that leap or were. you just very open-minded good question. i was open-minded so it i was. cognizant of the concern and it. definitely has. it has specific. challenges you know cut and pasting for. example you're cutting pasting code and. if your editors aren't supportive of. that if you're put into a terminal. and particularly in the past when.

terminals didn't necessarily have the. intelligence to manage it now now now. ipython and jupyter notebooks handle it. just fine so there's really no problem. but in the past it created some. challenges formatting challenges also. mixed tabs and spaces if you're if. editors weren't you weren't clear on. what was happening you would have these. issues so there were really concrete. reasons about it that i heard and. understood i never really encountered a. problem with it. personally like it was occasional. annoyances but. i really like the fact that it didn't. have all this extra characters right.

that. these extra characters didn't show up in. my visual field when i was just trying. to process understanding a snippet of. code yeah there's a cleanness to it but. i mean the idea is supposed to be that. pearl also has a cleanness to it because. of the minimalism of like how many. characters it takes to express a certain. thing yeah so it's very compact yeah but. what you realize with that compactness. comes. there's a culture that uh prizes. compactness and so the code gets more. and more compact and less and less. readable to a point where it's like.

uh like to be a good programmer in pearl. you write code that's basically. unreadable right there's a culture like. correct and you're proud of it yeah. you're proud of it. right exactly and it's like feels good. and it's really selective like it means. you have to be an expert in perl to. understand it yeah whereas python was. allowed you not to have to be an expert. you'd have to take all this brain energy. you could leverage what i said you could. leverage your english language center. which you're using all the time i've.

wondered about other languages. particularly non-uh latin based. languages you know. latin-based languages where the. characters are at least similar i think. people have an easier time but i don't. know what it's like to be a japanese or. a chinese person trying to. learn a different um. different syntax like what would. computer programming look like in a in. that i haven't looked at that at all but. it certainly doesn't you know leveraging. your your chinese language center i'm. not sure python or any programming does. that. but that was a big deal the fact that it. was accessible i could be a scientist.

what i really liked is. many programming languages really demand. a lot of you and you can get a lot you. know you do a lot if you learn it but. python enables you to do a lot without. demanding a lot of you. there's a there's nuance to that. statement but it certainly was it's more. accessible so more people could actually. as a as a scientist as somebody or. engineer. who was trying to solve another problem. besides point programming i could still. use this language and get things done. and and be happy about it i was also. comfortable in c. at that time and matlab you did a little.

matlab i did a lot before that exactly. so i was comfortable in. those three languages were really the. tools i used during my studies and. schooling. um but to your point about language. helping you think one of the big things. about matlab is it was and apl before it. i don't know if you're a you remember. apl. apl is uh actually the predecessor of. array-based programming which i think is. really an underappreciated. if i talk to people who are just steeped. in computer programming computer science. like most the people that microsoft has. hired in the past for example.

microsoft as a company generally did not. understand array-based programming like. culturally they didn't understand it so. they kept missing the boat kept missing. the understanding of what this what this. was. they've gotten better but there's still. a whole culture of folks that doesn't. programming that's yeah you know that's. that's systems programming or web. programming or lists and maps and you. know what about an n-dimensional array. oh yeah that's just an implementation. detail well. you can think that but then actually if. you have that as a construct you. actually think differently apl was the.

first language to understand that it was. in the 60s. right the challenge of apl is apl had. very dense not only glyphs like new. characters new glyphs they even had a. new keyboard because to produce those. glyphs this is back in the early days of. computing when you know the. query keyboard maybe wasn't as. established like what we could have a. new keyboard no big deal. but it was a big deal and it didn't. catch on and the language apl. very much like pearl as people would. pride themselves on how how much could. they write the game of life in.

30 characters of apl apl has characters. that mean. uh summation and uh they have adverbs. you know they have adjectives and these. things called adverbs which are like. methods like reduction reduction it. would be an adverb on an ad operator. right so. but doing using these tools you could. construct and then you start to think at. that level you think in end dimensions. it's something i like to say and you. start to think differently about data at. that point you know now you're. it really helps yeah i mean. outside of programming if you really.

internalize linear algebra as a course. i mean it philosophically allows you to. think of the world differently yes it's. almost like liberating you don't have to. you don't have to think about the. individual numbers in the n-dimensional. array you can think of it as an object. in itself and all of a sudden this world. can open up. now you're saying matlab and apl were. like the early c. i don't know if many languages got that. right ever no no no they didn't still. even still i would say i mean. numpy is a as a inheritor of the.

traditions that i would say apl j. was a another version that was what it. did is not have the glyphs just have. short characters but still a latin. keyboard could type them and then. numeric inherited from that in terms of. let's add arrays plus broadcasting plus. methods reduction even some of the. language like rank is a concept that's. in that was in python it's still in. python. for the number of dimensions right. that's that's different than say the. rank of a matrix which people think of.

as well so it's it came from that. tradition but numpy. is a very pragmatic practical tool uh. numpy inherited from numeric and we can. get to where numpy came from which is. the current array. at least current as of. 2016-17 now there's a ton of them over. the past two or three years but we can. get into that too so if we just sort of. linger on the early days of. what was your favorite feature of python. do you remember like what yeah. it's so interesting. to linger on like the.

what. what really makes you connect with the. language i'm not sure it's. obvious to introspect that no it isn't. and i've thought about that at some. length i'm not i think definitely the. fact that i could read it later yeah. that i could use it productively without. becoming an expert and the other. language i had to put more effort into. right. that's like an empirical observation. like you're not analyzing any one aspect. of the language it just seems. time after time you look back it's. somehow readable it's somewhat readable.

then it was sort of i could take. uh executable english yeah and translate. it to python more easily like i didn't. have to go there was no translation. layer. as an engineer or as a scientist i could. think about what i wanted to do and then. the syntax wasn't that far behind it. yeah right now there was some there have. some there's some warts there still it. wasn't perfect like there's some areas. where i'm like ah it'd be better if this. were different or if this were different. some of those things got out of the. language too i was really grateful for. some of the early pioneers in the python. ecosystem back because python got.

written in 91 is when the first version. came out but guido was very open to. users and one of the sets of users were. people like jim hugin and david asher. and paul dubois. and. conrad hinson these were people that. were on the main list and they were just. asking for things like hey we really. should have complex numbers in this. language so let's you know there's a j. there's a one j right and the fact they. want the engineering root of j is. interesting. i don't i don't think that's entirely. favorite engineers i think it's because. i is so often used as the index of a for.

loop. so i think that's actually. probably right i mean there's there's a. pragmatic aspect like the complex. numbers were there i love that the fact. that i could write nd arrays constructs. and that reduction was there very simple. to write summations and and and. broadcasting was there i could do. addition of whole arrays. um so that was cool those were something. i loved about it. i don't know what to start talking to. you about because you've been you've. created so many incredible projects that. basically changed the whole landscape of. programming but okay let's start with uh.

let's go chronologically. with scipy you create a scipy over two. decades ago now yes right yeah i said. i'd love to talk about sci-fi sci-fi was. really my baby. what is it. uh what was its goal what is its goal. how does it work yeah fantastic so scipy. was effectively here i'm using python to. do. stuff that i previously used matlab to. use and i was using numeric which is an. array library that made a lot of it. possible but there's things that were. missing like i didn't have an ordinary.

differential equation solver i could. just call right i didn't have. integration hey i wanted to integrate. this function okay well i don't have. just a function i can call to do that. um these are things i remember being. critical things that i was missing. optimization i just want to pass a. function to an optimizer and have it. tell me what the optimum value is. uh those are things like well why don't. we just write a library that adds these. tools. and i started a post on the mailing list. and they're previously been you know. people have discussed i remember conrad. hinson saying wouldn't it be great if we. had this optimizer library or david ash.

would say this stuff and and i'm you. know i'm a. ambitious i am this is the wrong word. and eager. and. uh. probably more time than sense i was you. know poor graduate student. uh my wife thinks i'm working on my phd. and i am but part of a phd that i loved. was the fact that it's exploratory. you're not just you know. taking orders fulfilling a list of. things to do you're trying to figure out. what to do and so i thought well you. know i'm writing tools for my own use. and a phd so.

i'll just start this project and so in. 99 98 was when i first started to write. libraries for python particularly when i. fell in love with python 98 i thought. well there's just a few things missing. like oh i need a reader to read dicom. files i was in medical imaging.com was a. format that i want to be able to load. that into python okay how do i write a. reader for that so i wrote something. called. it was an i o package right and that was. my very first extension module which is. c so i wrote c code to extend python so. that the pos in python i could write. things more easily that that combination.

kind of hooked me it was the idea that i. could here's this powerful tool i can. use as a scripting language and a high. level language to think about but that i. can extend easily. easily in c that easily for me because i. knew enough c right and then guido had. written a link i mean the only the hard. part of extending python was something. called the way memory management works. and you have to reference counting and. so there's there's a tracking of. reference counting you have to do. manually. and if you don't you have you have. memory leaks and uh so that's hard plus. then c you know it's just much more you.

have to put more effort into it it's not. just i have to now think about pointers. and have to think about stuff that. is different i have to kind of you're. like putting a new cartridge in your. brain like you're okay i'm thinking. about mri now i'm thinking about. programming and there are distinct. modules you end up having to think about. so it's harder when i was just in python. i could just think about mri and. high-level writing. but i could do that and that kind of i. liked it i found that to be enjoyable. and fun and so i ended up oh well let me. just add a bunch of stuff to python to. do integration.

well and the cool thing is is that you. know the power of the internet i just. looking around and i found oh there's. this net lib which has. hundreds of fortran routines that people. written in the 60s and the 70s and the. 80s in fortran 77 fortunately it wasn't. for trend sixties i've been imported to. fortran 77. and 1477 is actually a really great. language. fortune 90 probably is my favorite 4chan. because it's also it's got complex. numbers got a raise and it's pretty high. level now the problem with it is you'd. never want to write a program in fortune.

90 or fortune 77 but it's totally fine. to write a subroutine in. right and so and then 4chan kind of got. a little off course when they tried to. compete with c plus plus but at the time. i just want libraries to do something. like oh here's an order different. equation here's integration here's run. cut integration. already done i don't have to think about. that algorithm and you could but it's. nice to have somebody who's already done. one and tested it and so i sort of. started this journey in 98 really if you. look back at the main list there's sort. of this this productive era of me. writing an extension module to connect.

wrench cut integration to python. and making an ordinary digital equation. solver and then. releasing that as a package so we could. call od pack i think i called it then. quad pack and then i just made these. packages eventually that became. multi-pack because they're originally. modular you can install them separately. but a massive problem in python was. actually just getting your stuff. installed. at the time releasing software for me. like today it's people think what does. that mean well then it meant. some poorly written web page i had some.

bad web page up and i put a tarball just. a gzip tar ball of source code that was. the release. but okay can we just stand that because. that the community aspect of creating. the package and sharing that yes. that's rare. that to have to both have. at that time so like that was pretty. early yeah so well not not rare maybe. maybe you can uh correct me on this but. it seems like in the scientific. community so many people you were.

basically solving the problems you. needed to solve. to process the particular application uh. the data that you need and. to also have the mind that i'm going to. make this usable for others. that's um i would say i was inspired i'd. been inspired by linux i've been. inspired by you know linus linus and him. making his code available and i was. starting to use linux at the time and i. went this is cool so i had kind of been. previously primed that way and generally. i was i was into science because i liked. the sharing notion i like the idea of.

hey let's if collectively we build. knowledge and share it we can all be. better off okay so you weren't energized. by that so it's energized value already. yeah right and i can't deny that i was. i'm sort of uh had this very. i liked that part of science that part. of sharing and then all of a sudden oh. wait here's something and here's. something i could do. and then i slowly over years learned how. to share better so that you could. actually engage more people faster one. of the key things was actually giving. people a binary they could install. right so that wasn't just your source.

code good luck. compile this and then get it compiled. ready to install you just you know so in. fact a lot of the journey from 98 even. through 2012 we used to when i started. anaconda was about that like it's why uh. you know it's really the key as to why. the scientists with dreams of doing mri. research ended up starting a software. company that installs. software i work with. a few folks now that don't program. like on the creative side and the video. side the audio side and because my whole.

life is running on scripts i have to try. to get them to i'm have now the task of. teaching them how to do python enough. yeah to run the scripts and so i've been. actually facing this whether it's on the. condor some. with the task of how do i minimally. explain basically to my mom how to write. a python script. and it's an interesting challenge. it's a to-do item for me to figure out. like what is the minimal amount of. information i have to teach what are the. tools you use that one you enjoy it to. your effect of it they're related to two.

related questions and then the debugging. like the the iterative process of. running the script to figure out what. the error is maybe even for some people. to do the fix yourself yeah so do you. compile it do this like how do you. distribute that code to them and it's. interesting because i think. it's exactly what you're talking about. if you increase the circle. of empathy that the circle of people. that are able to use your programs. you increase it its like effectiveness.

and its power and so yeah you have to. think. you know can i write scripts can i write. programs that can be used by biomedical. engineers by all kinds of. people that don't know programming and. actually maybe plan to see. have them catch the bug of programming. so that they start on their journey. that's a huge responsibility and. ultimately has to do with the amazon. one-click buy. like how how frictionless can you make. the early steps frictionless is actually. really key to growing any community is.

every any friction point you're just. going to lose you're going to lose some. people yeah right now sometimes you may. want to. intentionally do that if you're early. enough on you need. a lot of help you need people who have. the skills you might actually it's. helpful you don't necessarily have too. much too many users as opposed to. contributors if the co if you're early. on. anyway there's uh uh sci-fi started in. 98 but it really emerged as this. collection of modules that i was just. putting on the net people were. downloading and they you know. i think i got 100 users right by the end.

of that year but there but the fact that. i got 100 users and more than that. people started to email me with fixes. like and that was actually intoxicating. right that was the. that was the you know here i'm writing. papers and i'm giving conferences and i. get people would say hello but yeah good. job but mostly it was you're reviewed. with. it it's competitive yeah right you. publish a paper and people were like oh. it wasn't my paper you know. i was starting to see that sense of. academic. life where it was so much i thought.

there was a cooperative effort but it. sounds like we're here just to. one-up each other right and. you know it's not it's not true across. the board but a lot of that's there but. here in this world i was. getting responses from people all over. the world. uh you know i remember pierrot peterson. in estonia right was one of the first. people and he sent me back this make. file because the first thing it is yeah. your build thing stinks and here's a. better make file now it was a complex. make file i think i never understood. that make file actually but it worked. and it did a lot more and so then thanks. this is cool and that was my first kind.

of engagement with community. development but you know the process was. he sent me a patch file i had to upload. a new tar ball and i just found i really. loved that and the style back then was. here's a main list it was very it wasn't. as it certainly were the tools that are. available today it was very early on but. i really started that's the whole year i. i think i did about seven packages that. year right and then by the end of the. year i collected them into a thing. called multi-pack so 99 there was this. thing called multi-pack and that's when.

a high school student knows a high. school student at the time a guy named. robert kern. took. that package and made a windows. installer. right and then of course a massive. increase of usage so by the way most of. this development was under linux yes yes. it was on linux i was a linux developer. doing it on munix box i mean at the time. i was actually getting into i had a new. hard drive he did some kernel. programming to to make the hard drive. work i mean not programming but. modification to the kernel so i could. actually hard drive working.

i i love that aspect of it i was also in. you know at school i was building a. cluster i took mac computers like uh and. you put yellow dog linux on them uh they. were at the mayo clinic they were just. they're all these macs that were older. they were just getting rid of and so i. kind of got permission to go grab them. together i put about 24 of them together. in a cluster and a cabinet. and put yellow dog linux on them all and. i wrote a c plus plus um. program to do mri simulation that was. what i was doing. at the same time for my day job so to.

speak so i was loving the whole process. at the same time i was oh i need to. ordinary differential equation that's. why ordinary difference equations were. key was because that's the heart of a. block equation for simulated mri is a. ode solver and so that's. but i actually did that. it doesn't happen at the same time. that's why it kind of what you're. working on and what you're interested in. they're coinciding i was definitely. scratching my own itch. in terms of building stuff and uh which. helped in the sense that i was using it. for me so at least had one user yeah i. had one person who's like well i know. this is better i like this interface.

better and i had the experience of. matlab to guide some of what those apis. might look like but you know you're just. doing yourself you're building all this. stuff but with the windows installer it. was the first time i realized oh yeah. the binary installer really helps people. and so. that led to spending more time on that. side of things so around 2000 so i. graduated my phd in 2000 end of year. 2000. so. 99 doing a lot of work there 98 do a lot. of work there 99 kind of spending more. time on my phd you know helping people.

use the tools thinking about what i want. to go from here there was a company. there's a guy actually eric jones and. travis vott they were two friends who. founded a company called nthot it's here. in austin still here. and they. eric contacted me at the time when i was. a uh. i was a graduate student still and he. said hey why don't you come down we want. to build a company. you know we want we're thinking of you. know a scientific. company and we want to take what you're. doing and kind of add it to some stuff. that he'd done he'd written some tools.

and then pierre peterson had done ftp. let's come together and build pull this. all together and call it sci-fi. so that's the origin of the scipy brand. it came from you know multi-pack and a. whole bunch of modules i'd written plus. a few things from some other folks and. then pulled together in a single. installer. sci-fi was really a distribution of. python masquerading as a library. how did you think about sci-fi in. context of python in context of numeric. like what we saw scipy as a way to make. an r d environment for python like use.

python uh dependent on numeric so. numeric was the array library we. depended on and then from there ext. extend it with a bunch of modules that. allowed for and at the time the original. vision of scipy was to have. plotting was to have you know replied. you know the rebel environment and kind. of a whole really a whole data. environment. um that you could then install and get. going with and that was kind of the. thinking. it didn't really evolve that way right. it sort of had a but one. it's really hard to do massive scale. projects in a.

with with open source collectives. actually there's a there's sort of an. intrinsic uh cooperation limit. as to which you know too many cooks in. the kitchen you know you can do amazing. infrastructure work when it comes down. to bringing it all together into a. single deliverable that actually. requires a little more. a little more product management that is. not. it doesn't really emerge from the same. dynamic so it struggled you know. struggled to get. almost too many voices it's hard to have. everybody agree you know consensus. doesn't really work at that scale you.

end up with politics you know with the. same kind of things that's happened in. large organizations trying to decide on. what to do together um. so consensus building was still was was. challenging at scale as more people came. in right early on it's fine because. there's nobody there and so it works but. then as you get more successful the more. people use it all of a sudden oh there's. this this. scale at which this doesn't work anymore. and we have to come up with different. approaches so sci-fi came out officially. in 2001 was the first release most the. time i remember the days of getting that.

release ready it was a windows installer. and there was there were bugs on how you. know the windows compiler handled. complex numbers and you were you're. chasing segmentation faults and it was. it's a lot of work there's a lot of. effort had nothing to do with my. area of study at the same time i just. got an offer so he wondered if i wanted. to come down and help him start that you. know start that company with his friend. and i at the time i was like i was. intrigued but i was squaring a path an. academic path and i just got an offer to. go and teach at my alma mater so i took.

that tenure track position. and saipo and kind of then i started. work on sci-fi as a professor too. okay so that that's i left i've got the. mayo clinic graduated wrote my thesis. using sci-fi wrote you know there's. there's images that were created. now the plotting tool i used was. something from yorick actually it was a. plotting a plt kind of a plotting. language that i used york is a. programming language it was a. programming language had a plotting tool. dislin. it we had integration to dislike i ended. up using dislin plus some some of the.

plotting from yorick. linked to from python anyway it was a. people don't plot that way now but this. is before and scipy was trying to add. plotting yeah right. it didn't have much success really the. success of plotting came from john. hunter. who had a similar experience to my. experience my kind of maverick. experience as a person just trying to. get stuff done and kind of having more. time than than money maybe right and. john hunter created what not plot lube. he's the creator of map yeah so john. hunter was uh you know he wasn't a. student at the time but he was an actor.

he was working in quant field and he. said we need better plotting so he just. went out and said cool i'll make a new. project and we'll call it matplotlib and. he released in 2001. about the same time. that scipy came out and it was separate. library separate install. use numeric sci-fi use numeric. and so scipy you know 2001 released. scipy and then m-thot created a. conference called scipy which. was brought people together to talk. about the space another conference is. still ongoing it's one of the favorite. conferences of a lot of people because. it's. it's changed over the years but early on.

it was you know a collection of 50. people who care about. scientists mostly. practicing scientists who want to care. about. coding and doing it well and not using. matlab. i remember being driven by you know i. like matlab but i didn't like the fact. that like so i'm not opposed proprietary. software i'm actually not an open source. zealot i love open source for the what. it brings but i also see the role for. proprietary software what i didn't like. was the fact that i would develop code. and publish it and then effectively. telling somebody here to run my code you.

have to have this proprietary software. right and there's also culture around. matlab. as much because i've talked to a few. folks. math works great it's my life yeah. i mean there's just a culture they try. really hard but it's just there's this. corporate ibm style culture that's like. or whatever i don't don't want to say. negative things about ibm or whatever. but there's a. no it's it's really that connection. something i'm in the middle of right now. is is the business of open source and. how do you connect the ethos of. cooperative development with the.

necessity of. of creating profits. right and like right now today you know. i'm still i'm still in the middle of. that that's actually the early days of. of me exploring this question. because i was writing sci-fi i mean as. an aside i also had so i had three kids. at the time i have six kids now i got. married early wanted a family uh i had. three kids and i remember reading i. remember read richard stallman's post. and i was i was a fan of stallman i. would read his work i liked this. collective ideas he would have certainly. the ideas on ip law i read a lot of.

stuff but then he said you know. okay. well how do i make money with this how. do i make a living how do i pay for my. kids all this stuff was in my mind a. young graduate student making no money. thinking i got to get a job and he said. well you know i think just be like me. and don't have kids right that's just. don't don't that's his take on this that. was just that was that was the what what. he said in that moment right that's the. thing i read and i went. okay this is a train i can't get out. yeah. there has to be a way to preserve the. culture of open source and still be able. to make sufficient money to feed you yes.

exactly there's got to be well so that. actually led me to a study of economics. because at the time i was ignorant and. it really was i'm actually i'm. embarrassed for educational system that. they could let me and i was. valedictorian in my high school class. and i did super well in college and like. academically i did great. right but the fact that i could do that. and then be clueless about this key part. of life. it led me to go there's a problem like i. should i should have learned this in. fifth grade i should learn this in. eighth grade like everybody should come. out with a basic knowledge of economics.

you're an interesting example because. you've created tools that uh change the. lives of probably millions of people and. the fact that you don't understand at. the time of the creation of those tools. the basics economics of how like to. build up giant system is a problem yeah. it's a problem and so i during my phd at. the same time this is actually in 98 99. at the same time i was in the library i. was reading books on capitalism i was. reading books on marxism i was reading. books on you know what is this thing. what does it what does it mean yeah and. i encountered a basically what i.

encountered a set of writings from. people that said they were the. inheritors adam smith but adam smith for. the first time right which is the wealth. of nations and kind of this notion of. emergent. emergent uh societies and realized oh. there's this whole world out here of. people. and in the challenge the economics is. also political. like because economics you know. people. different parties running for office. they'll. they want their economic friends they. want their economist to back them up. right or to to be there.

to be their magicians like the magicians. in pharaoh's court right the people that. are going to say hey this is you should. listen to me because i've got the expert. who says this. and so it gets really muddled right but. i was looking at from as a scientist as. a scientist going what is this space. what does this mean how do people how. does paris get fed how does how what is. money how does it work i found a lot of. writings i really loved i found some. things that i really loved and i learned. from that it was writings from people. like von mises he wrote a pre-order. paper in 1920 that still should be read. more than it is it's got i mean it was.

the economic calculation problem of the. socialist commonwealth it's basically in. response to the bolshevik revolution in. 1917. and his basic argument was it's. not going to work to not have private. property you're not going to be able to. come up with prices the bureaucrats. aren't going to be able to determine how. to allocate resources without a price. system and a price system emerges from. people making trades and they can only. make trades if they have authority over. the thing they're trading. and that that that creates information. flow that you just don't have. if you try to top down it right right.

it's like huh that's a really good point. yeah the prices have a signal that's. used and it's important to have that. signal. when you're trying to build a community. of productive people like you would in. the software engineering yeah the prices. are actually an important. signaling mechanism yeah right and that. money. is just a bartering tool right so this. is the first time i've encountered any. of this concept right and the fact that. oh this is actually. really critical like it's so critical to. our prosperity and.

that. we're dangerously. not learning about this not teaching our. children about this you know so you had. the three kids you had to make some. stuff how to make some money right i had. to figure it out but i didn't really. care i mean i was never i've never been. driven by money just need it right right. to eat so what how did that resolve. itself in terms of sci-fi. so i would say it didn't really resolve. itself it sort of started a journey that. i'm continuing on i'm still on i would. say i don't think it resolved itself but. i will say. i i went in wide eyes wide open like i. knew that there were problems with you.

know um giving stuff away and creating. uh the the ex market externalities the. fact that yeah people might use it and i. might not get paid for it and i'll have. to figure something else out to get paid. like at least i can say i'm not bitter. that a lot of people have used stuff. that i've written and i haven't. necessarily benefited economically from. it like yeah i've heard other people be. you know bitter about that when they. write or they talk like oh i should have. got more value out of this and i'm also. i want to create systems that let people. like me who might have these desires to.

do things let them benefit so it. actually creates more of the same. not to turn on your bitterness module. but. there's some aspect i wish there was. mechanisms for me to reward whoever. created scipy and numpy because it. brought so much joy to my life i. appreciate that i mean the tip dark. notion was there i appreciate that and i. think but there should be a very there's. surely mechanism mechanism i totally. agree i would love to talk about some of. the ideas i have because i actually came. across i think i've come up with some. interesting notions that could work but. they'll require.

you know anything that will work takes. time to emerge right like things don't. just turn overnight that's definitely. one thing i've also understood and. learned. is any fixes. that's why it's kind of funny we often. give credit to you know oh this. president gets elected and oh look how. great things have done. and. i saw that when when i had a transition. in a condo when a new ceo came in right. and it's like the success that's. happening there's an inertia there yeah. right and sometimes the decision you. made like 10 years before is the reason. why the successes see right exactly so. we're sort of just running around taking.

credit for stuff credit assignment has. like a delay to it yes. that this makes the credit assignment. basically wrong more than right wrong. more than right exactly and so i'm like. oh this is you know that's the stuff i. would i would read a ton about you know. early on so i don't i feel like i'm with. you like i want the same thing i want to. be able to and honestly not for. personally i've been happy i've been. i've been happy i feel like i don't have. any i mean we've been done reasonably. okay but i've had to pursue it like. that's that's really what started my. um trajectory from academia is reading.

that stuff letting me say oh. entrepreneurship matters. so i love software but entre but we need. more entrepreneurs and i want to. understand that better so once i kind of. had that that virus infect my brain. it even though i was on a trajectory to. go to a tenure track position at a. university. and i was there for six years i was kind. of already out the door. when i started and we can get into that. but yeah um what can i just ask a quick. question on.

is there some design principles that. were in your mind around sci-pi like is. there some key ideas that were just like. sticking to you that this is this is the. fundamental ideas yeah i would say so i. would think it's basically accessibility. to scientists like give them give. scientists and engineers tools they. don't have to think a lot about. programming so give them really good. building blocks give them functions that. they want to call and. sort of just the right length of. spelling. you know. there's a. one tradition in programming where it's. like you know make very very long names.

right and you can see it in some. programming languages where the names. get you know take half the screen and. i in. in the 4chan world characters would have. to be six six letters early on right and. that's way too too much too too little. but i was like. i like to have names that were. informative but short so even though. python. well this is a different conversation. but. documentation is doing some work there. so when you look at great scientific. libraries and functions there's.

there's a richness of documentation that. helps you get into the details. the first glance that a function gives. you the intuition of all it needs to do. by looking at the headers and so on but. to get the depths of all the. complexities involved all the options. involved documentation does something. else documentation is essential yeah so. that was actually a. so we thought about several things one. is we wanted plotting we wanted. interactive environment we wanted good. documentation these are things we knew. we wanted the reality is those took. about. 10 years to evolve.

right given the fact that we didn't have. a big budget it was all volunteer labor. it was sort of. um. when nthot got created and they started. to you know try to find. projects people would pay for pieces and. they were able to fund some of it. not nearly enough to keep up with what. was necessary. no criticism just simply the reality i. mean it's it's hard to start a business. and then do consulting and also promote. an open source project that's still. fairly new. cyborg is fairly niche. we stayed connected all while i was a. student sorry a professor i went to byu.

and started to teach electrical. engineering all the applied math courses. i loved teaching signal processing. probability theory electromagnetism i. was the if you look at rate my professor. which my kids love to do. i wasn't. like i got some bad reviews because. people. what was the criticism um i would speak. too high too high of a level like i. definitely had a calibration problem. coming out of uh graduate work. where i hate to be condescending to. people like i really have a ton of. respect for people fundamentally like my. fundamental thing is i respect people.

sometimes that can lead to a i was i was. thinking they were. they had more knowledge than they did. and so i would just speak at a very high. level yeah assume they got it but they. need to rise to the standard that you. set i mean that's one of the some of the. greatest teachers do that yeah and i. agree and that was kind of what was. inspiring me but but you know you also. have to. i i cannot say i was uh i was articulate. to some of the greatest teachers right i. was you know like one one classic. example when i first taught at byu my.

very first class it was overheads. transparencies overheads. before projectors are really that common. i saw transparencies i'm writing my. notes out i go in room's half dark. i just blaring through these. transparencies here it is here it is. here it is. and i gave a quiz after two weeks. nobody knew anything. nothing i had gotten anywhere. and i realized okay i'm not this is not. working so i took put away the. transparencies and i turned around just. started using the chalkboard. and what it did is it slowed me down.

right the chalkboard just slowed me down. and gave people time to process and to. think and then that made me focus my. writing wasn't great on their chalkboard. but. i really love that part of like the. teaching so that that entered scipy's. world in terms of we always understood. that there's a didactic aspect of sci-fi. kind of how do you take the knowledge. and then produce it the challenge we had. was the scope like ultimately scipy was. everything right and so 2001 when it. first came out people were starting to. use it no this is cool this is a tool we. actually use. at the same time 2001 time frame there.

was a little bit of um like the hubble. space telescope the folks at hubble has. started saying hey python we're gonna. use python for processing images from. hubble and so perry greenfield was a. good friend and running that program and. he. had called me before i left to byu and. said you know. we want to do this but numeric actually. has some challenges in terms of you know. it's not the array doesn't have enough. types. uh we need more operations you know. broadcast needs to be a little more. settled uh they wanted record arrays. they wanted you know record arrays are. like a data frame but a little bit.

different but they wanted more. structured data so. he had called me even early on then and. they said yeah hey would you want to. work on something to make this work i. said yeah i'm interested but i'm going. here and i you know we'll see if i have. time so in the meantime while i was. teaching and sci-fi was emerging and i. had a student i was constantly while i. was teaching trying a way to fund this. stuff. so i had a graduate student uh my only. graduate student a. chinese fellow. luhongza is his name great guy he wrote. a bunch of stuff for iterative iterative. linear algebra like got into writing.

some of the iterative literary algebra. uh tools that are currently there in. sci-fi and they've gotten better since. but this is in 2005. kept working on sci-fi but. perry had started working on a. replacement to numeric called namurai. and in 2004 a package called in the. image it was an image processing library. that was written for nomare and it had. in it a morphology tool i don't know. what morphology is it's open dilations. close you know there's sort of this. as a medical imaging student i knew what.

it was because it was used in. segmentation a lot and in fact i wanted. to do something like that in python in. scipy but just had never gotten around. to it so when it came out that it worked. only on the num array. and scipy needed numeric and so we. effectively had the beginning of this. split. and numeric and number i didn't share. data they were just two so you could. have a gigabyte of numeric memory data. and gigabyte of numeric data and they. wouldn't share it yeah and so you have. these any of these scientific libraries. written on top.

i got really bugged by that yeah i got. really like oh man this is not good. we're not cooperating now we're not. we're sort of redoing each other's work. and we're just this young community. so that's what led me even though i knew. it was risky because my. you know i had i was on a tenure track. position 2004 i got reviewed they said. hey things are going okay you're doing. well paper's coming out but you're kind. of spending a lot of time in this open. source stuff maybe do a little less of. that and a little more of the paper. writing and grant writing which was. naive but it was definitely the time you. know the thinking it still goes on still.

goes on. you're basically creating a thing which. enables science in the 21st century. right um maybe don't emphasize that so. much in your for your tenure right. it illustrates some of the challenges. yeah. it does it's and it's people mean well. yeah like but we've gotten broken in a. bunch of ways. certain things programming understanding. the role of software engineering and. programming exactly society is a little. bit less exactly no i was in electrical. engineering position right so it was.

even worse. there. yeah it was very they were very focused. and so you know good people and i had a. great time i loved my time i loved my. teaching i loved all the things i did. there the problem was this split was. happening in this community i loved i. saw people and i go oh my gosh this is. going to be. this is not great and so. i happened you know fate. i had a class i had signed up for it. it's a i was trying to build an mri. system so i just i had a kind of a radio. a instead of a radio a digital radio. class as a digital mri class.

uh and i had people sign up two people. signed up then they dropped and so i had. nobody in this class. so. and i didn't have any other courses to. teach and i thought oh i've got some. time. and i'll just write i'll just write a. reply a merger of the american memory. like i'll basically take the numeric. code base add the features number i was. adding. and then kind of come up with a single. array library that everybody can use so. that's where numpy came from was my. thinking hey i can do this and who else. is going to because at that point i'd. been around the community long enough. and i'd written enough c code i knew.

i knew the structures and i in fact my. first contribution to numeric had been. writing the c api documentation that. went in the first documentation for. numpy for numeric sorry this is paul. dubois david asher connor hinson and. myself. i got credit because i wrote this. chapter which is all the c api of. numeric all the c stuff so i said ah i'm. probably the one to do it you know. nobody else is going to do this so it's. sort of out of a sense of duty and. passion. knowing that. i don't think my academic advice i don't.

think the department here is going to. appreciate this but i. it's the right thing to do i was like. can we just. link on that moment because. the importance of the way you thought. and the action you took. i feel is. is understated and. is rare and i would love to see so much. more of it because what happens as the. tools become more popular. uh there's a split that happens. and it's a truly heroic and impactful.

action to in those early in that early. split to step up. and you it's like great leaders. throughout history like get what is the. brave heart like get on a horse and row. the troops because i think that can have. make a big difference we have tensorflow. versus pytorch in the machine learning. we have the same problem today yes. i wonder it's actually bigger i wonder. if it's possible. to in the early days to rally the troops. it is possible especially in the early. days the longer it goes the harder right.

the more energy and the factions the. harder but in the early days it is. possible and it's extremely. helpful and there's a willingness there. yeah but but but the challenge is. there's usually not a willingness to. fund it yeah there's not a willingness. to you know like i was literally walking. into a field saying i'm going to do this. and here i am like you know i have five. kids at home now. pressure builds sometimes my wife hears. these stories and she's like you did. what. i thought we were gonna i thought you.

were actually on a path to make sure we. had resources and money but oh wow but. again there's a there's an aspect i'm a. very hopeful person i'm an optimistic. person by nature i love people i learned. that about myself later on uh. part of my. my religious beliefs actually lead to. that and it's why i hold them dear. because it's actually how i feel about. that's why it's what leads me to this. these attitudes sort of this hopefulness. and this sense of yeah it may not it may. not work out for me financially or maybe. but that's not the ultimate gain like.

that's a thing but it's not that's not. the score card. uh for me and so i just wanted to be. helpful and i knew and partly because. these sci-fi conferences because the. mailing-list conversations i knew there. was a lot of need for this. right so i had this it wasn't like i was. alone in terms of no feedback i had. these people who knew. but it was crazy like people at the time. said yeah we didn't think you'd be able. to do it yeah we thought it was crazy. and also instructive like practically. speaking. that you had a cool feature that you. were chasing the morphology like the yes.

like it's it it's not either the end. result it's not some visionary thing i'm. going to unite the community you were. like correct you were actually. practically this is what one person. actually could do. uh and actually build because that is. important because you can get over your. skis yeah. you can definitely get over your skis. and i had in fact this almost got me. over my skis right i would say well in. retrospect i hate looking back. we i can tell you all the flaws with. numpy right you want to go into it. there's lots of stuff that i'm like oh.

man that's embarrassing that was wrong i. wish i had somebody stop me with a wet. fish there yeah like i needed in fact. what i wished i'd had was somebody. with more experience and certainly. library writing an array library there's. like i wish i had me i could go back in. time and go do this do that there's. important being cause there's things we. did. that are still there that are. problematic that created challenges for. later and and i didn't know it at the. time didn't understand how important. that was and in many cases didn't know. what to do like there was pieces of the. design of nunpai i didn't know what to. do until five years ago.

now i know what they should have been. been but i didn't know at the time and. nobody and i couldn't get the help. anyway so i wrote it it took about it. took four months to write the first. version then about 14 months to make it. usable. but it was it wasn't it was that first. four months of intense writing coding. getting something out the door that. worked. that was it was it was definitely. challenging and then the big thing i did. was create a new type object called. d-type that was probably the. contribution and the fact that i added.

uh broad not just broadcasting but. advanced indexing so that you could do. um masks indexing and indirect indexing. instead of just slicing in so for people. who don't know maybe you can elaborate. yeah numpy i guess the vision in the. narrowest sense is to have this object. that represents. n-dimensional arrays. and like at any level of abstraction you. want but basically it could be a black. box that you can investigate in ways. that you would naturally want to.

investigate yes such objects yes exactly. so you could do math on it easily math. on it easily so you had so it had an. associated library of math operations. and effectively scipy became an even. larger upright set of. math operations so the key for me was. i was going to write numpy and then move. side pi to depend on numpy. in fact early on one of the initial. proposals was that we would just write. scipy and it would have the numeric. object inside of it and it would be. scipy.array or something. that turned out to be problematic.

because numeric already had a little. mini library of linear algebra and some. functions and it had enough momentum. enough users that nobody wanted to they. wanted backward compatibility one of the. big challenges of numpy was i had to be. backward compatible with both numeric. and memory in order to allow both of. those communities to come together there. was a ton of work in creating that. backward compatibility that also create. echoes in today's object like some of. the complexity in today's object is. actually from that. goal of backward compatibility these.

other communities which if you didn't. have that you'd do something different. which is instructive because a lot of. things are there. you know what is that there for it's. like well it was it was a remnant it's. an artifact of its historical existence. um by the way i love the empathy and the. lack of ego behind that because i feel. you see that in the split. in the javascript frameworks for example. the arbitrary branching right is. you i think in order to unite people you. have to kind of put your ego aside and.

truly listen to others like you do what. do you love about nama ray what do you. love about numeric like actually get a. sense we're talking about languages. earlier sort of empathize to the culture. of the people that love something about. this particular api. some. some the the naming style or the. the the use the actual usage patterns. and like truly understand them and so. that you can like. create that same draw in. the united i completely agree and you. have to also have enough passion that.

you'll do it it can't be just like a. perfunctory yeah oh yes i'll listen i'm. really i listen to you and then i'm not. really excited about it you so it really. is an aspect it's a it's a philosophical. like there's a philia there's a love of. esteeming of others it's actually at the. heart of. what. it's sort of a life philosophy for me. right that i'm constantly pursuing and. that helped absolutely helped it makes. me wonder in a philosophical like. looking at human civilization as one. object. it makes me wonder how we can copy and. paste travis's in the circle.

well in some aspects maybe some aspects. right right exactly well i. it's a good question how do we teach. this how do we how do we encourage it. how we lift it because so much of the. software world it's it's giant. communities right but it seems like so. much is moved by like little individuals. you talk about like linus tarwald. it's like can you could you have not. could you have had linux without him. could you it's like guido and python you. know. python i mean the sci-fi community.

particularly it's like i said we wanted. to build this big thing but ultimately. we didn't what happened is we had uh. mavericks and champions like john hunter. created matt plotlib we had fernando. perez who created ipython and so we sort. of inspired each other but and then it. kind of there's sort of a culture of of. this. selfless give the stewardship mentality. as opposed to ownership mentality with. stewardship and and. and community um focused. community focused but intentional work. like not not waiting for everybody else. to do the work but you're doing it for. the benefit of others and not you're not.

worried about what you're going to get. you know. you're not worried about the credit. you're not worried about we're going to. get you're worried about i later. realized that i have to worry a little. about credit not because i want the. credit because i want people to. understand what led to the results. like i don't it's not it's not about me. it's i want to understand this is what. led to the result so let's like i think. doing and this is what had no impact on. the result like let's let's promote this. just like you said i want to promote the. attributes so let that help make us. better off how do we make more of west. west mckinney like west mckinney was.

critical to the success of python. because of his creation of pandas which. is. the roots of that were all the way back. in uh. american number a and numpy where numpy. created an array of records. west started to use that almost like a. data frame except it's an array of. records and data frame the challenge is. okay if you want to augment it at. another column you have to insert you. have to do all this memory movement to. insert a column whereas data frames. became oh i'm going to have a loose. collection of arrays.

so it's a record of arrays that is the. heart of a data frame and we thought. about that back in the memory days but. wes. ended up doing the work to build it and. then then also the operations that were. relevant for data processing what i. noticed is just the each of these little. things creates just another tick another. up so numpy ultimately took a little. while about six months in people started. to join me you know francesc. alted robert kern charles harris. and these these people are many of the. unsung heroes i would say people who are.

you know they don't they sometimes don't. get the credit they deserve. because they were critical both to. support. like you know it's it's hard and you. want you need some support people need. support and i needed just encouragement. and they were helping encouraged by. contributing and and once the big thing. for me was when john hunter. he had previously done kind of a simple. thing called numerix to kind of you know. between the american memory he had a. little high level tool that would just. select each one from matplotlib in 2006. he finally said we're going to just make.

numpy the dependency of matplotlib. as soon as he did that and i remember. specifically when he did that i said. this okay we've done it like that was. when i knew we had succeeded success. and before then it was still you know to. ensure but that kind of started a roller. coaster and then 2006 to 2009. and then i've been floored by the by. what it's done like i had i knew it. would help i didn't have no idea how. much it would help. right so and it has to do with again the. language thing that just people started. to think in terms of numpy like yes.

and that opened up a whole. new way of thinking and part of the. story that would cut you kind of. mentioned but. maybe you can. elaborate is it seems like at some point. in this story. python took over science and data. science yeah and. uh not bigger than that. the scientific community. started to think like programmers or.

started to utilize the tools of. computers to do like at a scale that. wasn't done with fortran like at this. gigantic scale they started to opening. their heart and then python was the. thing i mean there's a few other. competitors i guess but python i think. really really took over i agree there's. a lot of stories here that are kind of. during this journey because this is sort. of the start of this journey in 2006. so. my tenure committee i applied for tenure. in 2006 2007. it came back i split the. department i was very polarizing i had.

some huge fans and then some people said. no way right so it was very i was a. polarizing figure in the department it. went all the way up to the university. president. ultimately my department chair had the. had this sway and they didn't say no. they said come back in two years and do. it again. and i went. at that point i was like i said i i i. mean i had this interest in. entrepreneurship this interest in in. not the academic circles not the like. how do how do we make industry work so i. do have to give credit to that expert.

that exploration of economics because. that led me. oh i had a lot of opinions i was i was. actually very libertarian at the time. and i'm still i have some libertarian. trends but i'm more of a i understand. i'm more of a collectivist libertarian. so you value broadly philosophically. freedom i value broadly philosophy. freedom but i also understand the. the power of communities like the power. of of collective behavior and and so. what's that balance right that makes. sense. um so by the time i was just i got to go.

out and explore this entrepreneur world. so i left academia i said no thanks. called my friend eric here who had his. company was going i said hey could i. join you and start this trend and and he. would at that time they were using. sci-fi a lot they were trying to get. clients and so i came down to texas and. in texas where i. sort of it's my entrepreneurial world. right i left academia and went to. entrepreneur world in in 2007. so moved. here in 2007 kind of took a leap knew. nothing really about business knew.

nothing about a lot of stuff there. there's you know for a long time i've. kept some connections to a lot of. academics because i still value it i. still love the. scientific tradition i still value the. the essence and the soul and the heart. of what is possible. don't like. a lot of the administration and the kind. of. we can go into detail about why and. where and how this happens what are. those challenges i mean i i don't know. but i'm with you so. well i'm still affiliated with mit.

i still love mit because there's magic. there yeah there's people i talk to like. researchers. faculty. in those conversations and the white. board and and just the conversation. that's magic there. all the other stuff the administration. all that kind of stuff. seems to. um. you don't you don't want to say too. harshly criticize sort of bureaucracies. but there's a lag that seems to get in. the way of the magic yeah and i don't.

i'm still have a lot of hope that. that can change because. i don't often see that particular type. of magic. elsewhere in industry so like we need. that and we need that flame going and um. it's the same thing as exactly as you. said it has the same kind of elements. like the open source community does. and but then if you like the reason i. stepped away the reason i'm here just. like you did in austin is like if i want.

to build one robot i'll stay at mit but. if i want to build millions. and make money enough to where i can. explore the magic of that then you can't. and i think. that dance is um that translational. dance has been lost a bit yeah right and. there's a lot of reasons for that i'm. certainly not an expert on this stuff. like an opinion like anybody else but i. i realized that i wanted to explore. entrepreneurship which i knew and really. figure out and it's been a driving. passion for 20 years 20 25 years how do.

we connect. capital markets and company um because. again i fell in love with the notion. that oh profit seeking on its own is not. a bad thing it's actually a coordination. mechanism for allocating resources that. you know not in an emergent way right. that respects. everybody's opinions right so this is. actually powerful so so. i i say all the time when i make a. company and we do something that makes. profit what we're saying is hey we're. collecting of the world's resources and. voluntarily people are asking us to do.

something they like. and that's a huge deal and so i really. like that energy so that's what i came. to do and to learn and to try to figure. out and that's what i've been kind of. stumbling through since for the past 14. years 2007 2007. so you were still. so no i was just emerging just right. one thing i've done i've done it's worth. mentioning because it. emphasized the exploratory nature of my. thinking at the time i said well i don't. know how to fund this thing i've got a. graduate student i'm paying for and i. got no funding for him and i had done. some fundraising from the public to try.

to get public fundraiser for my lab. i didn't really want to go out and just. do the fundraising circuit the way it's. traditionally done. so i wrote a book and i said i'm going. to write a book and i'm going to charge. for it it was called guide to numpy and. so ultimately numpy became documentation. driven development because i basically. wrote the book and made sure the stuff. worked to the book would work. so it really helped actually make numpy. become a thing so that doc writing that. book. and it was not a i mean it's not a page. turner i mean kind of not a book you. pick up and go oh this is great over the.

fire but it was it's where you could. find the details like how'd all this. work and a lot of people love that book. and so a lot of people end up so i but i. said look i i need to so i'm going to. charge for it. uh and i got some flack for that not. that much just just probably five angry. messages people you know yelling at me. saying i was you know. bad guy for for charging for this book. one of them richard. no just kidding no i i haven't really. had any interaction with him personally. uh like i said um but but there were a. few but but actually surprisingly not.

there was actually a lot of people like. no it's fine you know you can charge for. a book that's no big deal we know that's. a way you can you can try to make money. around open source so so what i did what. i i did in an interesting way i said. well. you know kind of my ideas around around. ip law and stuff i love the idea you can. share something you can spread it like. once it's the fact that you have a thing. and copying is free. but the creation is not free so how do. we how do you fund the creation and. allow the copying right and then. software it's a little more complicated. than that because creation is actually a. continuous thing you know it's not like. you build a widget that's done it's sort.

of a process of emerging and continuing. to to create but i wrote the book and. had this market determined price thing i. said look i need i think i said 250 000. if i make 250 000 from this book it's. it'll make it free so as soon as i get. that much money or i said five years. right so there's a time limit like. that's forever cool i didn't know the. story yeah. i i released it on this and it's. actually interesting because one of the. people who also thought that was. interesting ended up being chris white. who was the director of darpa project.

that we got funding through at anaconda. and the reason he even called us back is. because he remembered my name from this. book and he thought that was interesting. and so. even though we hadn't gone to the demo. days we applied and the people said yeah. nobody ever gets this without coming to. the demo day first that's the first time. i've seen it but it's because. i knew you know chris had done this and. had this interaction so it did have. impact i was actually really really. pleased by the result i mean i ended up. i ended up in three years i mean 90 000. so i sold 30 000 copies by myself i just. put it up on you know use paypal and.

sold it uh made and those are my first. taste of kind of. okay this can work to some degree and. and i you know all over the world right. from germany to japan to it was actually. it did work and so i appreciated the. fact that paypal existed and had a way. to make to get the money the. distribution was simple. this is pre-amazon book stuff so it was. just published in a website it was the. popularity of sci-fi emerging and. getting company usage. i ended up not letting it go the five.

years and not trying to make the full. amount because. you know a year and a half later i was. at m thought i had left academias at m. thought and i kind of had a full-time. job and then actually what happened is. the documentation people there's a group. that said hey we wanna do documentation. for scipy as a collective and they're. essentially. needing the stuff in the book right and. and so. they kind of ask hey can we just use the. stuff in your book and at that point. said yeah i'll just open it up so that's. but it has served his purpose in the. money that i made actually funded my. grad student like it was actually you.

know i paid him 25 000 a year. uh out of that money the funny thing is. if you do very similar kind of. experiment now with numpy or something. like it you could probably make a lot. more it's probably true. because of the tooling and the community. building yeah i agree like the and. social media there there's just a. virality to that kind of idea i agree. there'd be things to do i've thought. about that but and really had thought. about a couple of books or a couple of. things that could be done there and i. just. haven't right even. i tried to hire a ghostwriter this week.

this year too to speak effect would help. but it it didn't. part of my problem is this i've been so. excited by a number of things that steps. in from that like so i came here worked. at nthot for four years uh graciously. you know eric made me president and we. started to work closely together we. actually helped him buy out his partner. um. it didn't end great like unfortunately. eric and i aren't real aren't friends. now um i still respect him i have a lot. i mean i wish we were but uh. he didn't like the fact that i that. peter and i started anaconda right that.

was not i mean. um so i'm there's two sides of that. story so i'm not gonna go into it right. sure um but you. as human beings and you wish you still. could be friends i do. i do it saddens me i mean that that's um. that's the story of great minds building. great companies yeah somehow it's sad. that um yeah when there's that kind of. and. i i i hold him in a steam i'm grateful. for him i think he's they're doing you. know their thoughts still exist they're.

doing great work uh helping scientists. they still run the scipy conference. they're in they have an r d platform. they're selling now that's a a tool that. you can go get today right so. um they've been. thought has played a role in the scipy. in in supporting the community around. sci-fi i would say they ended up not. being able to. they ended up building a tool suite to. write gui applications like that's where. they could actually make that the. business could work and so this. supporting scipy and numpy itself wasn't. as possible like they didn't they tried.

i mean it was not just because it was. just because the business aspect so and. i wanted to build a company that could. do that could get venture funding. right better for worse i mean that's a. longer story we could talk a lot about. that but and that's that's where. anaconda came that's renaconic in it so. let me let me ask you it's it's a little. bit for fun because you built this. amazing thing and so let's. let's talk about uh like an old warrior. looking over old battles. um you've you know there's a sad.

letter in 2012 that you wrote uh to the. numpy mailing list announcing that. you're leaving numpy yeah and what some. of the things you've listed. some some of the things you regret or. not regret necessarily but some things. to think about. if you could go back and you could fix. stuff about numpy or. both sort of in a personal level but. also like looking forward what kind of. things would you like to see changed. good question so i think there's. technical questions and social questions. right there um. first of all you know i wrote numpy as a.

service. and i spent a lot of time doing it and. then other people came help make it. happen numpy succeeded because the work. of a lot of people right so it's. important to understand that. i'm grateful for the opportunity the. role i had i could play and grateful. that things i did had an impact. but they only had the impact they had. because the other people that came to. the story and so they were essential but. the way data types were handled the way. data types we had array scalers for. example. that that are really just an um a.

substitute for a type concept. right so we had. array scalers are actual python objects. so that there's for every for a 32-bit. float or a 16-bit float or a 16-bit. integer. python doesn't have a natural it's just. as one integer there's one float well. what about these. lower precision types these larger. precision types. so we had them in numpy. so that you could have a collection of. them but then have an object in python. that was one of them. and there's questions about like in.

retrospect i wouldn't have created those. of an improved the type system. and like made the type system actually a. python type system as opposed to. currently it's a python one level type. system i don't know if you know the. difference between python one python two. it's kind of technical kind of depth but. python two one of its big things that. guido did it was really brilliant it was. he actually. python one. all classes new objects were where one. if you as a user wrote a class it was an. instance of a single python type called. the ob called the class type.

right in python 2. he used a meta typing hook to actually. go oh we can extend this and have users. write classes that are new types. so it's able to have your user classes. be actual types and the python type. system got a lot more rich. i barely understood that at the time. that numpy was written and so i. essentially in python numpy created a. type system that was python one era. it was every. every d type is an instance of the same. type as opposed to having new d types be.

really just python types with additional. metadata what's the cost of that is it. efficiency is a usability uh it's. usability primarily the cost isn't. really efficiency it's it's it's the. fact that it's clumsy to create new. types. uh it's hard and then one of the. challenges you want to create new types. you want a quaternion type or you want. to uh add a new you know posit type or. you want to um so it's hard now in the. and and now. if we had done that well. when number came on the scene where we.

could actually compile python code it. would integrate with that type system. much cleaner and now all of a sudden you. could. do gradual typing more easily you could. actually have python when you add number. plus better typing could actually be a. uh. you'd smooth out a lot of rough edges. but there's already there's like but are. you talking about from the perspective. of developers within numpy or users and. not buy because developers have new not. really users of numpy so much it's the. development of numpy so you're thinking. about like.

how to design numpy so that it's. contributors yeah the contributors are. it's easier it's easier it's less work. to make it better and to keep it. maintained and and where that's impacted. things for example is. the gpu like all of a sudden gpus start. getting added and. we don't have them in numpy like numpy. should just work on gpus right the fact. that we have to have to download a whole. other object called kupai to have arrays. on gpus is just an artifact of. history because there's no fundamental. reason for it well that's really. interesting if we could sort of go on.

that tangent briefly is. you have. pi torch and other. library like tensorflow that basically. tried to mimic uh yeah. like you've created a sort of platonic. form. basically. yeah exactly well the problem was they. didn't realize that yeah the platonic. form has a lot of edges they're like. well we should cut those out before we. present it so i i wonder if you can. comment is there like a difference. between their implementations do you. wish that they were all using numpy over.

like in this abstraction yeah. and sorry to interrupt that there's. gpus a6 there might be other. neuromorphic computing there might be. other kind of or the aliens will come. with a new kind of computer like an. abstraction that numpy should just. operate nicely over the things that are. more and more and smarter and smarter. with uh with this multi-dimensional. arrays yeah yeah i have there's several. comments there we are working on. something now called. datadashapis.org data.api.org you can go.

there today and it's. it's our answer it's my answer you know. it's not just me it's me and ralph and. and athen and aaron and a lot of. companies are helping us at quansite. labs. uh it's not unifying all the arrays it's. creating an api that is unified um so. we do care about this and trying to try. to work through it. actually the chance to go and meet with. the tensorflow team and the pi torch. team and talk to them after uh after. exiting anaconda just talking about. because.

the first year after leaving a con in. i became deeply aware of this and. realized that oh this split in the array. community that exists today. makes what i was concerned about in 2005. pretty parochial. it's a lot worse right now there's a lot. more people. so the perhaps the industry can sustain. more stacks right there's a lot of money. but it makes it a lot less efficient i. mean this. but i've also learned to appreciate it's. okay to have some competition it's okay. to have different implementations.

but it's better if you can at least. refactor some parts i mean you're going. to be more efficient if you can refactor. parts it's uh it's nice to have. competition over things. overweight they're innovative. competition right they're innovative. yeah innovative and then maybe on the. infrastructure right uh whatever however. you define infrastructure right. maybe it's nice to have controversial. exactly i agree and i think but it was. interesting to hear the stories i mean. tensorflow came out of. the c-plus plus library uh jeff dean. wrote i think that was uh basically uh.

how they were doing inference right and. then they realized oh we could do this. tensorflow thing. that close library then what was. interesting to me was the fact that both. google and facebook. did not it's not like they supported. python or numpy initially they just. realized they had to they they came to. this world and then all the users like. hey where's the numpy interface. oh and then they kind of came late to it. and then they had these bolt-ons. tensorflow's bolt on. i don't mean to offend but it was so bad. yeah it's the first time that i i i'm.

usually so i mean. one of the challenges i have is i don't. criticize enough because. in the sense that i don't give people. input enough you know if um i think it's. universally agreed upon that the. bolt-ons on tensorflow right but i went. through it there was a talk given at a. mallorca in in spain and it got a great. guy i came and gave a talk i said you. should never show that api again at a pi. data conference. like that. that's terrible like you're taking this. beautiful system you've created and like. you're corrupting all these poor python. people forcing them to write code like.

that or thinking they should. uh fortunately you know they adopted. keras as their and that's keros is. better and so keras tensorflow is fine. is reasonable but um they bolted it on. facebook did too like facebook had their. own. c plus library for doing inference and. they also had the same you know reaction. they had to do this one big difference. is facebook maybe because the way it's. situated in the in part of fair part of. the research library tensorflow is. definitely used and you know they have. to make they couldn't just open it up.

and let the community you know change. what that is because. i guess they were worried about. disrupting their operations. facebook's been much more open to having. community input on the structure itself. whereas google and tensorflow they're. really eager to have user community. users people use it and build the. infrastructure but it's much more wild. like it's harder to become a contributor. to tensorflow and it's also this is very. difficult question to answer and i don't. mean to be throwing shade at anybody but. you have to wonder it's the microsoft. question.

of when you have a tool like pi torch or. tensorflow. how much are you tending to the hackers. and how much are you tending to the big. corporate clients correct and so correct. like the ones that so do you tend to the. millions of people that are giving you. almost no money. or do you tend to the peop the few that. are giving you a ton of money i tend to. um. stand with the people right because i. feel like if you uh nurture the hackers. you will make the right decisions in the. long term that will make the companies.

happy i lean that way too. totally but then you have to find the. right date but it's a balance yeah it's. because you can lean to the hackers and. run out of money yeah exactly. exactly. which has been some of the challenge. i've faced yes in the sense that like i. like i would look at some of the. experiments like numpy the fact that we. have the split is a factor of i wasn't. able to collect more money towards. number development yeah right i mean i. didn't succeed at in the early days of. getting enough financial contribution. and umpi so maybe i could work on it.

right i couldn't work on it full-time i. had to just. catch an hour here an hour there. and i've basically not liked that like. i've wanted to be able to do something. about that for a long time and trying to. figure out how well there's lots of ways. i mean possibly one could say you know. we had an offer from microsoft at early. days of anaconda uh the 2014 they. offered to come by us right. the problem was the right people that. microsoft didn't offer to buy us and. they were still. they were it was really uh we were like. a second. they had really bought they just bought. r the r company called um.

it was not our studio but it was another. r company that was emergent and it was. kind of a well we should also get a. python play but they were really. doubling down on r. right and so it was like it was where. you would go to die so it's not it. wasn't it was before satya was there. satya had just started just started. right and if the and the offer was. coming from someone two levels down from. him gotcha right and if it come from. scott guthrie so i got a chance to meet. scott guthrie great guy i like him if it.

offered to come from him. probably would be at microsoft right now. that'd be fascinating that would be. really nice actually especially given uh. what microsoft has since done for the. open source community yes i think. they're doing well i really like some of. the stuff they've been doing they're. still working and they've you know. they've hired guido now and they've. hired a lot of python developers. he retired then he came out of. retirement and he's working out i was. just talking to him and he didn't. mention this person well. i should i should have been further.

because i know he loved dropbox but i. wasn't sure what he was doing who he was. up to well he was kind of saying he'd. retire but uh and it's it's literally. been. five years since i last sat down and. really talked to guido right um. guido is a technology uh expert right. he's a so i i came i was excited because. i'd finally figure out the type system. for numbai i wanted to kind of talk. about that with him and i kind of. overwhelmed him could you stay in that. mo just for a brief moment because. you're a fascinating person in the. history of program and he is a. fascinating person what have you learned.

from guido about. programming about life yeah yeah uh a. lot actually i've been a fan of guidos. you know we have a chance to talk some i. wouldn't say you know we talk all the. time not only at all he may um but we. talked enough to i respect his back when. i first started number one the first. things i did was i had a i asked guido. for a meeting with him and paul dubois. in san mateo and i went and met him for. lunch and basically to say maybe we can. actually part of the strategy for numpy.

was to get it into python 3 and maybe be. part of python so we talked about that. that's cool about that approach right i. would have loved to be a fly in the. world that was a that was good and over. over the years for guido i learned. so he was open like he was willing to. listen to people's ideas. right and know over the years now. generally you know i'm not saying. universally that's been true but but. generally that's been true so he's. willing to listen he's willing to defer. like on the scientific side he would. just kind of defer he didn't really. always understand what we were doing.

like and he'd defer one place where he. didn't enough. was we missed a matrix multiply operator. like that finally got added to python. but about 10 years later than it should. have. but the reason was because nobody. it took it takes a lot of effort and i. learned this while i was writing numpy i. also wrote tools to give a python dev. and i added some pieces of python um. like the memory view object i wanted the. structure of numpy into python so we. didn't get numpy into python but we got. the basic structure of it in the python. like so you could build on it nobody did.

for a while but eventually database. authors started to. and it was it's a lot better they did. and also antoine petro and stefan craw. actually fixed the memory view object. because i wrote the underlying. infrastructure in c but the python. exposure was terrible until they came in. and fixed it partly because i was. writing numpy and numpy was the python. exposure i didn't really care about if. you didn't have numpy installed anyway. guido. opened up ideas technology you know. brilliant like really. i really got a lot of respect for him. when i saw what he did with the clap.

with the this type class merger thing. that was actually tricky right and then. and then willing to share willing to. share his ideas so the other thing early. on in 1998 i said i start wrote my first. extension module the reason i could is. because he wrote this blog post on how. to do reference counting. right and without it i would have been. lost right but he was he was willing to. at least try to write this post and so. he's been motive he's been motivated. early on with python it was a computer. science for everybody we kind of have. this early on desire to oh maybe we.

should be pushing programming to more. people so he had this. populist notion i guess or populist. sense. um so learn that there's a certain skill. i've seen it in other people too. of. engaging with contributors sufficiently. to because when somebody engages with. you and wants to contribute to you if. you ignore them they go away so building. that early contributor base requires. real engagement. with other people and he would do that. can you also comment on this tragic. uh stepping down from his position as.

the benevolent dictator for life. over the wars. uh uh you know the walrus operator the. walrus operator was the bat last battle. i don't know if that's the cause of it. but uh this there's this for people who. don't know you can look up there's the. walrus operator which is uh looks like a. colon and an equal sign yeah and equal. sign and. it actually does. maybe the thing that you. that an equal sign should be doing yeah. maybe right exactly uh yeah but it's.

just historically equal sign means. something else it just means assignment. so he stepped down over this what do you. think about the pressure of leadership. some of the you mentioned the letter i. wrote in umpire at the time that was a. hard time actually i mean you know. there's been really hard times it was. hard you know. you get criticized right and you get. pushed and you get um not everybody. loves what you do like anytime you do. anything that has impact at all. you're not universally loved right you. get some real critics and.

that's an important energy because. it's impossible if you do everything. right you need people to be pushing but. sometimes people can get mean yeah. people can. i. i prefer to get people to benefit the. doubt i don't immediately assume they. have bad intentions. and maybe for other you know maybe other. maybe that doesn't happen for everybody. they for whatever reason their past. their experience with people they they. sometimes have bad and they so they. immediately attribute to you bad. intentions so you're like where this. come from i mean i definitely open the. criticism but i think you're. misinterpreting the whole point.

uh because i i would get that you know. certainly when i started anaconda you. know. i've been. sometimes i say to people uh i know i'm. i care enough about entrepreneurship to. make some open source people. uncomfortable. and i care enough about open source to. make investors uncomfortable. so i sort of you know create you create. kind of doubters on both sides so when. you have and this is just a. a plea to the listener and the public. i've noticed this too. that. there's a tendency and social media.

makes this worse. when you don't have perfect information. about the situation you tend to fill the. gaps with the. the worst possible or at least a bad. uh story that fills those gaps and. i think it's good to live life. uh maybe not fully naively but filling. in the gaps with the with the with the. good. with the. best with the positive with the with the. hopeful explanation of why you see this. so if you see somebody like you trying. to make money on a book about numpy.

there's a million stories around that. that are positive and those are good to. think about. to project positive intent on to people. because. for many reasons usually because people. are good and they do have good intent. and also when you project that positive. intent people step up to that. too yes so like it's it has a great. point it has this kind of viral nature. to it and of course. what twitter early on figured out on. facebook is that they can make a lot of. money and engagement from the negative.

yes so like there's this we're fighting. this mechanism i agree it's just. challenging it's like easier it's just. easier to be to be negative and then for. some reason something in our minds. really enjoys sharing that and getting. getting all excited about the negativity. we do yeah but but the protective. mechanism perhaps that we're we're going. to eat and if we don't exactly for us to. be effective as a group of people in a. software engineering project you have to. project positive intent i think i. totally agree totally agree and i think. that's very so that that happens in this.

in the space but python has done a. reasonable job in the past but here's a. situation where i think it's it's. starting to get this pressure where it. didn't i was i really didn't i didn't. know enough about what happened i've you. know talked to several people about it. and i know. i think most of the steering committee. members today. uh one one person nominated me for that. role but it's the wrong role for me. right now right. um i have a lot of respect for the. python developer space and the python. developers i also understand the gap. between computer science python. developers and array programming. developers or science developers in fact.

python succeeds in the array space the. more it has people in that boundary and. there's often very few like i was. playing a role in that boundary and you. know working like everything to try to. keep up with the. with the what even what guido was saying. like. i'm a c programmer but not a computer. scientist like i was a engineer and. physicist and mathematician. and i don't i didn't always understand. what they were talking about and why. they would have opinions the way they. did so you have to listen and try to. understand then you also have to explain.

your point of view in a way they can. understand and that takes a lot of work. and that that communication. is always the challenge and it's just. what we're describing here about the. negativity is just another form of that. like how do we come together and it does. appear we're wired anyway to at least. have a. there's a part of us that will enemy you. know friend enemy and. and we see yeah it's like why are we. wiring on the enemy front yeah so so why. are we pushing that why are we promoting. that so deeply let's assume friend until. proven otherwise yes yeah. so because you have such a fascinating.

mind and all this let me just ask you. these questions so one interesting side. on the python. history is the move from python 2 to. python 3. you mentioned move from python. 1 to python 2 but. the move from python 2 to python 3 is a. little bit interesting because it took a. very long time it uh it broke. in a quite a small way backward. compatibility but even that small way. seemed to have been very painful for. people is there lessons.

tons of lessons from uh from how long it. took and how painful it seemed to be. yeah tons of lessons well i mentioned. here earlier that. numpy was written in 2005. it was in 2005 that i actually went to. guido to talk about getting numpy into. python 3. like my strategy was to. oh we're moving to python 3. let's have. that be and it seems funny in retrospect. because like wait python 3 that was in. 20 2020 right when we finally. ended support for python 2 or at least. 2017. the reason it took a long time a.

lot of time i think it was because. one of the things is there wasn't much. to like about python 3. 3.0 3.1 it. really wasn't until 3.3 like i consider. python 3.3 to be python 3.0. but it wasn't until python 3.3 that i. thought there's enough stuff in it. to make it worth anybody using it. right and then three four started to be. oh yeah i want that and then three five. as the matrix multiply operator and now. it's like okay we gotta use that plus. the libraries that started leveraging. the some of the features of python. exactly yeah so it really the challenge.

was it was. but it also illustrated a truism that. you know it's when you have inertia when. you have a pop when you have a group of. people using something it's really hard. to move them away from it you can't just. change the world on them and python 3. you know made some i think it fixed some. things guido had always hated i don't. think he didn't like the fact print was. a statement he wanted to make it a. function but in some sense there's a bit. of gratuitous change to the language and. you could argue and there's people have. but. there was. one of the challenges was there wasn't.

enough features and too many just. changes. without features and. so that empathy for the end user as to. why they would switch wasn't wasn't. there i think also it illustrated just. the funding realities like python wasn't. funded like it was also a project with a. bunch of volunteer labor. right it had more people so more. volunteer labor but it was still it was. fun in the sense that least guido had a. job and i i've learned some of the. behind the scenes on that now since. since talking to people who lived. through it and. uh maybe not on air we can talk about.

something. but it's interesting to see but guido. had a job but he but his full-time job. wasn't just work on python yeah like he. had other things to do. it's just wild it is wild isn't it as. well how few people are funded yes how. much impact they have yes. maybe that's a feature not a bug i don't. know maybe yes exactly at least early on. like it's sort of i know yeah it's like. olympic athletes are often severely. underfunded but maybe that's what brings. out the greatness perhaps yes correct no. exactly. maybe this is.

essential part of it because i do think. about that in terms of i currently have. an incubator for open source startups. like what i'm trying to do right now is. create the environment i wish that. existed when i was leaving academia with. numpy and trying to figure out what to. do i'm trying to create those. opportunities and environments so. uh and that's that's what drives me. still is how do i make the world easier. for the open source entrepreneur uh so. let me stay. i mean i could probably stand numb by. for a long time but um this is fun. question so andre kapathy leads the.

tesla autopilot team and uh he's also. one of the most like legit. uh programmers uh i know. it's like he builds stuff from scratch a. lot and that's how he builds intuition. about how a problem works he just built. it from scratch and i always love that. and the primary language he uses is. python. for for the intuition building but he. posted something on. twitter. saying that they got a significant. improvement.

on some aspect of their uh like data. loading i think. by switching away from np dot square. root. so the numpy's implementation of square. root to math that square root and then. somebody else commented that you can you. can get even a much greater improvement. by using the vanilla. python square root which is like power. 0.5 power 0.5. and it's fascinating to me i just wanted. to. so that absolutely i mean that was some.

shade throwing at some no no. but also we're talking about it's a good. way to ask. the trade-off between usability and. efficiency. broadly in numpy but also on these like. specific weird quirks of like a single. function yep so. on that point. if you use a numpy math function on a. scalar. it's going to be slower than using a. python function on that scalar yeah but. because the the math object in p in.

numpy is more complicated right because. you can also call that math object on an. array. and so effectively it goes through a. similar machine there aren't enough of. the which you would do in a and you. could do like checks and. fast paths. so yeah if you're basically doing a list. if you run over a list in fact for. problems that are less than. a thousand. even maybe 10 000 it's probably the if. you're going more than 10 000 that's. where you definitely need to be using. arrays but if you're less than that and.

for reading if you're doing a reading. process and essentially it's not compute. bound it's i o bound and so you're. you're really taking lists of thousand. at a time and doing work on it yeah you. could be faster just using python. straight up python see but also. and then this is the so sorry to. introduce there's the fundamental. questions. when you look at the long arc of history. it's very possible that np. square is much faster it could be so. like in terms of like don't worry about. it it's the the evils of over.

optimization or whatever all the. different quotes are on that it's is uh. sometimes obsessing about this. particular little uh quark is not it's. not. it's efficient like for somebody like uh. if you're if you're trying to optimize. your path i mean i agree premature. optimization. creates all kinds of challenges right. because now but you may have to do it i. believe the quote is it's the root of. all it's root of all evil right. let's give dude i think or take that to. somebody else. well doc newt is kind of like mark twain.

people just attribute it don't matter. and it's fine because brilliant so no i. was a tech user myself and so i have a. lot of respect and he did more than that. of course but. uh yeah. someone i really appreciate in the. computer science space yeah i don't i. think that's appropriate there's a lot. of little things like that where people. actually if you understood it you go. yeah of course that's the case yeah like. and the other part and the other part i. didn't mention and. number was a thing we wrote early on and. i was really excited by number because. it's something we wanted it was a.

compiler for python syntax and i wanted. it from the beginning of writing numpy. because. of this function question like. taking the power of arrays is really. that you can write functions using all. of it. it has implicit looping right so you. don't worry about this n-dimensional for. loop with you know four loops four four. statements you just say oh big. four-dimensional array i'm gonna do this. operation this plus this minus this. reduction. and you get this it's called. vectorization in other areas but you can. basically think at a high level and get.

massive amounts of computation done with. the added benefit. of oh it can be paralyzed easily it can. be put in parallel you don't have to. think about that in fact it's worse to. go decompose your you write the for. loops and then try to infer parallelism. from for loops that's actually harder. problem than to take the array problem. and just automatically parallelize that. problem that's what. and and so functions in numpy are called. universal functions u func so square. root is an example of a u func there are. others sine cosine add subtract.

in fact one of those first libraries to. scipy was something called special where. i added bessel functions and like all. these special functions that come up in. physics and i added them as u func so. they could work on arrays so i. understood you function very very well. from day one inside of numeric that was. one of the things we tried to make. better in numpy was how do they work can. they do broadcasting what does. broadcasting mean. but one of the problems is okay what do. i do with a python scalar. so what happens the python scalar gets. broadcast to a zero dimensional array.

and then it goes through the whole same. machinery as if it were a ten thousand. dimensional array and then that then. then it kind of unpacks the element and. then does the addition. that's not to mention the function it. calls. in the case of square root is just the c. lab square root right in some cases like. python's power there's some. optimizations they're doing. for that are that could be faster than. just calling this the c lab square root. in the interpreter or the in the no in. the c code in the python runtime in the. pythagorean so they're. they really optimize it and they have.

the freedom to do that because they. don't have to worry about it's just a. scalar it's just a scalar right they. don't have to worry about the fact that. oh this could be an object with many you. know many pieces they're not the u funk. machinery is also generic in the sense. that uh type casting and broadcasting. broadcasting's idea of i'm gonna go i. have a zero dimensional array i have a. scalar with a four dimensional array and. i add them. oh i have to just kind of concourse the. shape of this guy. to make it work against the whole. four-dimensional array so it's the idea. if i can do a one-dimensional array. against a two-dimensional array and have.

it make sense well that's what numpy. does is it challenges you to reformulate. rethink your problem yes as a. multi-dimensional rate problem versus. like. move away from scalars completely right. exactly yeah exactly in fact that's. where some of the edge cases boundaries. are is that well the they're still there. and this is where array scalers are. particular so arrays are particularly. bad in the sense that they were written. so that you could optimize the math on. them but that hasn't happened. right and so their default is to use is.

to coerce the arrays together to a zero. dimensional array and then use the. number the numpy machinery that's what. and you could specialize but it doesn't. happen all the time so in fact when we. first wrote number we'd do comparisons. and say look it's a thousand x speed up. we're lying a little bit in the sense. that well first do with the. the 40x slowdown of using array scalers. inside of a loop because if you used to. use python scalars you'd already be. 10 times faster yeah but then we would. get 100 times faster over that using. just compilation.

but what we do is compile the loop from. out of the interpreter to machine code. and then that's always been the power of. python is this extensibility so you can. people say oh python's so slow well sure. if you do all your logic in the runtime. of the python interpreter yeah but the. power is that you don't have to you. write all the logic which you do in the. high level is just high level logic and. the the actual calls you're making could. be on. gigabyte arrays of data and that's all. done at compiled speeds. and the fact that integration is. one can happen but two is separable.

that's one of the uh their language like. julia says we're gonna be all in one you. can do all of it together and then. there's the jury's out is that possible. i tend to think that you're gonna. there's separate concerns there you want. to pre-compile them but generally you. will want to pre-compile your. some of your loops like scipy is a. compilation step to install sci-fi it. takes about two hours if you have many. machines maybe you can get it down to. one hour but to compile those libraries. takes about takes a while you don't want. to do that at runtime. you don't do that all the time you want.

to have this precompiled binary. available that you're then just linking. into so there's real questions about. the whole you know source code. code is running binary code is more than. source code it's created object code. it's the linker it's the loader it's the. how does that interpret it inside the. virtual memory space there's a lot of. details there that actually i didn't. understand for a long time until i you. know read books on the topic and it led. to. the more you know the better off you are. and you can do more details but. sometimes it helps with abstractions too.

well the problem as we mentioned earlier. with abstractions is you kind of. sometimes assume. that whoever implemented this thing. had your case in mind and found the. optimal solution yes or like you assume. certain things i mean there's a lot of. correct one of the really powerful. things to me. early on. i mean it sounds silly to say but with. python probably one of the reasons i. fell in love with it is dictionaries yes.

um so. obviously probably most languages have. some. concepts some mapping concept but it. felt like it was a first class citizen. and it was just my brain was able to. think in dictionaries but then there's. the thing that i guess i still use to. this day is order dictionaries. because that seems like a more natural. way to construct dictionaries yeah and. and from a computer science perspective. the running time cost is not that. significant but there's a lot of things. to understand about dictionaries.

that. the abstraction kind of. doesn't necessarily incentivize you to. understand. right do you really understand the. notion of a hashmap and how that. dictionary is implemented but you're. right dictionaries are a good example of. an abstraction that's powerful and i. agree with you one of the love i agree i. love dictionaries too it took me a while. to understand that once you do you. realize oh they're everywhere and python. uses them everywhere too like it's. actually constructed that one of the. foundational things is dictionaries and. it does everything with x-rays yeah so. it is it's powerful order dictionaries. came later but it is very very powerful.

it took me a little while coming from. just the array programming entirely to. understand these other objects like. dictionaries and lists and. tuples and binary trees. like i said i wasn't a computer. scientist so i studied a raise first and. so i was very erase-centric and you. realize oh these others don't have. purposes and value actually. um i agree there's a friendliness about. like one way to think about a raise. is um. arrays are just. not like full of numbers.

but to make them accessible to humans. and make them less error-prone to human. users sometimes you want to attach names. human interpretable names. that are sticky to those arrays so yeah. that's how you start to think about. dictionaries yes. you start to convert numbers into. something that's human interpretable and. that's actually the tension i've had. correct with numpy because correct i've. built so much tooling around. human uh.

human interpretability and also. protecting me. from a year later not making the. mistakes by being i wanted to force. myself to use english versus uh. numbers. yes so there's a there's a project. called labeled arrays like very. early it was recognized that oh we need. we we're indexing numpy we're just. numbers all the columns and particularly. the dimensions i mean if you have an. image you don't necessarily need to. label each column or row but if you have. a lot of images so you have another. dimension you at least like to label the.

dimension as this is x this is y z or. this is give us some human meaning or. some domain circle meaning. that was one of the impetuses for pandas. actually was just oh we do need to label. these things. and label label array was an attempt to. add that like a lighter weight version. of that and there's been like that's an. example of something i think numpy could. add. could be added to numpy. but one of the challenges again how do. you fund this like like i said one of. the tragedies i think is that so i i. never had the chance to i was never paid.

to work on empire right so i've always. just done in my spare time always taken. from one thing taken from another thing. to do it. and at the time i mean today it would be. the wrong time today like pay me to work. on empire now would not be a good use of. effort but. but we are finally at quansite labs i'm. actually paying people to work on numpy. and scipy which is i'm thrilled with i'm. excited by uh i've wanted to do that. it's what i wanted to do from day one it. just took me a while. to figure out a mechanism to do that. even like in the university setting. respecting that from like.

pushing students young minds the younger. graduate students to contribute. and then figuring out financial uh. mechanisms that enable them to. contribute and then sort of reward them. for their um. innovative scientific journey that that. would be nice but then also just the. better allocation of resources. well you know it's 20-year anniversary. since 9 11 and i was just looking we. spent over six trillion dollars in in. the middle east after 9 11.

in the various efforts there and sort of. to. put politics and all that aside it's. just you think about the education. system all the other ways we could have. possibly allocated that money. to me yeah to take it back. the amount of impact you would have by. allocating a little bit of money to uh. the programmers yeah that build the. tools that run the world is fascinating.

i mean it is i. it i don't know. i think uh again there is some aspect to. uh being broke. as somewhat of a feature not a bug that. you make sure that you manage that right. now i know i it's so i but i don't think. that's a big part so it's like i think. you can you can have enough money and. actually be wealthy while maintaining. your values agreed i think agreed. there's an old adage that you know. nations that trade together don't go to. war together yeah right i i've often. thought about you know nations that code.

together. [Laughter]. one thing i love about open source is. it's global it's multinational like. there aren't national boundaries one of. the challenges with business and open. source is the fact that business is. national like businesses are entities. that are recognized in legal. jurisdictions right and have laws that. are respected in those jurisdictions and. hiring and yet the open source ecosystem. is not it's not it's not there like. currently one of the problems we're. solving is hiring people all over the. world. right because we. it's a global effort and i've had the. chance to work and i've loved the chance.

i've never been to uh like a iran but i. once had a conference i was able to talk. to people there right and talk to folks. in uh pakistan never been there but we. had a a. call where there are people there like. just scientists and normal people and. you know and and it's there's a there's. a certain amount of. humanizing right that gets away from the. like we often get the memes of society. that bubble up and get her get discussed. but the memes are not even an accurate.

reflection of the reality of what people. are. well if you look at the major power. centers that are leading to something. like cyber war in the next few decades. it's united states it's russia and china. right and th those three countries in. particular have incredible developers. so if they work together yeah i think. that's one way the politicians can do. their stupid bickering but like the. there's a layer of infrastructure of. humanity yeah if they collaborate. together that that i think can prevent.

major uh major. military conflict which would i think. most likely happen at the cyber level. versus the actual hot war level you're. right no i think that's good that's good. prediction. nations that code together uh. don't go to war yeah they don't go to. war together that's that's a hope right. that's one of the philosophical hopes. but yeah. so you mentioned uh the project of. number. which is um. fascinating so from the early days there. was kind of a push back on python that.

it's not fast. you know you see c if you want to write. something that's fast you use cc plus if. you want to write something that's. usable and friendly but slow you use. python. and so. what is uh number what is its goal how. does it work great yeah yes that's what. the argument and the reality was people. would write. high-level code and use compiled code. but there's still user story use cases. where you want to. write python but then have it still be. fast you still need to write a for loop.

like before number it was always don't. write a for loop you know write it in a. vectorized way you put in an array. and often that'll that can make a memory. trade off like quite often you can do it. but then you make maybe use more memory. because you have to build this array of. data that you don't necessarily need all. the time. so. number was it started from a desire to. have. a kind of a vectorized that worked. vectorized was a was a tool in numpy it. was released you give it a python. function and it gave you a universal. function a u-function would work on.

arrays so get the function that just. worked on a scalar like you could make a. like the classic case was a simple. function and if then statement in it so. uh sine x over x function sync function. if x equals zero return one otherwise do. sine x over x. the challenge is you don't want that. loop had one in python so you want a. compiled version of that. um but the ufo the vectorize and numpy. would just give you a python function so. it would take the array of numbers and. at every call do a loop back into python. so it was very slow it gave me the.

appearance of a u func but it was very. slow so i always wanted a vectorize that. would take that python scalar function. and produce a u-func working on binary. native code so in fact i had somebody. work on that with pi pi to see if. pipette could be used to produce a u. func like that early on um in 2009 or. something like that 2010. um they didn't. work that well it was kind of pretty. bulky but in 2012. uh peter and i just started anaconda we. had i just i. i'd learned to raise money.

that's a different topic but i've. learned to you know raise money from. friends family and fools as they say and. that's a good line. oh that's a good sign but you know so i. we're trying to do something we were. trying to change the world peter and i. are super ambitious we wanted to make. array computing and we had ideas for. really what's still still the energy. right now how do you do. at scale data science we had a bunch of. ideas there but one of them. i just talked to people about llvm and i. was like there's a way to do this i just.

i went uh i heard about my friend dave. beasley at a compiler course so i was. looking at compilers like and i realized. oh this is what you do and so i wrote a. version. of number that just basically. mapped. python byte code to lvm. nice right so. and the first version is like this works. and it produces code that's fast this is. cool for you know obviously a reduced. subset of python i didn't support all. the python language there had been. efforts to speed up python in the past. but those efforts were i would say not. from the array computing perspective not.

from the perspective of wanting to. produce a vectorized improvement they. were from perspective of speeding up the. runtime of python which is fundamentally. hard because python allows for some. constructs that aren't you can't speed. up like it's this generic you know what. is this variable so i from the start did. not try to replicate python's semantics. entirely i said i'm going to take a. subset of the python syntax and let. people write syntax in python but it's. kind of a new language really so it's. almost like for loops like focusing on.

for loops scalar arithmetic. you know typed you know really typed. language. a type subset. that was the key so but we wanted to. add inference of types so you didn't. have to spell all the types out because. when you call a function. so python is typed it's just dynamically. typed so you don't tell it what the. types are but when it runs every time an. object runs there's a type for the. variables you know what it is. and so that was the design goals of. number were to make it possible to write. functions that could be compiled.

and and have them use for numpy arrays. like the needed support numpy race and. so uh how does it work you have a. comment within python that tells to do. like how do you help out compiler yeah. so. there isn't much actually you don't it's. kind of magical in the sense that just. looks at the type of the objects and. then does type inference to determine. any other variables it needs and then it. was also because we had a use case that. that could work early like one of the. challenges of any kind of new new.

development is if you have something. that to make it work it's going to take. you a long time it's really hard to get. out of the ground if you have a project. where there's some incremental story it. can start working today and solve a. problem then you can start getting it. out there getting feedback because. number today you know numbers nine years. old today right the first. two three versions were not great right. but they solved a problem and some. people could try it we could get some. feedback on it not great and that it was. very focused very very fragile very. substantive.

the subset it would actually compile was. small and so if you wrote python code. and said to the way it worked did you. write a function and you say at jit use. decorators so decorators just these. little constructs let you. decorate code with an app and then a. name. the atgit would take your python. function and actually just compile it. and replace the python function with a. another function that interacts with. this compiled function got it and it. would just do that and. it would you know we went from python. bytecode then we went to ast i mean. writing compiler is actually i learned a.

lot about why computer science is taught. the way it is because. compilers can be hard to write there's. they use tree structures they use all. the concepts of computer science that. are needed and it's actually hard to to. you can it's easy to. to write a compiler and have it be. spaghetti code like the passes become. challenging and we ended up with three. versions of number right number got. written three times. what's uh. what programming language is number. written in python. okay yeah python. so really yeah it's fascinating yeah so.

python but then the whole goal of number. is to translate python byte code. to llvm and so lvm actually does the. code generation in fact a lot of times. they'd say yeah it's super easy to write. a compiler if you're not writing the. parser nor the code generator right so. for people who don't know llvm is the. compiler itself so you're compatible. yeah it's really badly named low level. virtual machine which that part of it is. not used it's really low doesn't mean. that. love chris but the name makes you imply.

that the virtual machine is what it's. all about it's actually the ir and the. library the the code generation that's. the real beauty of it the fact that what. i love about llvm was the fact that it. was a plateau you could collaborate on. right instead of the internals of gcc or. the internals of the intel compiler like. how do i extend that. and it was a place we could collaborate. and. we were early i mean people had started. before it's a slow compiler like it's. not a fast compiler so. for. some kind of jits like jits are common.

in language because one uh every browser. has a javascript jet it does real-time. compilation of the javascript to machine. code. for people who don't know jet is just in. time compilation thank you yeah just in. time compilation they're actually really. sophisticated in fact i got jealous of. how much effort was put into the. javascript jets yes well it's kind of. incredible what they've done yes that. was good. i completely agree i'm very impressed um. but you know number wasn't it was it was. an effort to make that happen with. python and so we used some of the money.

raised for anaconda to do it and then we. also applied for this darpa grant and. used some of that money to continue the. development and then we used proceeds. from service pro projects we would do we. get consulting projects on. uh that we would then use some of the. profits to invest in numbers so we ended. up with a team of two or three people. working on number. it was a fits and starts right and. ultimately the fact that we had a. commercial version of it also we were. writing so part of the way i was trying. to fund numbers say well let's do. the free number and then we'll have a. commercial version of number called. number pro and what number pro did is it.

targeted gpus so we had the very first. cuda jit and the very first. jit compiler that in 2013. for 13 you could run not just a vue funk. on cpu but a u function gpus and it was. awesome automatically parallelize it and. get 1000x speed and that's a that's an. interesting funding mechanism because. you know large companies. or larger companies care about. speed exactly in just this way so it's.

it's it's exactly a really good way yeah. there's been a couple things you know. people will pay for one they'll pay for. really good user interfaces. right and so and so i'm always looking. for what are the things people will pay. for that you can actually adapt to the. open source infrastructure one is. definitely user interfaces the second is. speed yeah like a better run time faster. run time and then when you say people. you mean like a small number of people. pay a lot of money but then there's also. this other mechanism that that's true a. ton of people pay that's true a little. bit first i gotta. we mentioned anaconda we mentioned uh.

friends family and fools. so uh anaconda is yet another so there's. a company but there's also a project. correct that is exceptionally impactful. uh in in terms of uh for many reasons. but one of which is bringing a lot more. people into the. um. into the community of folks who use. python so what is. anaconda. what is its goals yeah maybe what is. conda versus anaconda yeah i'll tell you.

a little bit of the history of that. because anaconda we we wanted to do uh. we wanted to scale python because we you. know that was the peter and i had the. goal of when we started on anaconda we. actually started as continuum analytics. was the name of the company that started. it got renamed to anaconda in 2015. but we uh we said we want to scale. analytics numpy's great pandas is. emerging. but these need to run at scale with lots. of machines the other thing we wanted to. do was make user interfaces that were. web we wanted to make sure the web did.

not pass by the python community that we. had a ways to translate your data. science to the web so those are the two. kind of technical areas we thought oh. we'll build products in this space and. that was the. idea. very quickly in but of course the thing. i knew how to do was to do consulting to. make money and to make sure my family. and friends and fools that it invested. didn't lose their money so it's a little. different than if you take money from a. venture fund if you take money from a. venture fund the venture fund they want. you to go big or go home and they're. kind of like expecting 9 out of 10 to. fail or 99 out of 100 to fail.

it's different i was i was out of. barbell strategy i was like i can't fail. i mean i may not do super well but i. cannot lose their money so i'm going to. do something i know can return a profit. but i want to have exposure to an upside. so that's what happened in anaconda we. didn't there was lots of things we did. not well in terms of that structure and. i've learned from since and have it. better but we've uh we did a really good. job of kind of attracting the interest. around the area to get good people. working and then get funneled some money.

on some interesting projects super. excited about what came out of our. energy there like a lot did. so what are some of the interesting. things. dask number bokeh conda uh there was a. data shader panel. holovis. um these are all tools that are. extremely relevant in terms of helping. you build applications build tools build. you know faster code. um there's a couple of days jupiter lab. jupiter lab came out of this too. fascinating yeah.

okay so uh well bokeh does plotting is. that okay is plotting so bokeh was one. of the foundational things to say i want. to do plot and python but have the. things show up in a web right that's. right that's right that's right so. applauding to me still with all due. respect to matplotlib and bokeh is feels. like still an unsolved problem not. it's a big problem right because you're. i mean i don't know it's a visualization. broadly yes right i think we've got a. pretty good api story around certain use.

cases of python plotting yeah but. there's a difference between static. plots versus interactive plots versus. i'm an end user i just want to write a. simple. you know pandas started the idea of. here's a data frame on a dot plot i'm. just going to attach plot as a method to. my object. which was a little bit controversial. right but works pretty well actually. because. there's a lot less you have to pass in. right you can just say here's my object. you know what you are. you tell the visualization what to do. so. that and there's things like that that.

have not been you know super well. developed entirely but bokeh was focused. on interactive plotting so you could. it's a short path between interactive. plotting and. application dashboard application and. there's some incredible work that got. done there right and it was a hard. project because then you're basically. doing javascript and python so we we. wanted to tackle some of these hard. problems and try to and just go after. them we got some darpa funding to help. and it was super helpful it's a funny. story there we actually did two dark. proposals but one we were five minutes. late for.

and darpa has a very strict cutoff. window and so i we had two proposals one. for the bokeh and one for actually. number and the the other work. which one were you late for. the foundational numerical work so. fortunately chris let us use some of the. money to fund still some of the other. foundational work. but it wasn't as. yeah his hands were tired he couldn't do. anything about it uh. that was a whole interesting story. so one of the incredible projects that. you worked on is conda yes so what is.

that about yeah conda it was early on. like i said with scipy sci-fi was a. distribution masculine library and he. said talk to me talking about compiler. issues and trying to get the stuff. shipped and the fact that people can use. your libraries that they have it. so for a long time we've understood the. packaging problem in python and one of. the first things you did at that. consumer analytics became anaconda. was organize the pi data ecosystem in. conjunction with num focus we actually. started num focus uh uh with some other. folks in the community the same year we.

started anaconda i said we're going to. build a corporation but also got to. reify the community aspect and build a. nonprofit so we do both of those can we. pause real quick and and uh can you say. what is pi pi the python package index. like this whole. story yeah of packaging in python yeah. that's what i'm going to get to actually. this is exactly the journey i'm honest. to sort of explain packaging in python i. think it's best expressed so the. conversation i had with guido at a. conference where i said so. you know yeah packaging is kind of a.

problem megiddo said i don't ever care. about packaging i don't use it i don't. install new libraries i'm like i guess. if you're the language creator and if. you need something just put it put it in. the distribution maybe you don't worry. about packaging. but guido has never really cared about. packaging right and never really cared. about the problem of distribution. somebody else's problem and that's a. fair position to take i think as a. language creator. in fact there's a. philosophical question about should you. have different development packaging. managers should you have a package. manager per language is that really the. right approach i think there are some.

answers of. it is appropriate to have development. tools and there's an aspect of. development tool that is related to. packaging and every language should have. some. story there to help their developers. create so you should have language. specific language tools development. tools that relate to package managers. but then there's a very specific user. story around package management that. those language specific package managers. have to interact with and currently. aren't doing a good job of that that was. one of the challenges that if not seeing. that difference and it still exists in.

the in the difference today conda always. was a user. i'm i'm going to use python to do data. science i'm going to use python to do. something. how do i get this installed it was. always focused on that so it didn't have. like a develop you know. classic example is pip has a pip develop. it's like i want to install this into my. current development environment today. now. khan doesn't have that concept because. it's not part of the story for people. who don't know. pip. is a uh python. specific packaging manage package.

manager right that's. that's exceptionally popular that's. probably like the default thing it's the. default user yeah and so the story there. emerged because what happened is. in 2012 we had this meeting at the. google googleplex and guido was there to. come talk about what we're going to do. how we're going to make things work. better and. wes mckinney me peter peter has a great. photo of me talking to guido and he. pretends we're talking about this story. maybe we were maybe before but we did at. that meeting talked about it and asked. you know what. we need to fix packaging in python like. people can't get the stuff and he said.

go fix it yourself i don't think we're. gonna do it. all right the origin story right there. all right you said okay you said to do. this ourselves so. at the same time people did start to. work on the packaging story in python. it just took a little longer so in 2012. kind of motivated by our training. courses we were teaching like how to. very similar to what you just mentioned. about your mother like it was motivated. by the same purpose like how do we get. this into people's hands and it's this. big long process it takes too expensive. it was actually hurting numpy. development because i would hear people.

were saying don't make that change to. numpy because i just spent a week. getting my python environment and if you. change. if you change numpy i have to reinstall. everything and reinstalling such a pain. don't do it i'm like wait okay so now. we're not making changes to a library. because of the installation problem that. it'll cause for end users okay there's a. problem with pac there's a problem with. installation we've got to fix this so we. we we said we're going to make a. distribution of python and we'd. previously done that previously done. that at m thought. i wanted to make one that would give. away for free everyone could just get. like that was critical that we just get.

it you know it wasn't tied to a product. it was just you could get it. and then we had constantly thought about. well do we just leverage rpm do we but. the challenge had always been we want a. package manager that works on windows. mac os 10 and linux the same. right and it wasn't there like you don't. have anything like that you have and for. people who don't know rpm is red. operating system specific packaging. correct it's an operating specific yes. so do you create the the design uh. questions do you create an umbrella.

package manager then yes cross operating. system yes that was the decision and a. neighboring design question is do you. also create a package manager that spans. multiple programming languages correct. exactly that was the world we faced and. we decided to go multiple operating. systems multiple and programming. language independent because even python. in particular was important was scipy. has a bunch of 4chan in it right and. scikit-learn has links to a bunch of c. plus there's a lot of compiled code and. the python package managers especially.

early on didn't even support that so in. 2000 so we. we we released anaconda which was just a. distribution of libraries but we started. work on condo in 2012. first version of kana came out in early. 2013 2000 summer of 2013 and it was a. package manager so you could say con. install psychic scikit-learn in fact. that was the. scikit-learn was a fantastic project. that emerged kind of it was the the. classic example of the scikits i still. talked to earlier about scipy being too. big to be a single library well what the.

community had done is said let's make. side kits and there's psychic image. there's psychic learn there's a lot of. sidekicks. and it was a fantastic move you know the. community did i didn't do it. i was like okay that's a good idea i. didn't like the name i didn't like the. fact you typed scikit image i was like. that's going to be simpler sk learn we. got to make this smaller. i don't like typing all this stuff from. imports so i was kind of a pressure that. way but i love the energy i love the. fact that they went out and they did it. and dost people jared millman and then. of course guyell and and there's people. i'm not even naming that psychic learn.

really emerged this fantastic project. and the documentation around that is. also incredible this was incredible. exactly i don't know who did that but. they did a great job a lot of people in. inria a lot of people a lot of european. contributors um andreas there's some. andreas uh in the u.s there's a lot of. just people i just adore i think are. amazing people um. awesome use of sci-fi right i love the. fact that they were using sci-fi. effectively do something i love which is. machine learning. um but i couldn't install it because. there's so many pieces involved so many.

dependencies right yes so our our use. case of condo was con install cycle. learn. right and it was the best way to install. second learn in. 2013 to really 2018. 17 18. pip finally caught up i still don't i. still think it's you should khan install. second learn for the pip install second. learn but you can dip install second. learn the the issue is the package they. created was wheels and pip does not. handle the multi-vendor approach they. don't handle the fact you have c plus. libraries you're depending on they just.

stop at the python boundary and so what. you have to do in the real world is you. have to. vendor you have to take all the binary. and vendor it now if your change happens. underlying dependency. you have to redo the whole wheel so. tensorflow is a good example but you. should not pip install tensorflow it's a. terrible idea people do it because. the popularity of pip many people think. of course that's how i install. everything python yeah this is one of. the big challenges. you know you take a github repository or. just a basic blog post the number of.

times pip is mentioned over conda is. like 100x to one correct correct so they. just haven't that was increasing it. wasn't true early because pip didn't. exist like conda came first so but. that's like the long tail of the. internet documentation user generated so. that like you think how do i install. google how do i install tensorflow. you're just not going to see conda in. that first page not correct exactly and. that today you would you would have in. 2016. 2017 and it's sad because you saw the.

condos solves a lot of usability issues. correct like for especially super. challenging thing i don't know one of. the big pain points for me was uh just. on the computer vision side uh opencv. yeah installation that perfect example. yes i think is i don't know if. condos solved that pun has an open cv. package i don't know i i certainly know. pip has not solved. i mean there's complexities there. because right i actually don't know i. should probably know a good answer for. this but. you know if if you compile.

opencv with certain dependencies. you'll be able to do certain things so. there's this kind of flexibility of what. you like what br what options you. compile with yes and i don't think it's. trivial to do that in with condor or or. so has a notion of variance. of a package you can actually have. different compilation versions of a. package so not just the version's. different but oh this is compiled with. these optimizations. so kana does have an answer to those. flavors that's flavors basically.

as far as i know does not no no pip. generally. hasn't thought deeply about the binary. dependency problem right and that's. that's why fundamentally it it doesn't. work for the sci-fi ecosystem it barely. it you can sort of paper over it and. duct tape and it kind of works until it. doesn't it falls apart entirely so it's. been a mixed bag like. and i've been having lots of. conversations with people over the years. because again it's an area where if you. if you understand some things but not. all the things but they've done a great.

job of. community appeal this is an area where i. think anaconda. uh as a company needed to do some things. in order to. make condom more community-centric right. and this is a i talk about this all the. time there's there's a balance between. you have every project starts what i. call company backed open source even if. the company is yourself it's just one. person just you know doing business ads. but ultimately for products to succeed. virally and become massive influencers. they have to create they have to get. community people on board they have to.

get other people on board so it has to. become community driven and a big part. of that is engagement with those people. empowering people governance around it. and there was and what happened with. khan in the early days pip emerged and. we did we did do some good things condo. forge the kind of forage community is. sort of the community. recipe creation community mm-hmm the. condo itself i am still believe and and. you know peter is ceo of anakin he's my. co-founder i ran anaconda tell 2017 2018. is peter still in peter salanakande. right and we're still great friends we.

have great friends we talk all the time. i love him to death there's a long story. there about like why and how when we. could cover in some some other podcast. perhaps yeah sort of a more maybe a more. business focused one but but um. this is one area where i think condos. should be. more community driven like he he should. be pushing more to get more community. contributors to conda and let let let. the. not like anika shouldn't be fighting. this battle yeah right it's actually uh. it's really the developers like you said. like help the developers yeah and then.

they'll actually move us the right. direction but that was the problem i. have as many of the cool kids i know. don't use conda and that to me is. confusing it is confusing and it's. really a matter of kind of has some. challenges first of all kind of still. needs to be improved there's lots of. improvements we made and that it's that. aspect of wait who's doing this and the. fact that then the pipea really stepped. up like they were not solving the. problem at all and now they kind of got. to where they're solving it for the most. part and then effectively you could get. like conda solved a problem that was. there and it still does and it's still.

you know there's still great things it. can do. but um and we still use it all the time. at quansite and with other clients but. with uh but you can kind of do similar. things with pip and docker. right so. especially with the web development. community that part of it again is this. is the there's a lot of different kind. of developers in the python ecosystem. and. there's still a lack of of some clear. understanding i go to the python. conference all the time and there's only. a few people in the pipea who get it and. then others who are just massively.

trumpeting the power of pitt but just do. not understand the problem yeah so one. of the obvious things to me from a mom. from a non-programmer perspective is the. across. operating system usability that's much. more natural so yeah they use. windows and just it seems much easier to. uh to recommend conda there but then it. you should also recommend it across the. board so i'll i'll definitely surf but. what i recommend now is a hybrid i do i. mean i have no problem is it possible to. use oh it is it is what i like build the. environment with pip with conda build an.

environment with conda. and then pip install on top of that. that's fine be careful about pip. installing. opencv or tensorflow or because if. somebody's allowed that it's going to be. most surely done in a way that can't be. updated that easily so install like the. big. packages the infrastructure yeah. and then the weirdos yeah the the like. the weird like implementation for some. uh i had a there's a cool. library i used. that. based on your location.

and time of day and date tells you the. exact position of the sun relative. to the earth and it's just like a simple. library but yeah it's very precise and i. was like all right but that was. that was uh in this episode the thing. they did really well is python. developers who want to get their stuff. published they you have to have a pip. recipe yeah right i mean even if it's. you know the challenge is there's a key. thing that needs to be added to pip just. simply add the pip the ability to defer.

to a system package manager like because. it's you know recognize you're not going. to solve all the dependency problem. so let like give up and allow the allow. system packager to work that way. anaconda's installed and it has pip it. would default to conda to install this. stuff but red hat rpm would default to. rpm to install but it's all more things. like that's the that's a key. not difficult but somewhat where some. work feature needs to be added that's an. example of something like i've known we. need to do it i mean it's where i wish i.

had more money i wish i was more. successful in the in the business side. trying to get there but i wish my you. know my family friends in full community. that i know was larger was larger and. had more money because i know tons of. things to do effectively with more. resources. but. you know i have not yet been successful. with channel. tons of it you know some you know i'm. happy with what we've done. we we've created again at quansite. what we created to get anaconda started. we created analytics and it kind of. started done it again with quansite. super excited by that by the way it took.

three years to do it what is kwan site. what is its mission yeah we've talked a. few times about different fascinating. aspects of it but let's like big picture. what is big picture quan site kwansai is. uh it's mission is to connect data to an. open economy so it's basically. consulting the pi data ecosystem right. it's a consulting company and what i've. said when i started it was we're trying. to create products people and technology. so it's divided into two groups and a. third one as well the two groups are a. consulting services company that just.

helps people do data science and data. engineering and data management. uh better and more efficient full stack. like full stock science full thing will. help you build a infrastructure if. you're using jupiter we need we do staff. augmentation need more programmers help. use das more effectively help us gpus. more effectively just basically a lot of. people need help so we do training as. well to help people you know both uh. immediate help and then get get learn. from somebody uh we've added a bunch of. stuff too we've kind of separated some. of these other things into another. company called open teams that we.

currently started one things i loved. what we did at anaconda was creating a. community innovation team and so i want. to replicate that this time we did a lot. of innovation in anaconda i wanted to do. innovation but also. contribute to the projects that existed. like create a place where maintainers so. the scipy and numpy and number and all. these projects we already started can. pay people to work on them and keep them. going so that's labs quonsite labs is a. separate organization it's a non-profit. mission. the profits of quansite help fund it in.

fact every project that we. have at quansite a portion of the money. goes directly to quansite labs to help. keep it funded so we've gotten several. mechanisms we keep quansite labs funded. and currently so i'm really excited. about labs because it's been a mission. for a long time what kind of projects. are within labs so labs is working to. make the software better like make numpy. better make scipy better make it it's it. only works on open source. so. you know if somebody wants to so you. know companies do we have a thing called. the community work order we call it if a. company says i want to make spider. better okay cool.

um you can pay for a month of. a developer of spider. or developer of numpy or developer of. scipy you're not you can't tell them. what you want them to do you can give. them your priorities and things you wish. you wish existed and they'll work. on those priorities with the community. to get what the community wants and what. emerges what the community wants is. there some aspect on the consulting side. that is helping. as we were talking about morphology and. so on is there specific applications. that are particularly like. driving sort of inspiring the need for.

updates to science correct absolutely. absolutely gpus are absolutely one of. those and new hardware beyond gpus i. mean. tesla's dojo chip i'm hoping we'll have. a chance to work on that perhaps. um. things like that are definitely driving. it the other thing is driving is. scalable like speed and scale. uh how do i write numpy code or numpy. like code if i want it to run across a. cluster you know that's das or maybe. it's ray i mean there's sort of ways to. do that now or there's moden and there's. so pandas code numpy code sci-fi code.

second learn code that i want to scale. so that's one big area have you gotten a. chance to chat with andre and elon about. partic because like no i would love to. by the way i have not but i'd love to i. just saw their tesla ai days uh video. yeah super exciting so that's one of the. you know i love great engineering. software engineering teams and. engineering teams in general and they're. doing a lot of incredible stuff with. python they're like they are. revolutionizing so many aspects of the. machine learning pipeline i agree that's. operating in the real world and so much.

of that is python like you said the guy. running you know andre capati running. autopilot is tweeting about optimization. of. in fact we have at quonset we've been. fortunate enough to work with facebook. on pytorch directly so we have about 13. developers at quonsite some of them are. in labs working directly on pi torch on. torch right so i basically started. quantity i went to both tensorflow and. pi torch and said hey i want to help. connect what you're doing to the broader.

sci-fi ecosystem because i see what. you're doing we have this bigger mission. we want to make sure we don't you know. lose energy here so uh and facebook. responded really positively and. i didn't get the same reaction not yet. not yet i love the folks. so i really love the folks tensorflow. too they're fantastic i think it's the. just. how it integrates with their business i. mean like i said there's a lot of. reasons just the timing the integration. with their business what they're looking. for they're probably looking for more. users and i was looking to kind of have.

some development effort and they. couldn't receive that as easily i think. so i'm hoping i'm really hopeful uh and. love love the people there. what's the idea behind open teams so. open teams i'm super excited about open. teams because it's one of the. i mentioned my idea for investing. directly in open source so that's a. concept called ferro ss but one of the. things we when we started quansite we. knew we would do is we developed. products and ideas and new companies. might come out at anaconda this was. clear right anaconda we did so much. innovation that like five or six.

companies could have come out of that. and we just didn't structure it so they. could but in fact they have you look at. das. there's two companies coming on a desk. you know bokeh could be a company. there's like lots of companies that. could exist off the work we did there. and so i thought oh here's a recipe for. an incubation. a concept that we could actually spawn. new companies and new new innovations. then the idea has always been. well money they earn should come back to. fund the open source project so so labs. is you know i think there should be a. lot of things like quansite labs i think.

this concept is one that scales you. could have a lot of open source research. labs. along the way so in 2018 when the bigger. idea came how to make open source. investors i said oh i need to write i. need to create a venture fund so we. created a venture fund called quonset. initiate at the same time it's an angel. fund really it's you know we started to. learn that process how do we actually do. this how do we get lp's how do we. actually go in this direction and build. a fund. and i'm like every venture fund should. have an associate open source research. lab there's just no reason like our. venture fund. the carried interest. portion of it goes to the lab it's it.

directly will fund the lab that's. fascinating brother so you use the power. of the organic formation of teams in the. open source community and then like. naturally. that leads to a business that can make. there are so many yeah correct it always. maintains and loops back to the opening. looks back to open source exactly it's a. natural fit there's something there's. there's absolutely a repeatable pattern. there and it's also beneficial because. oh i have i have natural connections to. the open source if i have an open source. research lab like. they'll always they'll be out there.

talking to people and so. we've we've had a chance to talk to a. lot of early stage companies and we in. our fund focus on the early stages so. kwan site has the services the lab the. fund right in that process a lot of. stuff started to happen like oh you know. we started to do recruiting and support. and training and i was starting to build. a bigger sales team and marketing team. and people besides just developers and. one of the challenges with that is you. end up with different cultural aspects. you know developers. you know there's a. in any company you go to you kind of go.

look is this a business led company a. developer led company do they kind of. co-exist how are they what's the. interface between them there's always a. bit of attention there like we were. talking about before you know what is. the tension there with open teams i. thought wait a minute we can actually. just create like this concept of. quantity plus labs it's well worth while. it's specific to the pi data ecosystem. the concept is general for all open. source so open teams emerge as oh we can. create a business development company. for. many many quant sites like thousands of. kwan sites and it can be a marketplace. to connect essentially be the enterprise.

software company of the future if you. look at what enterprise software wants. from the customer side and during this. journey i've had the chance to work and. and sell to lots of companies exxon and. shell and jv morgan bank of america like. the fortune 100. and talk to a lot of people in. procurement and see what are they buying. and why are they buying so you know i. don't know everything but i've learned a. lot about oh what are they really. looking for and they're looking for. solutions. they're constantly given products from. the from enterprise software.

here's open source lead enterprise. software now i buy it and they have to. stitch it together into a solution. open source is fantastic for gluing. those solutions together. so. whereas they keep getting new platforms. they're trying to buy what most open. source what most. enterprises want is tools that they can. customize that that are as inexpensive. as they can yeah and so you almost want. to maintain the connection to the open. source because that's yes so open teams. about solving enterprise software. problems brilliant brilliant idea by the. way with a connect but we do it honoring. the topology we don't hire all the.

people we are a network connecting the. sales energy and the in the procurement. energy and we we were on the business. side get the deals closed and then have. a network of partners like quansite and. others who we hand the deals to right to. actually do the work and then we off we. then we have to maintain i feel like we. have to maintain some level of quality. control so the client can rely on open. teams to ensure the deliveries it's not. just here's a lead go figure that out. but no we're going to make sure you get. what you need. right by the way it's such a skill and i. don't know if i have the patience i will.

have the patience to talk to the. business people. or more specific i mean there's all. kinds of flavors of business people or. they're like yeah. marketing people. there's a challenge i hear what you're. saying because i've had the same. challenge yeah and it's true there's. sometimes you think okay this is wait. this is way overwrought yeah so you have. to become an adult you have to because. the the companies have needs they have. ways to make money and they and they. also want to learn and grow and yet it's. your job to kind of educate them on the. best way like the value of open source.

for example right and i'm really. grateful for all my experiences over the. past 14 years understanding that side of. it and still learning for sure but not. just understanding from companies but. also dealing with marketing. professionals and sales professionals. and people that make a career out of. that understanding what they're thinking. about and also understanding what let's. make this better like we can really make. a place like open teams i see as the. transmission layer between companies and. open source communities. producing enterprise software solutions. like eventually we want to like today. we're taking on sas and matlab and tools. that we know we can replace for folks.

really anytime you have a software tool. an organization. where you have to do a lot of. customization or make it work for you. because now you're just buying this. thing off the shelf and it works it's. like okay you buy the system then you. customize a lot usually with expensive. consultants to actually make it work for. you. all of those should be replaced by open. source foundations with the same. customers. such important work such important work. in these giant organizations they're. doing exactly that taking some. proprietary software and hiring a huge. team of consultants that customize it.

and then that's whole thing gets. outdated quick correct and so i mean. that that's brilliant right the one one. solution to that. is how it would like kind of what. tesla's doing a little bit of which is. basically build up a software. engineering team yeah like build a team. from scratch from scratch and companies. are doing it well that's what they're. doing right now yeah right exactly. that's okay and you're creating an. apology for some of that like right you. just don't have to do it that's not the. only answer yeah right and so other. companies can access this new more. flexible we really that's really really.

say open team is the future of. enterprise software. um it's we're still early like this idea. just percolated over the past year as. we've kind of grown quan sight and. realize the extensibility of it uh we. just finished in our seed round uh the. work to help you know get more sales. people and then push the push the. messaging correctly and there's lots of. tools we're building to make this easier. like we want to automate the processes. we feel like a lot of the power is the. efficiency of the sales process there's. a lot of wasted energy. in small teams. and the sales energy to get into large.

companies and make a deal there's a lot. of money spent on that stuff creating. the tools and processes. makes that super seamless so a single. company can go oh i've got my contract. with open teams we've got a subscription. they can get they can make that. procurement seamless and then the fact. they have access to the entire open. source ecosystem and we have a you know. so we have a part of our work that's. embracing open source ecosystems and. making sure we're doing things useful. for them we're serving them and then. companies making sure they're getting. solutions they care about and then. figuring out which which targets we have.

you know yeah we're we're not taking on. all of open source all of enterprise. software yet but but we're well this. feels like the future the idea and the. vision is brilliant uh can ask you uh. why do you think microsoft bought github. and what do you think is the future. great point i thought it was a brilliant. move i think they did because microsoft. has always had a developer centric. culture. like they always have like one things. microsoft's always done well is. understand that their power is. developers right it's been. balmer didn't necessarily make a make a.

good meme about how he approached that. but. they're broadening that i think that's. why because they recognize github is. where developers are at right and so but. do they have a vision like open teams. type of situation right so yeah. are they just basically throwing money. at developers to show their support i. think so without uh a topology like you. put it like a a a way to leverage that. like to give developers actual money. right i don't i don't think so i think. they're still it's an enterprise. software company and they make a bunch.

of money they make a bunch of games they. have a big they're a big company they. sell products i think part of it is they. know there's opportunity to make money. from github right there's definitely a. business there you know uh to sell to. developers or to sell to people using. development i think there's part of that. i think part of it is also there's they. had definitely wanted to recognize that. that that you need to value open source. to get great developers which is which. is an important concept that was. emerging over the past 10 years that you. know. pi data we were able to convince jp. morgan to support pi data because of.

that fact right that was where the money. for them putting a couple hundred. thousand into supporting pi data for uh. several conferences was they want. developers and they realized that. developers want to participate in open. source so enterprise software folks. don't always understand how their. software gets used having spent a lot of. time on the floors at jpmorgan at in. shell at exxon mobil you see oh these. companies have large development teams. and and then you're they're kind of. dealing with the what's being delivered. to them so i really feel kind of a. privilege that i had a chance to learn. some of these people and see what.

they're doing and and even work. alongside them uh you know as a. consultant uh. using my using open source and trying. how do we make this work inside of our. large organization some of it is. actually for a large organization some. of it is messaging to the world that you. care about developers and you're the. cool yep you're you care like for. example like if ford because i talked to. them like car companies right they. they want to attract you know you want. to take on tesla and autopilot you want. to take that right and so.

what do you do there you show that. you're cool like you sh you try to show. off that you care about developers and. they have a lot of trouble doing that. and like one way i think like ford. should have bought github they just to. show off better yeah yeah like these old. school companies and it's in a lot of. good point a lot of different industries. there's probably different ways it's. probably an article. that you care to developers and the. developers it's it's it's exactly what. you like. for example. just spitballing here but like ford or.

somebody like that could give. a hundred million dollars to the. development of numpy and. uh like like literally look at like the. top most popular projects in python. and just say. we're just gonna give money right like. that's gonna immediately make you cool. they could actually yeah and in fact. they set up num focus to make it easy. yeah but the challenge was is also you. have to have some business development. like it's a bit of a it's a bit of a.

seating problem right and you look at. how i've talked to the folks at linux. foundation know how they're doing it i. know how. in starting num focus because. we had two babies in 2012 one was. anaconda one was num focus right and. they were both important efforts they. had distinct journeys and super grateful. that both existed and still grateful. both exist um. but there's different energies in. getting donations. as there is getting. um this is important to my business like. i'm selling you something that. this is not a it's a salt is i'm gonna. make money this way if you can tie it if.

you can tie the message to an roi for. the company it becomes more effective. it's much more effective right so. and there are rational arguments to make. i've tried to have conversations with. mark especially marketing departments. like very early on it was clear to me. that. oh you could just take a fraction of. your marketing budget and just spend it. on open source development and you get. better results from your marketing. like because how do those can i sorry. i'm gonna try not to go. here what have you learned from the. interaction with the marketing folks on.

that kind of because. you gave a great example of something. that will obviously be much better. investment in terms of marketing is. supporting open source projects the. challenge is not dissimilar from the. challenge you have in academia at the. different colleges right knowledge gets. very specific and very channeled right. and so people get they get a lot of. learning in the thing they know about. and it's hard then to bridge that and to. get them to think differently enough to. have a sense that you might have.

something to offer because it's. different it's like well how do i. implement that how do i what do i do. with that like do i which budget do i. take from do i slow down my spend on. google ads or am i spent on facebook ads. or do i not hire a content creator and. say like like there's an operational. aspect to that that some that you have. to be the cmo. right or the ceo you have to get the. right level so you have to hire a high. position level right because they care. about this in this right or they won't. know how right right and and because you. can also do it very clumsily yeah right.

and i've seen it because you can you. actually have to honor and recognize the. the the people you're going to and the. fact that if you just throw money at. them it could actually create more. problems can i just say this is not you. saying can i just because i just need i. need i need to say this. i've been very surprised. how often marketing people are terrible. at marketing. i feel like the best marketing is doing. something novel and unique that. anticipates the future. it feels like so much of the marketing.

practice. is like what they took in school or. maybe they're studying for what was the. best thing that was done in the past. decade. and they're just repeating that over and. over as opposed to innovating like. taking the risk to me marketing a great. point is taking the big risk that's a. great being the first one to risk yeah. and there's an aspect of data. observation from that risk right that's. that's that's you i think because shared. what they're doing already but it. absolutely it's it's about i think it's. content like there's this whole world on. content marketing that you could almost.

say well yeah. it can get over you can get you can get. inundated with stuff that's not relevant. to you whereas what you're saying. would be highly relevant and highly. useful and highly highly beneficial yeah. but it's it's a it's risky i mean that's. why sort of uh there's a lot of. innovative ways of doing that test is an. example of people that basically don't. do marketing uh they do marketing in a. very like. it's like elon hired a person who's just. good at twitter for running tesla's. twitter account. you know right right that's exactly what. you want to be doing you want to be.

constantly innovating in uh right. there's an aspect of telling i mean i've. i've definitely seen. people doing great work where you're not. talking about it like i would say that's. actually a problem i have right now with. quansite labs once the lab's been doing. amazing work really excited about it we. have not been talking about it enough we. haven't been and there's different ways. to talk about it there's different ways. to there's different channels to wish to. communicate right there's also. like. i'll just throw some um shade at. companies i love. uh so for example irobot i just had a. conversation with them they make roombas.

sure and uh they i think i love the. incredible robots but like every time. they do commerci like advertisements not. advertisement but like marketing type. stuff it just looks so corporate. and to me the incredible maybe wrong in. the case of irobot i don't know but to. me when you're talking about engineering. systems. it's really nice to show off the magic. of the engineering and the software and. all the all the geniuses behind this.

product and the tinkering and like the. raw authenticity of what it takes to. build that system versus uh the. marketing people who want to have like. pretty people like standing there all. pretty with the robots like moving. perfectly so i to me there's some aspect. it's like speaking to the hackers you. have to throw some. uh bones some. some care towards the the engineers the. developers because. there's some aspect one for the hiring. but two there's an authenticity to that.

kind of communication that's really. inspiring to the end user as well like. if they know that brilliant people the. best in the world are working your. company they start to believe that that. product that you're creating is really. it's interesting because your initial. reaction would be wait there's different. users here why would you do that to you. know my wife bought a rumba rumba but. she and she you know loves developers. she loves me but she doesn't. care about. that yeah. so essentially what you said is actually. the authenticity because everyone has a.

friend everyone knows people there's. word of mouth i mean if you worded my. mouth is so so yeah exactly. because i think it's the lack of that. that realization there's this halo. effect right and also influences your. general marketing i interesting for some. stupid reason i do have a platform and. it seems that the reason i have a. platform many others like me millions of. others is like the authenticity and like. we get excited naturally about stuff. yeah and like i don't want to get. excited about that irobot video because. it's as boring as marketing as corporate.

as opposed to. i wanted to do some fun this is this is. me like a shout out to irobot is they're. not letting me get into the robot. yeah well there's an aspect of that they. could be benefiting from a from a. culture of modern modularity like. add-ons and right like that could. actually dramatically help if you've. seen that over history i mean apple is. an example of a company like that or or. the like i i can see i can see what your. point is is that you have something that. needs to be. it needs to be adopted broadly the. concept needs to be adopted broadly and.

if you want to go beyond this one device. you need. to engage this community yeah and. connecting to the the open source as you. said i got to ask you. your programmer. uh one of the most impactful programmers. ever. you've led many programmers you lead. many programmers. what are some from a programmer. perspective. what makes a good programmer what makes. a productive programmer is there a. device you can give to be a great. programmer great great question and.

there are times in my life i'd probably. answer this even better than i hope. maybe give an answer today because i've. thought about this at numerous times. like right now i've spent on so much. time recently hiring sales people that. your mind is like. something else on something else but i i. reflected on the past and also uh you. know i have some really the only way i. can do this i have some really great. programmers that i work with who lead. the teams that they they lead and my. goal is to inspire them and and. hopefully help them encourage them and. be uh help them encourage with their. teams i would say there's a number of.

things couple things one is um curiosity. like you i think a programmer without. curiosity. is uh mundane like you'll lose interest. you don't do your best work so it's sort. of it's an affect it's sort of are you. you have some curiosity about things. i think two don't try to do everything. at once recognize that you're you know. we're limited as humans you're limited. as a human and each one of us are. limited in different ways you know we. all have our different strengths and. skills so it's adapting the art of. programming to your skills one of the. things that always works is to limit.

what you're trying to solve. right so um. if you're part of a team usually maybe. somebody else has put the architecture. together and they've gotten given a. portion for you if you if you're young. if you're not part of a team is sort of. breaking down the problem into smaller. parts is essential for you to make. progress it's very easy to take on a big. project and try to do it all at once and. you get lost and then you do it badly. and so. thinking about you know um very. concretely what you're doing to find you. know defining the inputs and outputs to.

finding what you want to get done. um even just talking about that in like. writing down. before you write code just what are you. trying to accomplish and being very. specific about it. uh really really helps i think um. using other people's work. right. don't don't be afraid that somehow. you're. like you should do it all like. nobody does stand on the shoulders. of giants. but but but don't just copy and paste. that's particularly relevant in the era.

of codex and the uh you know the. auto-generated code which is essentially. i see as an indexing of stack overflow. right exactly secondly it's like it's a. search engine it's a search engine over. stack overflow basically so it's not i. mean we've had this for a while yeah but. really you want to cut and paste but but. but not blindly like. absolutely i've cut and paste to. understand but then you understand oh. this is what this means oh this is what. it's doing and understanding as much as. you can so it's critical that's where. the curiosity comes in if you're just. blindly cutting and pasting you're not.

going to understand and so understand. and then you know. be. uh be sensitive to hype cycles. right every for every view often there's. always a oh test driven development is. the answer oh object oriented is the. answer oh. there's always an answer you know agile. is the answer. be cautious of jumping onto a hype cycle. like likely there's signal like there's. a thing there that's that's actually. valuable you can learn from but it's. almost certainly not the answer to. everything you need what lessons do you. draw from you having created numpy and.

scipy like. in service of sort of answering the. question of what it takes to be a great. programmer and giving advice to people. yeah how can you be the next person to. create a cyborg yeah so one is listen. to listen to who uh to uh to people that. have a problem. right which is everybody right but. but listen and listen to many and and. try to uh then do. like don't you you're gonna have to do. an experiment you know do fall down.

don't be afraid to fall down don't be. afraid you're the first thing you do is. probably going to suck and that's okay. right. it's it's honestly i think iteration is. the key to innovation. and and it's that it's almost that. psychological um. hesitation we have to just. uh iterate like yeah we know you you. know it's not great but next i want to. be better i mean just just keep learning. and keep breathing and keep improving so. it's it's an attitude um and then. it doesn't take intense concentration.

right good things don't happen just. it's not quite like tick tock or like. facebook you know you can't scroll your. way to good programming right there are. you know sincere like hours of. of deep don't be afraid of the deep. problem like often people will run away. from something because oh i can't solve. this and it you might be right but give. it an hour give it a couple of hours and. see. and you know just um five minutes not. gonna not gonna give you that was it. lonely when you were building sci-pod. and number hugely yeah absolutely lonely.

in the sense of. you have to have an inner drive. and that inner drive for me always comes. from i have to i have to see that this. is right. in some angle i have to believe it that. this is the right approach the right. thing to do. with scipy it was like oh yeah the world. needs. libraries in python clearly python is. popular enough with enough influential. people that to start and it needs more. libraries. so that is a good in itself so i'm going. to go do that good so find a good find a. thing that you know is good and just.

just work on it um so that has to happen. and it is and you kind of have to have. enough realization of your mission. to be okay with the naysayer or the fact. that notably joins you up front in fact. one thing i've talked to people a lot. i've seen a lot of projects come and. some fail like not everything i've done. has actually worked perfectly i've tried. a bunch of stuff that okay that didn't. really work or this isn't working and. why. but you see the patterns and one of the. key things is. you can't even know. for. six months i say 18 months right now if.

you're starting a new project you got to. give it a good 18 month run before you. even know if the feedback's there like. it's you're not gonna know in six months. you might have the perfect thing but six. months from now it's still kind of still. emerging so give it time because you're. dealing with humans and humans have an. inertial. inertia energy that just doesn't change. that quickly. so let me ask a silly question but uh. you know. like you said you're focused on the. sales side of things currently but. you know back when you're actively. programming maybe in the 90s you talked.

about ids. what's your. a setup that you have that brings you. joy keyboard number of screens yeah well. linux i do still like to program. somebody's not as much as i used to i. have two projects i'm super interested. in. trying to find funding for them trying. to figure out some good teams for them. but i could talk about those um but what. i yeah what get i'm an emacs guy great. thank you. the superior editor everybody i've got. i've i don't often delete tweets but one. of the tweets i deleted when i said.

emacs was better than vim and then the. hate i got it is i was like i'm walking. away from this. i do too i i don't push it i mean i'm. not i'm just joking of course yeah. exactly it's kind of like but people do. take the editor seriously they take it i. did it with your life. it is. but uh there's something there's. something beautiful to me about emacs. but there's for people that love them. there's something beautiful to them i. mean i do use them for quick editing. like command line if i say quick editing. i will still sometimes use it but not. much like it's simple corrective.

correctness single edited character so. when you were developing sci-fi you were. using ebx yep siphon numpy are already. in an e-max on a linux box. and uh. cvs and then svn version control git. came later like git has i love. distributed branch stuff i think git is. pretty complicated but i love the. concept and uh also of course github is. uh and then git lab make git. definitely consumable. um but that came later did you ever. touch lisp at all like what were you.

what yeah emotional feelings about the. parenthesis great question so i find. myself appreciating this today much more. than i did early because when i came to. programming i knew programming but i was. a domain expert right and to me the. parentheses were in the way. it's like wow. it's just all this like it just gets in. the way of my thinking about what i'm. doing so why would i have all these. right. um that was my initial reaction to it uh. you know now as i appreciate kind of the. structure that kind of naturally maps to. the to logical thinking about a program.

i can appreciate them right and why it's. actually you could you you could create. editors that make it not so. problematic right honestly um yeah so i. actually have a much more appreciation. of lisp and things like closure and. there's hyvee which is a python you know. a list that compiles the python byte. code. um i think it's challenging like. typically. these languages are you know i even saw. a whole data science programming system. in lisp that somebody created which is. you know cool but again it's the i think. it's the lack of recognition of the fact.

that there exists what i call occasional. programmers yes people are never going. to be programmers for a living they. don't want to have all this the cuteness. in their head they want just. it's why basic you know microsoft had. the right idea with basic in terms of. having that be the language of. visual basic the language of excel and. um sql sql server they should have. converted that to python 10 years ago. but the world would be a better place. that they had but. there's also uh there's a beauty and a. magic to the history behind a language.

and lisp you know some of the most. interesting people in in the history of. computer science and artificial. intelligence have used lisp so yes you. feel well it's back to that language. when you when you have a language you. can think in it yeah and it helps you. think about it attracts a certain kinds. of people that think a certain kind of. way and then that's that's there okay so. what about like small laptop with a tiny. keyboard or is there like. a screen you know good question i've. never gotten into the many screens to be. honest i mean and. maybe it's because in my head i kind of.

just i just swap between windows. like. partly because i guess i i really can't. process three screens at once anyway. like. i just i'm looking at one and i just. flip you know i flip an application open. so. where it's really helpful is actually. when i'm trying to you know here's data. and i want to input it from here right. that's the only time i really need. another screen so now because you're. both a developer lead developers but. then there's also these businesses and. there's sales people you're working with.

large companies operations people hiring. people yeah the whole thing which. operating system is your favorites. though at this at this point so linux. was the earliest so yeah i love love. linux as a as a server side and it was. early days i was had my own linux. desktop um i've been on mac laptops for. 10 years now yeah that this is what. leadership looks like. as you switch to mac. okay great yeah pretty much i mean. just the fact that i had to do. powerpoints i had to do presentations. and you know plug in i just couldn't.

mess with plugging in laptops it. wouldn't project and so uh you mentioned. also quant labs and things like that uh. can you give advice on how to hire great. programmers. and great people yeah i would say. produce an open source project. yeah get people contributing to it and. hire those people yeah. i mean you're doing it sort of uh. you may be perhaps a little biased but. that's probably 100. really good advice.

i find it hard to hire i still find it. hard to hire like. in terms of i don't think like it's not. hard to hire if i've worked with. somebody for a couple of weeks. but a cup an hour or two of interviews i. i have no idea so that. instinct that radar of knowing if you're. good or not that you've you found that. you're still not able to it's really. hard i mean the resume can help but. again the resume is like a presentation. of the things they want you to see not. the reality of. of and there's also um.

you know you have to understand what. you're hiring for there are different. stages and different kinds of skills and. so it isn't just. one of the things i talk a lot about. internally at my companies is that the. whole idea of measuring ourselves. against the unit a single axis is flawed. because we're not it's a. multi-dimensional space and how do you. order a multi-dimensional space there. isn't one ordering so this whole idea. you you immediately have projected into. a thing when you're talking about hiring. or best or worst or better or not better.

so what is the thing you're actually. needing. and you can even hire for that there is. such a thing generally i really value. people who have the affect. they care about open source like so in. some cases their affinity to open source. is simply a kind of a filter of an. affect. however i have found this interesting. dichotomy between. open source contributors and. product creation. there's i don't know if it's fully true.

but there does seem to be the more. uh the more experience the more affect. somebody has an open source community. the less ability to actually produce. product that they have. but the other one's kind of true too the. more product focused star i find a lot. of people talk to a lot of people who. produce really great products and they. they have a they're looking over the. open source communities kind of wanting. to participate and play but they've. played here. and they do a great job here and then. they. don't necessarily have some of the same. i don't think that i don't think that's.

entirely necessary i think part of it is. cultural how that's how they've emerged. except because one of the things that. open source communities often lack is. great product management like some. product management energy that's. brilliant but you want both of those. energies in a second place together yes. you really do and so it's a lot of it's. creating these teams of people that have. these needed skills and attributes that. are. hard and so. so one of the big things i look for is. somebody that fundamentally recognizes. their need to learn to learn like one of. the values that we we have and all of. the things we do is learning like.

if somebody thinks they know it all. they're going to struggle and some of. that is just. there's more basic things like humility. just. being humble in the face of all the. things you don't know and that's like. step one of learning that's step one of. learning right and you know i've spent a. lot of time learning right other people. spend a lot more time but i've spent a. lot of time learning i mean my whole. goal was to get a phd because i loved. school and i wanted to be a scientist. and then. what i found is what's been written.

about elsewhere as well is the more i. learned the more i didn't know the more. i realized man i i. i know about this. but this is such a tiny thing in the. global scope of what i might want to. know about so. i need to be listening a whole lot. better than than i am just. talking. that's changed a little bit actually my. wife says that i used to be a better. listener now that i have i'm so full of. all these ideas i want to do she kind of. says you got to give people time to talk. so you you've uh succeeded on multiple. dimensions so one is the. 10-year track faculty.

uh the others just creating all these. products then building up the businesses. then working with businesses. uh do you have advice for young people. today in high school in college of how. to. live a life. as a non-linear and as successful as. yours. a life that could be uh they could be. proud of well like that's that's a super. compliment i'm humbled by that actually. i i would say. a life they can be proud of honestly one. thing i've said to people is.

first um find people you love and care. about them. like family matters to me a lot and. family means people you love and have. committed to. right so it can be whatever you you mean. by that but it's it you need to have a. foundation. uh so find people you love and want to. commit to and do that. um because it anchors you in a way that. nothing else can. right and then. and then you find other things and then. kind of from out there you find other. kinds of things you can commit to. whether it's ideas or or people or.

groups of people. um so you know especially in high school. i would say don't settle on what you. think you know. right. give yourself 10 years to think about. the world like there's i see a lot of. high school students who who seem to. know everything already i i think i did. too i think it's maybe natural but. but recognize that. the things you care about you might. change your perspective over time. i certainly have over time as senator. you know i was really passionate about. one specific thing and i was kind of.

softened you know i was a big um i. didn't like the federal reserve right. and. there's still we can have a longer. conversation about monetary policy and. finances but. but. i'm a little more uh nuanced in my in my. perspective at this point. um. but you know that's that's one area. where you learn about someone going i. want to attack it you know. build don't destroy. like build like someone so often the. tendency is to not like something they. want to go attack it. build something build some to replace it.

yeah build up you know attract people to. your new thing. you'll get you'll be far more far better. right you don't need to destroy. something to build something else. um so that's i guess generally. uh and then you know definitely. uh like curiosity you know follow your. curiosity. and and let it um. don't just follow the money. and all of that like you said is. grounded in um. family friendship and ultimately love.

yes. which is uh. a great way to end it travis you're one. of the most impactful people in the. engineering computer science in the. human world so i truly appreciate. everything you've done. and i really appreciate that you would. spend your valuable time with me it was. an honor it was a real pleasure for me i. appreciate that thanks for listening to. this conversation with travis oliphant. to support this podcast please check out. our sponsors in the description. and now let me leave you with something.

that in the programming world is called. hodgson's law. every sufficiently advanced lisp. application will eventually be. re-implemented in python. thank you for listening and hope to see. you next time. you.