Jensen Huang's journey from GPUs to AI revolution

That's Jensen Wong.

and whether you know it or not, his decisions are shaping your future. He's the CEO of Nvidia, the company that skyrocketed over the past few years to become one of the most valuable companies in the world because they led a fundamental shift in how computers work.

Unleashing this current explosion of what's possible with technology. NVIDIA's done it again. We found ourselves being one of the most important technology companies in the world and potentially ever. A huge amount of the most futuristic tech that you're hearing about in AI and robotics and gaming and self-driving cars and breakthrough medical research relies on new chips and software designed by him and his company. During the dozens of background interviews that I did to prepare for this, what struck me most was how much Jensen Wong has already influenced

all of our lives over the last 30 years, and how many said it's just the beginning of something even bigger. We all need to know what he's building and why, and most importantly, what he's trying to build next. Welcome to Huge Conversations. Thank you so much. I'm so happy to do it. Before we dive in, I wanted to tell you

Okay. Okay. You can ask me anything you want. Okay. Okay.

and we've covered... I'm the worst person to be an explainer video. I think you might be the best. And that's what I'm really hoping that we can do together is make a joint explainer video about how can we actually use technology to make the future better. And we do it because we believe that when people see those better futures, they help build them. So the people that you're going to be talking to are awesome. They are optimists who want to build those better futures.

But because we cover so many different topics, we've covered supersonic planes and quantum computers and particle colliders, it means that millions of people come into every episode without any prior knowledge whatsoever. You might be talking to an expert in their field who doesn't know the difference between a CPU and a GPU. Or a 12-year-old who might grow up one day to be you, but is just starting to learn. For my part, I've now been preparing for this interview for several months. I've

including doing background conversations with many members of your team, but I'm not an engineer. So my goal is to help that audience see the future that you see. So I'm going to ask about three areas. The first is, how did we get here? What were the key insights that led to this big fundamental shift in computing that we're in now? The second is, what's actually happening right now? How did those insights lead to the world that we're now living in that seems like so much is going on all at once? And the third is,

what is the vision for what you see coming next? In order to talk about this big moment we're in with AI, I think we need to go back to video games in the 90s. At the time, I know game developers wanted to create more realistic-looking graphics, but the hardware couldn't keep up with all of that necessary math. NVIDIA came up with a solution that would change not just games, but computing itself.

could you take us back there and explain what was happening and what were the insights that led you and the NVIDIA team to create the first modern GPU so in the early 90s when we first started the company we observed that in a software program inside it there are just a few lines of code maybe 10% of the code does 99% of the processing and that 99% of the processing could be done in parallel however

The other 90% of the code has to be done sequentially. It turns out that the proper computer, the perfect computer, is one that could do sequential processing and parallel processing, not just one or the other. That was the big observation. And we set out to build a company to solve computer problems that normal computers can't. And that's really the beginning of NVIDIA.

My favorite visual of why a CPU versus a GPU really matters so much is a 15-year-old video on the NVIDIA YouTube channel where the Mythbusters, they use a little robot shooting paintballs one by one to show solving problems one at a time or sequential processing on a CPU. But then they roll out this huge robot that shoots all of the paintballs at once, doing smaller problems all at the same time or parallel processing on a GPU.

so nvidia unlocks all of this new power for video games why gaming first the video games requires parallel processing for processing 3d graphics and we chose video games because one we loved the application it's a simulation of virtual worlds and who doesn't want to go to virtual worlds and and we had the good observation that video games

has potential to be the largest market for entertainment ever. And it turned out to be true. And having it being a large market is important because the technology is complicated. And if we had a large market, our R&D budget could be large. We could create new technology. And that flywheel between technology and market and greater technology was really the flywheel that got NVIDIA to become one of the most important technology companies in the world. And it was all because of video games.

I've heard you say that GPUs were a time machine. Yeah. Could you tell me more about what you meant by that? A GPU is like a time machine because it lets you see the future sooner. One of the most amazing things anybody's ever said to me was a quantum chemistry scientist. He said, Jensen, because of NVIDIA's work, I can do my life's work.

That's time travel. He was able to do something that was beyond his lifetime, within his lifetime. And that's because we make applications run so much faster. And so you get to see the future. And so when you're doing weather prediction, for example, you're seeing the future. When you're doing a simulation, a virtual city with virtual traffic, and we're simulating our self-driving car through that virtual city, we're doing time travel.

So parallel processing takes off in gaming. And it's allowing us to create worlds in computers that we never could have before. And gaming is sort of this first incredible case of parallel processing unlocking a lot more power. And then, as you said, people begin to use that power across many different industries. The case of the quantum chemistry researcher.

When I've heard you tell that story, it's that he was running molecular simulations in a way where it was much faster to run in parallel on NVIDIA GPUs even then than it was to run them on the supercomputer with the CPU that he had been using before. Yeah, that's true. So, oh my God, it's revolutionizing all of these other industries as well. It's beginning to change how we see what's possible with computers. And my understanding is that in the early 2000s, you see this and you...

realize that actually doing that is a little bit difficult because what that researcher had to do is he had to sort of trick the GPUs into thinking that his problem was a graphics problem. That's exactly right. No, that's very good. You did some research. So you create a way to make that a lot easier. That's right. Specifically, it's a platform called CUDA, which lets programmers tell the GPU what to do using programming languages that they already know, like C. And that's a big deal because it gives way more people easier access to all of this computing technology.

Could you explain what the vision was that led you to create CUDA? Partly researchers discovering it, partly internal inspiration, and partly solving a problem. And a lot of interesting ideas come out of that soup. Some of it is aspiration and inspiration. Some of it is just desperation. And so in the case of CUDA, it was very much the same way.

and probably the first external ideas of using our GPUs for parallel processing emerged out of some interesting work in medical imaging. A couple of researchers at Mass General were using it to do CT reconstruction. They were using our graphics processors for that reason and it inspired us. Meanwhile, the problem that we're trying to solve inside our company

has to do with the fact that when you're trying to create these virtual worlds for video games you would like it to be beautiful but also dynamic water should flow like water and explosions should be like explosions so there's particle physics you want to do fluid dynamics you want to do and that is much harder to do if your pipeline is only able to do computer graphics

and so we have a natural reason to want to do it in the market that we were serving. So researchers were also horsing around with using our GPUs for general purpose acceleration. And so there were multiple factors that were coming together in that soup. When the time came, we decided to do something proper and create a CUDA as a result of that. Fundamentally, the reason why I was certain that CUDA was going to be successful, and we...

put the whole company behind it was because fundamentally our GPU was going to be the highest volume parallel processors built in the world because the market of video games was so large. And so this architecture has a good chance of reaching many people. It has seemed to me like creating CUDA was this incredibly optimistic, huge, if true thing to do where you were saying, if we create a way for many more people to use much more computing power,

They might create incredible things. And then, of course, it came true. They did. In 2012, a group of three researchers submits an entry to a famous competition where the goal is to create computer systems that could recognize images and label them with categories. And their entry just crushes the competition. It gets way fewer answers wrong. It was incredible. It blows everyone away. It's called AlexNet, and it's a kind of AI called the neural network.

My understanding is one reason it was so good is that they used a huge amount of data to train that system. And they did it on NVIDIA GPUs. All of a sudden, GPUs weren't just a way to make computers faster and more efficient. They're becoming the engines of a whole new way of computing. We're moving from instructing computers with step-by-step directions to training computers to learn by showing them a huge number of examples.

This moment in 2012 really kicked off this truly seismic shift that we're all seeing with AI right now. Could you describe what that moment was like from your perspective? And what did you see it would mean for all of our futures? When you create something new like CUDA, if you build it, they might not come. And that's always the cynics perspective. However, the optimist perspective would say, but if you don't build it, they can't come.

And that's usually how we look at the world. You know, we have to reason about intuitively why this would be very useful. And in fact, in 2012, Ilya Suskovor and Alex Krzyzewski and Jeff Hinton in the University of Toronto, the lab that they were at, they reached out to a GeForce GTX 580 because they learned about CUDA and that CUDA might be able to be used as a parallel processor for training AlexNet. And so our inspiration that GeForce could be the

The vehicle to bring out this parallel architecture into the world and that researchers would somehow find it someday was a good strategy. It was a strategy based on hope, but it was also reasoned hope. The thing that really caught our attention was simultaneously we were trying to solve the computer vision problem inside the company and we were trying to get CUDA to be a good computer vision processor.

And we were frustrated by a whole bunch of early developments internally with respect to our computer vision effort and getting CUDA to be able to do it. And all of a sudden, we saw AlexNet, this new algorithm that is completely different than computer vision algorithms before it, take a giant leap in terms of capability for computer vision. And when we saw that, it was partly out of interest, but partly because we were struggling with something ourselves.

And so we were highly interested to want to see it work. And so when we looked at AlexNet, we were inspired by that. But the big breakthrough, I would say, is when we saw AlexNet, we asked ourselves, you know, how far can AlexNet go? If it can do this with computer vision, how far can it go? And if it could go to the limits of what we think it could go, the type of problems it could solve

what would it mean for the computer industry and what would it mean for the computer architecture and we were we were we rightfully reasoned that if machine learning if deep learning architecture can scale the vast majority of machine learning problems could be represented with deep neural networks and the type of problems we could solve with machine learning is so vast that it has the potential of reshaping the computer industry altogether

which prompted us to re-engineer the entire computing stack, which is where DGX came from and this little baby DGX sitting here. All of this came from that observation that we had to reinvent the entire computing stack layer by layer by layer. Computers, after 65 years since IBM System 360 introduced modern general-purpose computing, we've reinvented computing as we know it.

So think about this as a whole story. So parallel processing reinvents modern gaming and revolutionizes an entire industry. Then that way of computing, that parallel processing, begins to be used across different industries. You invest in that by building CUDA. And then CUDA and the use of GPUs allows for a step change in neural networks and machine learning and begins a sort of revolution that we're now seeing.

only increase in importance today. All of a sudden, computer vision is solved. All of a sudden, speech recognition is solved. All of a sudden, language understanding is solved. These incredible problems associated with intelligence, one by one by one by one, where we had no solutions for in the past, desperate desire to have solutions for, all of a sudden, one after another gets solved every couple of years. It's incredible.

Yeah, so you're seeing that in 2012. You're looking ahead and believing that that's the future that you're going to be living in now. And you're making bets that get you there, really big bets that have very high stakes. And then my perception as a layperson is that it takes a pretty long time to get there. You make these bets. Eight years, 10 years. So my question is,

If AlexNet happened in 2012, and this audience is probably seeing and hearing so much more about AI and NVIDIA specifically 10 years later, why did it take a decade? And also, because you had placed those bets, what did the middle of that decade feel like for you? Well, that's a good question. It probably felt like today. To me, there's always some problem, and then there's some reason to be

to be impatient. There's always some reason to be happy about where you are, and there's always many reasons to carry on. And so I think, as I was reflecting a second ago, that sounds like this morning. But I would say that in all things that we pursue, first you have to have core beliefs. You have to reason from your best principles and

and ideally you're reasoning from it from principles of either physics or deep understanding of the industry or deep understanding of the science wherever you're reasoning from you reason from first principles and at some point you have to believe something and if those principles don't change and the assumptions don't change then there's no reason to change your core beliefs and then along the way there's always some

evidence of success and that you're leading in the right direction. Sometimes you go a long time without evidence of success and you might have to course correct a little, but the evidence comes and if you feel like you're going in the right direction, we just keep on going. The question of why did we stay so committed for so long, the answer is actually the opposite. There was no reason to not be committed.

because we believed it and I've believed in NVIDIA for 30 plus years and I'm still here working every single day and there's no fundamental reason for me to change my belief system and I fundamentally believe that the work we're doing in revolutionizing computing is as true today even more true today than it was before and so we'll stick with it until otherwise there's of course

very difficult times along the way. You know, when you're investing in something and nobody else believes in it and costs a lot of money and, you know, maybe investors or others would rather you just keep the profit or, you know, whatever it is, improve the share price or whatever it is. But you have to believe in your future. You have to invest in yourself. And we believe this so deeply that we invested, you know, tens of billions of dollars

before it really happened. It was 10 long years, but it was fun along the way. How would you summarize those core beliefs? What is it that you believe about the way computers should work and what they can do for us that keeps you not only coming through that decade, but also doing what you're doing now, making bets I'm sure you're making for the next few decades?

The first core belief was our first discussion was about accelerated computing, parallel computing versus general purpose computing. We would add two of those processors together and we would do accelerated computing. And I continue to believe that today.

The second was the recognition that these deep learning networks, these DNNs that came to the public during 2012, these deep neural networks have the ability to learn patterns and relationships from a whole bunch of different types of data and that it can learn more and more nuanced features if it could be larger and larger. And it's easier to make them larger and larger to make them deeper and deeper or wider and wider. And so the scalability of the architecture is

is empirically true. The fact that model size and the data size being larger and larger, you can learn more knowledge is also true, empirically true. And so if that's the case, what are the limits? Unless there's a physical limit or an architectural limit or a mathematical limit,

and it was never found and so we believe that you could scale it then the question the only other question is what can you learn from data what can you learn from experience data is basically digital versions of human experience and so what can you learn you obviously can learn object recognition from images you can learn speech from just listening to sound you can learn even languages and vocabulary and syntax and grammar and all just by studying a whole bunch of letters and words so we've now

demonstrated that AI or deep learning has the ability to learn almost any modality of data and it can translate to any modality of data. And so what does that mean? You can go from text to text, right? Summarize a paragraph. You can go from text to text, translate from language to language. You can go from text to images. That's image generation. You can go from images to text. That's captioning. You can even go from

amino acid sequences to protein structures. In the future, you'll go from protein to words. What does this protein do? Or give me an example of a protein that has these properties, identifying a drug target. And so you could just see that all of these problems are around the corner to be solved. You can go from words to video. Why can't you go from words to

to action tokens for a robot. From the computer's perspective, how is it any different? And so it opened up this universe of opportunities and universe of problems that we can go solve. And that gets us quite excited. It feels like we are on the cusp of this truly enormous change. When I think about the next 10 years, I

unlike the last 10 years. I know we've gone through a lot of change already, but I don't think I can predict anymore how I will be using the technology that is currently being developed. That's exactly right. I think the last 10, the reason why you feel that way is the last 10 years was really about the science of AI. The next 10 years, we're going to have plenty of science of AI, but the next 10 years is going to be the application science of AI, the fundamental science versus the application science.

And so the applied research, the application side of AI now becomes, how can I apply AI to digital biology? How can I apply AI to climate technology? How can I apply AI to agriculture, the fishery, to robotics, to transportation, optimizing logistics? How can I apply AI to, you know, teaching? How do I apply AI to, you know, podcasting, right? And so

I'd love to choose a couple of those to help people see how this fundamental change in computing that we've been talking about is actually going to change their experience of their lives, how they're actually going to use technology that is based on everything we just talked about. One of the things that I've now heard you talk a lot about and I have a particular interest in is physical AI, or in other words, robots. My friends. Meaning humanoid robots, but also robots like

self-driving cars and smart buildings or autonomous warehouses autonomous lawn mowers or more from what i understand we might be about to see a huge leap in what all of these robots are capable of because we're changing how we train them up until recently you've either had to train your robot in the real world where it could get damaged or wear down

Or you could get data from fairly limited sources like humans in motion capture suits. But that means that robots aren't getting as many examples as they need to learn more quickly. But now we're starting to train robots in digital worlds, which means way more repetitions a day, way more conditions, learning way faster. So we could be in a big bang moment for robots right now. And NVIDIA is building tools to make that happen.

you have omniverse and my understanding is this is hey developers join us for the biggest ai competition with over ten thousand dollars in cash prizes yep that's right over 3d worlds that help train robotic systems so that they don't need to train in the physical world that's exactly right you just announced cosmos which is ways to make that 3d universe much more realistic so you can get all kinds of different

We're training something on this table, many different kinds of lighting on the table, many different times of day, many different, you know, experiences for the robot to go through so that it can get even more out of Omniverse. As a kid who grew up loving data on Star Trek, Isaac Asimov's books, and just dreaming about a future with robots, how do we get from the robots that we have now to

the future world that you see of robotics. Yeah. Let me use language models, maybe ChatGPT as a reference for understanding omniverse and cosmos. And so first of all, when ChatGPT first came out, it was extraordinary. And it has the ability to do, to basically from your prompt, generate text. However, as amazing as it was, it has the tendency to hallucinate

If it goes on too long or if it pontificates about a topic it is not informed about, it will still do a good job generating plausible answers. It just wasn't grounded in the truth. And so people called it hallucination. And so the next generation, shortly, it had the ability to be conditioned by context and

So you could upload your PDF, and now it's grounded by the PDF. The PDF becomes the ground truth. It could actually look up search, and then the search becomes its ground truth. And between that, it could reason about what is how to produce the answer that you're asking for. And so the first part is a generative AI. And the second part is ground truth. And so now let's come into the physical world, the world model.

We need a foundation model just like we need chat. ChatGPT had a core foundation model. That was the breakthrough. In order for robotics to be smart about the physical world, it has to understand things like gravity, friction, inertia, geometric and spatial awareness. It has to understand that an object is sitting there even when I look away. When I come back, it's still sitting there. Object permanence.

It has to understand cause and effect. If I tip it, it'll fall over. And so these kind of physical common sense, if you will, has to be captured or encoded into a world foundation model so that the AI has world common sense. And so we have to go – somebody has to go create that, and that's what we did with Cosmos. We created a world language model. Just like Chachapiti was a language model, this is a world model.

the second thing we have to go do is we have to do the same thing that we did with pdfs and context and grounding it with ground truth and so the way we augment cosmos with ground truth is with physical simulations because omniverse uses physics simulation which is based on principled solvers the mathematics is newtonian physics is the right it's the math we know okay all of the

The fundamental laws of physics we've understood for a very long time, and it's encoded into, captured into Omniverse. That's why Omniverse is a simulator. And using the simulator to ground or to condition cosmos, we can now generate an infinite number of stories of the future. And they're grounded on physical truth.

Just like between PDF or search plus ChatGPT, we can generate an infinite amount of interesting things, answer a whole bunch of interesting questions. The combination of omniverse plus cosmos, you could do that for the physical world.

So to illustrate this for the audience, if you had a robot in a factory and you wanted to make it learn every route that it could take, instead of manually going through all of those routes, which could take days and could be a lot of wear and tear on the robot, we're now able to simulate all of them digitally in a fraction of the time and in many different situations that the robot might face. It's dark, it's blocked, it's etc. So the robot is now learning much, much faster.

It seems to me like the future might look very different than today. If you play this out 10 years, how do you see people actually interacting with this technology in the near future? Everything that moves will be robotic someday, and it will be soon. The idea that we'll be pushing around a lawnmower is already kind of silly. Maybe people do it because it's fun, but there's no need to. And

every car is going to be robotic human or robots the technology necessary to make it possible is just around the corner and so everything that moves will be robotic and they'll learn how to be a robot in omniverse cosmos and will generate all these plausible physically plausible futures and the robots will learn from them and then they'll come into the physical world and it's exactly the same

A future where you're just surrounded by robots is for certain. And I'm just excited about having my own R2-D2. And, of course, R2-D2 wouldn't be quite the can that it is and roll around. It'll be, you know, R2-D2. It'll probably be a different physical embodiment, but it's always R2. So my R2 is going to go around with me. Sometimes it's in my smart glasses. Sometimes it's in my phone. Sometimes it's in my PC.

it's my car so r2 is with me all the time including you know when i get home you know where i left a physical version of r2 and you know whatever whatever that version happens to be you know we'll interact with r2 and so i think the idea that we'll have our own r2d2 for our entire life and it grows up with us that's a certainty now yeah i think a lot of

news media. When they talk about futures like this, they focus on what could go wrong. And that makes sense. There is a lot that could go wrong. We should talk about what could go wrong so we can keep it from going wrong. That's the approach that we like to take on the show is what are the big challenges so that we can overcome them? What buckets do you think about when you're worrying about this future? Well, there's a whole bunch of the stuff that everybody talks about. Bias or toxicity or just

hallucination you know speaking with great confidence about something it knows nothing about and as a result we rely on that information generating that's a version of generating fake information fake news or fake images or whatever it is of course impersonation it does such a good job pretending to be a human it could do an incredibly good job pretending to be a specific human and so

So the spectrum of areas that we have to be concerned about is fairly clear. And there's a lot of people who are working on it. Some of the stuff related to AI safety requires deep research and deep engineering. And that's simply, it wants to do the right thing, it just didn't perform it right, and as a result, hurt somebody. For example...

of the video. And then lastly,

You know, what happens if the AI wants to do a good job, but the system failed, meaning the AI wanted to stop something from happening, and it turned out just when it wanted to do it, the machine broke down. And so this is no different than a flight computer inside a plane having three versions of them. And then so there's triple redundancy inside the system, inside autopilot's.

And then you have two pilots. And then you have air traffic control. And then you have other pilots watching out for these pilots. And so the AI safety systems has to be architected as a community such that these AIs, one, work function properly. When they don't function properly, they don't put people in harm's way. And that there's sufficiently safety and security systems all around them

to make sure that we keep AI safe. And so the spectrum of conversation is gigantic. And we have to take the parts apart and build them as engineers. One of the incredible things about this moment that we're in right now is that we no longer have a lot of the technological limits that we had in a world of CPUs and sequential processing. And we've unlocked

not only a new way to do computing, but also a way to continue to improve. Parallel processing has a different kind of physics to it than the improvements that we were able to make on CPUs. I'm curious, what are the scientific or technological limitations that we face now in the current world that you're thinking a lot about? Well, everything in the end

is about how much work you can get done within the limitations of the energy that you have and so that's a physical limit and the laws of physics about transporting sick information and transporting bits flipping bits and transporting bits at the end of the day the energy it takes to do that

limits what we can get done and the amount of energy that we have limits what we can get done. We're far from having any fundamental limits that keep us from advancing. In the meantime, we seek to build better, more energy-efficient computers. This little computer, the big version of it was $250,000. Yeah, yeah. Those little baby digits, yeah. This is an AI supercomputer.

the version that I delivered this is just a prototype so it's a mock-up and so the the very first version was DJX1 I delivered to OpenAI in 2016 and that was $250,000 10,000 times more power more energy necessary than this version and this version has six times more performance Density is a tool to record your DJ set and master it after Density connects to DJ decks to record a

I know, it's incredible. We're in a whole new world. And it's only since 2016. And so eight years later, we've increased the energy efficiency of computing by 10,000 times. And imagine if we became 10,000 times more energy efficient, or if a car was 10,000 times more energy efficient, or electric light bulb was 10,000 times more energy efficient. Our light bulb would be right now

instead of 100 watts, 10,000 times less, producing the same illumination. And so the energy efficiency of computing, particularly for AI computing that we've been working on, has advanced incredibly. And that's essential because we want to create more intelligent systems and we want to use more computation to be smarter. And so energy efficiency to do the work is our number one priority.

When I was preparing for this interview, I spoke to a lot of my engineering friends. And this is a question that they really wanted me to ask. So you're really speaking to your people here. You've shown a value of increasing accessibility and abstraction with CUDA and allowing more people to use more computing power in all kinds of other ways. As applications of technology get more specific, I'm thinking of transformers in AI, for example.

For the audience, a transformer is a very popular, more recent structure of AI that's now used in a huge number of the tools that you've seen. The reason that they're popular is because transformers are structured in a way that helps them pay attention to key bits of information and give much better results. You could build chips that are perfectly suited for just one kind of AI model. But if you do that, then you're making them less able to do other things.

So as these specific structures or architectures of AI get more popular, my understanding is there's a debate between how much you place these bets on burning them into the chip or designing hardware that is very specific to a certain task versus staying more general. And so my question is, how do you make those bets? How do you think about whether the solution is a car that could go anywhere or it's really optimizing a train to go from A to B?

You're making bets with huge stakes, and I'm curious how you think about that. Yeah, and that now comes back to exactly your question, what are your core beliefs? And the core belief, either one, that Transformer is the last...

that any researcher will ever discover again, or that Transformers is a stepping stone towards evolutions of Transformers that are barely recognizable as a Transformer years from now.

and we believe the latter and the reason for that is because you just have to go back in history and ask yourself in the world of computer algorithms in the world of software in the world of engineering and innovation has one idea stayed along that long and the answer is no and so that's kind of the beauty that's in fact the essential beauty of a computer that it's able to do something today

that no one even imagined possible 10 years ago. And if you would have turned that computer 10 years ago into a microwave, then why would the applications keep coming? And so we believe in the richness of innovation and the richness of invention, and we want to create an architecture that let inventors and innovators and software programmers and AI researchers swim in the soup and come up with some amazing ideas.

look at transformers the fundamental characteristic of a transformer is this idea called a tension mechanism and it basically says the transformer is going to understand the meaning and the relevance of every single word with every other word so if you had 10 words it has to figure out the relationship across 10 of them but if you have a hundred thousand words or if your context is now as large as read a pdf and that can't read a whole bunch of pdfs and the context window is now like a million tokens

processing all of it across all of it is just impossible. And so the way you solve that problem is there are all kinds of new ideas and flash attention or hierarchical attention or all the wave attention I just read about the other day. The number of different types of attention mechanisms that have been invented since the transformer is quite extraordinary. And so I think that that's going to continue.

And we believe it's going to continue and that computer science hasn't ended and that AI research has not all given up. We haven't given up anyhow. And that having a computer that enables the flexibility of research and innovation and new ideas is fundamentally the most important thing. One of the things that I am just so curious about

You design the chips. There are companies that assemble the chips. There are companies that design hardware to make it possible to work at nanometer scale. When you're designing tools like this, how do you think about design in the context of what's physically possible right now to make? What are the things that you're thinking about with sort of pushing that limit today? The way we do it is even though we...

have things made like for example our chips are made by TSMC even though we have them made by TSMC we assume that we need to have the deep expertise that TSMC has and so we have people in our company who are incredibly good at semiconductor physics so that we have a feeling for we have an intuition for what are the limits of what today's semiconductor physics can do

And then we work very closely with them to discover the limits because we're trying to push the limits. And so we'll discover the limits together. We do the same thing in system engineering and cooling systems. It turns out plumbing is really important to us because of liquid cooling and maybe fans are really important to us because of air cooling. And we're trying to design these fans in a way almost like they're aerodynamically sound so that we could pass the highest volume of air, make the least amount of noise. So we have aerodynamics engineers in our company and

And so even though we don't make them, we design them and we have the deep expertise of knowing how to have them made. And from that, we try to push the limits. One of the themes of this conversation is that you are a person who makes big bets on the future. And time and time again, you've been right about those bets. We've talked about

GPUs, we've talked about CUDA, we've talked about bets you've made in AI. Self-driving cars, and we're going to be right on robotics. This is my question. What are the bets that you're making now? The latest bet, of course, we just described at the CES, and I'm very proud of it, and I'm very excited about it, is the fusion of Omniverse and Cosmos, so that we have this new type of generative world generation system, this multiverse system.

generation system. I think that's going to be profoundly important in the future of robotics and physical systems. Of course, the work that we're doing with human or robots, developing the tooling systems and the training systems and the human demonstration systems and all of this stuff that you've already mentioned,

We're just seeing the beginnings of that work and I think the next five years are going to be very interesting in the world of human robotics. Of course, the work that we're doing in digital biology so that we can understand the language of molecules and understand the language of cells and just as we understand the language of physics and the physical world, we'd like to understand the language of the human body and understand the language of biology. And so if we can learn that

and we can predict it, then all of a sudden our ability to have a digital twin of the human is plausible. And so I'm very excited about that work.

I love the work that we're doing in climate science and be able to, from weather predictions, understand and predict the high-resolution regional climates, the weather patterns within a kilometer above your head, that we can somehow predict that with great accuracy. Its implications is really quite profound. And so the number of things that we're working on is really cool.

we're fortunate that we've created this instrument that is a time machine and we need time machines in all these areas that we just talked about so that we can see the future and if we can see the future and we can predict the future

then we have a better chance of making that future the best version of it. And that's the reason why scientists want to predict the future. That's the reason why we try to predict the future in everything that we try to design so that we can optimize for the best version. So if someone is watching this and maybe they came into this video knowing that NVIDIA is an incredibly important company but not fully understanding why or how it might affect their life,

And they're now hopefully better understanding a big shift that we've gone through over the last few decades in computing, this very exciting, very sort of strange moment that we're in right now where we're sort of on the precipice of so many different things. If they would like to be able to look into the future a little bit, how would you advise them to prepare or to think about this moment that they're in personally with respect to how these tools are actually going to affect them?

Hey, still thinking about joining the 30-day Python using AI Bootcamp by AI for Techies? So what if in 30 days you could go from knowing nothing about Python to actually building apps and websites from scratch? With just 45 minutes a day and a step-by-step module guiding you, it's simple even for complete beginners. Plus, by the end of the bootcamp, you'll receive a certification, which is recognized by top companies that hire Python folks with AI skills. You'll be learning from the best industry experts,

Among them would also be IIT alumnus who have helped thousands of people like you start their tech career. And it works because folks just like you inexperienced in the tech field start from here and then land up 15 to 30 LPA jobs just after completing this bootcamp. Python combined with AI skills gives you a good chance against experienced folks to fight for these top packages. And the best part, it's just for Rs. 199. But you need to hurry up because these seats fill up pretty fast. So book yours now.

well there are several ways to reason about about the future that we're creating one way to reason about it is suppose the work that you do continues to be important but but the effort by which you do it went from you know being a week long to almost instantaneous you know

that the effort of drudgery basically goes to zero. What is the implication of that? This is very similar to what would change if all of a sudden we had highways in this country. And that kind of happened in the last industrial revolution. All of a sudden we have interstate highways. And when you have interstate highways, what happens? Well, suburbs start to be created and all of a sudden...

distribution of goods from east to west is no longer a concern and all of a sudden gas stations start cropping up on highways and fast food restaurants show up and you know some motels show up because people you know traveling across the state across the country just wanted to stay somewhere for a few hours or overnight and so all of a sudden new economies new capabilities new economies

What would happen if video conference made it possible for us to see each other without having to travel anymore? All of a sudden, it's actually okay to work further away from home and from work and live further away. And so you ask yourself kind of these questions. What would happen if I have a

software programmer with me all the time and whatever it is I can dream up the software programmer could write for me you know what what would that do

what would happen if I just had a seed of an idea and I rough it out and all of a sudden a prototype of a production was put in front of me and how would that change my life and how would that change my opportunity and what does it free me to be able to do and so on and so forth so I think that the next decade intelligence not for everything

but for some things would basically become superhuman. And so I can tell you exactly what that feels like. I'm surrounded by superhuman people, superintelligence, from my perspective, because they're the best in the world at what they do, and they do what they do way better than I can do it. And I'm surrounded by thousands of them.

and yet it never one day caused me to think all of a sudden I'm no longer necessary it actually empowers me and gives me the confidence to go tackle more and more ambitious things and so suppose now everybody is surrounded by these super AIs that are very good at specific things or good at some of the things

What would that make you feel? Well, it's going to empower you. It's going to make you feel confident. And I'm pretty sure you probably use ChatGPT and AI. I feel more empowered today, more confident to learn something today. The knowledge of almost any particular field, the barriers to that understanding it has been reduced.

I have a personal tutor with me all of the time. And so I think that that feeling should be universal. And if there's one thing that I would encourage everybody to do is to go get yourself an AI tutor right away. And that AI tutor could, of course, just teach you things, anything you like, help you program, help you write, help you analyze, help you think, help you reason. You know, all of those things

it's going to really make you feel empowered. And I think that's going to be our future. We're going to become superhumans, not because we have super... We're going to become superhumans because we have super AIs. Could you tell us a little bit about each of these objects? This is a new GeForce graphics card. Can I touch it? Yes. And this is the RTX 50 series.

It is essentially a supercomputer that you put into your PC. And we use it for gaming. Of course, people today use it for design and creative arts. And it does amazing AI. And the real breakthrough here, and this is truly an amazing thing, GeForce enabled AI and enabled Jeff Hinton, Ilya Suskovor, and Alex Koshevsky to be able to train AlexNet. We discovered AI.

and we advanced AI. Then AI came back to GeForce to help computer graphics. And so here's the amazing thing. Out of 8 million pixels or so in a 4K display, we are computing, we're processing only 500,000 of them. The rest of them, we use AI to predict. The AI guessed it. And yet the image is perfect. We inform it by the 500,000 pixels that we computed

and we ray traced every single one and it's all beautiful. It's perfect. And then we tell the AI, if these are the 500,000 perfect pixels in the screen, what are the other 8 million? And it goes, it fills in the rest of the screen and it's perfect. And if you only have to do fewer pixels, are you able to invest more in doing that because you have fewer to do? So then the quality is better. So the

extrapolation that the AI does exactly because whatever computing whatever attention you have whatever resources you have you can place it into 500,000 pixels now this is a perfect example of why AI is going to make us all superhuman because all of the other things that it can do it'll do for us allows us to take our time and energy and focus it on the really really valuable things that we do and so we'll take our own resource which is you know

energy intensive attention intensive and well dedicated to the few hundred thousand pixels and use AI to super res it up res it you know to everything else and so this this graphics card is now powered mostly by AI and the computer graphics technology inside is incredible as well and then this next one as I mentioned earlier in 2016 I built the first one for AI researchers and we delivered the first one to OpenAI and

And Elon was there to receive it. And this version, I built a mini version. And the reason for that is because AI has now gone from AI researchers to every engineer, every student, every AI scientist. And AI is going to be everywhere. And so instead of these $250,000 versions, we're going to make these $3,000 versions. And schools can have them. Students can have them. And you set it next to your

PC or Mac and all of a sudden you have your own AI supercomputer. Run Windows, Linux and more on your Mac with unmatched performance and you could develop and build AIs, build your own AI, build your own R2D2. What do you feel like is important for this audience to know that I haven't asked? One of the most important things I would advise is

For example, if I were a student today, the first thing I would do is to learn AI. How do I learn to interact with ChatGPT? How do I learn to interact with Gemini Pro? And how do I learn to interact with Grok? Learning how to interact with AI is not unlike being someone who is really good at asking questions. You're incredibly good at asking questions, and prompting AI is very similar.

You can't just randomly ask a bunch of questions. And so asking AI to be an assistant to you requires some expertise in artistry and how to prompt it. And so if I were a student today, irrespective of whether it's for math or for science or chemistry or biology, or it doesn't matter what field of science I'm going to go into or what profession I am, I'm going to ask myself, how can I use AI to do my job better?

If I want to be a lawyer, how can I use AI to be a better lawyer? If I want to be a better doctor, how can I use AI to be a better doctor? If I want to be a chemist, how do I use AI to be a better chemist? If I want to be a biologist, how do I use AI to be a better biologist? That question should be persistent across everybody. And just as my generation grew up as the first generation that has to ask ourselves, how can we use computers to do our jobs better? Hmm.

the generation before us had no computers my generation was the first generation that had to ask the question how do I use computers to do my job better remember I came into the industry before Windows 95 1984 there were no computers in offices and after that shortly after that computers started to emerge and so we had to ask ourselves how do we use computers to do our jobs better the next generation

doesn't have to ask that question but it has to ask the obviously next question how can i use ai to do my job better that is start and finish i think for everybody it's a really exciting and scary and therefore worthwhile question i think for everyone i think it's it's going to be incredibly fun ai is obviously a word that people are just learning now but it's just you know what would

It's made your computer so much more accessible. It is easier to prompt ChatGPT to ask it anything you like than to go do the research yourself. And so we've lowered the barrier of understanding. We've lowered the barrier of knowledge. We've lowered the barrier of intelligence. And everybody really ought to just go try it. The thing that's really crazy is if I put a computer in front of somebody and they've never used a computer, there is no chance they're going to learn that

computer in a day. There's just no chance. Somebody really has to show it to you. And yet with ChatGPT, if you don't know how to use it, all you have to do is type in, I don't know how to use ChatGPT. Tell me. And it would come back and give you some examples. And so that's the amazing thing. The amazing thing about intelligence is it'll help you along the way and make you superhuman along the way.

All right, I have one more question if you have a second. This is not something that I planned to ask you, but on the way here, I'm a little bit afraid of planes, which is not my most reasonable quality. And the flight here was a little bit bumpy. Very bumpy. And I'm sitting there, and it's moving, and I'm thinking about what they're going to say at my funeral. And after...

She asks good questions. That's what the tombstone is going to say. I hope so. And after I loved my husband and my friends and my family, the thing that I hoped that they would talk about was optimism. I hoped that they would recognize what I'm trying to do here. And I'm very curious for you. You've been doing this a long time. It feels like there's so much that you've described in this vision ahead. What would the theme be that you would want people to say?

about what you're trying to do. Very simply, they made an extraordinary impact. And I think that we're fortunate because of some core beliefs a long time ago and sticking with those core beliefs and building upon them. We found ourselves

Today, being one of the many most important and consequential technology companies in the world and potentially ever. And so we take that responsibility very seriously. We work hard to make sure that the capabilities that we've created are available to

large companies as well as individual researchers and developers across every field of science no matter profitable or not big or small famous or otherwise and it's because of this understanding of the consequential work that we're doing and the potential impact it has on so many people that we want to make this capability as

as pervasively as possible. And I do think that when we look back in a few years, and I do hope that what the next generation realized

is as a they well first of all they're going to know us because of all the you know gaming technology we create I do think that we'll look back and the whole field of digital biology and life sciences has been transformed our whole understanding of of material sciences has completely been revolutionized that robots are helping us do dangerous and mundane things all over the place and

that if we wanted to drive, we can drive, but otherwise, you know, take a nap or enjoy your car like it's a home theater of yours. You know, read from work to home. And at that point, you're hoping that you live far away and so you could be in a car for longer. And, you know, and you look back and you realize that there's this company almost at the epicenter of all of that.

and happens to be the company that you grew up playing games with. And I hope that that to be what the next generation learn. Thank you so much for your time. I enjoyed it. Thank you. I'm glad. Jensen, this is such an honor. Thank you for being here. I'm delighted to be here. Thank you. In honor of your return to Stanford, I decided we'd start talking about

the time when you first left. You joined LSI Logic, and that was one of the most exciting companies at the time. You're building a phenomenal reputation with some of the biggest names in tech, and yet you decide to leave to become a founder. What motivated you? Chris and Curtis. Chris and Curtis, I was an engineer at LSI Logic, and Chris and Curtis were at Sun.

And I was working with some of the brightest minds in computer science of all time, including Andy Bechtolsheim and others, building workstations and graphics workstations and so on and so forth. And Chris and Curtis said one day that they'd like to leave Sun, and they'd like me to go figure out what they're going to go leave for.

and I had a great job but they insisted that I figure out with them how to build a company so we hung out at Denny's whenever they dropped by which is by the way my alma mater my first company my first job before CEO was a dishwasher and I did that very well

and so anyways, we got together

and it was during the microprocessor revolution this is 1993 and 1992 when we were getting together the PC revolution was just getting going you know that Windows 95 obviously which is the revolutionary version of Windows didn't even come to the market yet and Pentium wasn't even announced yet and this is all before right before the PC revolution and it was pretty clear that the microprocessor was going to be very important and we

we thought, you know, why don't we build a company to go solve problems that a normal computer that is powered by general processing can't? And so that became the company's mission, to go build a computer, the type of computers and solve problems that normal computers can't. And to this day, we're focused on that. And if you look at all the problems in the markets that we opened up as a result, it's, you know, things like

computational drug design, weather simulation, materials design. These are all things that we're really, really proud of. Robotics, self-driving cars, autonomous software we call artificial intelligence. And then, of course, we drove the technology so hard that eventually the computational cost went to approximately zero and it enabled

enabled a whole new way of developing software where the computer runs the software itself, artificial intelligence as we know it today. So that was it. That was the journey. Yeah. Thank you all for coming. Well, these applications are on all of our minds today. But back then, the CEO of LSI Logic convinced his biggest investor, Don Valentine, to meet with you. He's obviously the founder of Sequoia. Yeah.

Now I can see a lot of founders here edging forward in anticipation, but how did you convince the most sought-after investor in Silicon Valley to invest in a team of first-time founders building a new product for a market that doesn't even exist? I didn't know how to write a business plan. So I went to a bookstore, and back then there were bookstores.

in the business book section, there was this book, and it was written by somebody I knew, Gordon Bell. And this book, I should go find it again, but it's a very large book. And the book says, How to Write a Business Plan. And that was a highly specific title for a very niche market. And it seems like he wrote it for 14 people, and I was one of them. And so I bought the book.

I should have known right away that that was a bad idea because, you know, Gordon is super, super smart and super smart people have a lot to say. And they want to, you know, I'm pretty sure Gordon wants to teach me how to write a business plan completely. And so I picked up this book. It's like 450 pages long.

Well, I never got through it. Not even close. I flipped through it a few pages, and I go, you know what? By the time I'm done reading this thing, I'll be out of business. I'll be out of money. And Lori and I only had about six months in the bank. We had already Spencer at Madison and a dog. So the five of us had to live off of whatever money we had in the bank. And so I didn't have much time.