Attaining The Third Dimension

Three-dimensional (3D) computer imagery is now central to the global film industry, not to mention games, design, and the emerging fields of virtual and augmented reality. But when ACM A.M. Turing Award recipients Ed Catmull and Pat Hanrahan began working together at Pixar, the goal of being able to produce truly realistic images on a computer seemed remote. Here, Ed and Pat discuss what it took to overcome the challenge—and how their work on Pixar’s pioneering image system, RenderMan, came about.

Pixar was formed in 1986, when Steve Jobs helped spin off the computer division from Lucasfilm, where Ed had been working since 1979. Pat, you joined shortly thereafter.

HANRAHAN: Within days, I think.

Your work on what became Render-Man traces back to the Lucasfilm days. Ed, can you talk about that?

CATMULL: George Lucas hired me to bring technology into the film industry, which included digital audio and video editing and, later, games. While we were working on modeling and animation, we had to solve a number of problems to do with complexity and motion blur. Rob Cook, Loren Carpenter, and I also set a goal of being able to create an image containing 80 million micropolygons. It was a ridiculously high goal—the state of the art for polygons at the time was about 40,000—but we wanted to force ourselves to think in a different way. In the end, a suggestion was made by an engineer named Rodney Stock. Rob Cook ran with it, and it became stochastic sampling, which today is the way the entire industry works.

Figure. Ed Catmull and Pat Hanrahan

This led to the creation of Lucasfilm’s rendering software, Reyes, which was RenderMan’s predecessor and stood for “renders everything you ever saw.”

CATMULL: Around the same time as we launched Pixar, Silicon Graphics was becoming a significant player in making chips and computers, and we and several other companies decided to come up with an interface that would give us a common way of describing objects, lighting, and shading. Pat was asked to be the architect because he’s so brilliant and because he understood languages. He took some of the concepts of shading and turned them into a shading language. He made all the final decisions on the software design, and once it was done, we renamed it RenderMan.

HANRAHAN: My thinking was motivated by a language called PostScript, which was the foundation of the original laser printer and which had revolutionized desktop publishing. We thought that if we could come up with a standard like PostScript for 3D rendering, we could catalyze a whole industry—not just movie-making, but product design, architecture, and so on.

CATMULL: The intent was to make it an industry standard. The concept of open software, where everybody can make changes, wasn’t really around at that time. For us, it was like, “let’s make it an open standard, and then anybody can implement the standard.” Steve Jobs didn’t object, because he understood that our real goal was to attract the best people.

To say that Pixar attracted top talent seems like an understatement.

HANRAHAN: Pixar was the place. There was just such a concentration of great people. But to come back to Reyes for a moment, I think everybody up to that point had been concentrated on what it would take to do real-time graphics. Obviously, games had to be real-time, and if you’re building a CAD program, you need to be able to interact with the model. What was amazing about Pixar is that people took this long view of the problem. There was this picture they had made of the road to Point Reyes [a national seashore and park preserve in Marin County, CA]. The idea, as I understood it, was that if we just went outside and looked around, we saw this unbelievable landscape. Suppose you wanted to make a movie that was located there; what would it take to do that? It was this amazing moonshot idea.

CATMULL: We picked a hard goal. In our case, it was the replication of reality. And now things look so realistic that I often can’t tell in films, and I’ve got a very good eye.

Pixar’s culture was also unique in giving artists and technologists equal footing.

CATMULL: From the beginning, we were careful to make sure that they looked at each other as peers. Each film had challenges, and the technical people loved the challenges. They couldn’t do everything, but they did what they figured they could within the constraints. Then, in trying to solve those problems, they often came up with something that the artists hadn’t thought of. It was a very positive back-and-forth, where the artists would look at something and say, “Oh, I’d like to put this in a movie,” and ask for more. And the technical people would say, “Oh, that sounds exciting. I’ll try to do that.” This dynamic led to a stunning increase in the visual complexity of the films, while at the same time everybody was clear that the goal was to make a good movie and not let anything else get in the way.

“We picked a hard goal. In our case, it was the replication of reality. And now things look so realistic that I often can’t tell in films, and I have a pretty good eye.”

HANRAHAN: A lot of technologists forget the context in which they’re operating. I think we all had a great appreciation for art and the importance of movies in our culture. And that’s why we didn’t get stuck on one specific technical feature that magically solved all the problems. Because it’s not just about creating photorealistic images, it’s about telling stories and depicting things we care about.

CATMULL: It’s about communicating. That’s why ARPA [the Advanced Research Projects Agency of the U.S. Defense Department] funded graphics in the first place. I think the relationship between computer graphics and the rest of the computing industry is poorly understood. We were a small industry, but we published at SIGGRAPH, and then the engineers at companies like NVIDIA and AMD went to SIGGRAPH and used those ideas to keep advancing their chips, because the game industry is so big that they could afford to. Of course, those chips also went into boards in the high-end workstations that academia and entertainment were using, which meant they could continue to advance the state of the art with new algorithms and then publish them at SIGGRAPH. We had this virtuous cycle.

Eventually, GPUs became more powerful than CPUs.

CATMULL: Pat was one of the first to realize that this immense computing power could be used in other areas. The concept of neural networks had been around a long time, but there just wasn’t enough processing power to do it. Suddenly, as a result of something happening in this cycle with the chip companies, the game industry, academia, entertainment, and SIGGRAPH, there was enough to power a whole new field.

HANRAHAN: Remember when Ed said we needed to do 80 million micro-polygons? We had a budget of 400 floating point operations per pixel when we were building Render-Man. We were always trying to get more compute cycles. When GPUs came along, and they literally had thousands of core machines instead of one core machine, I immediately realized that this was going to be the key to the future.

Ed, you’ve thought about how to create a culture that outlives you for a while. What’s it like now that you’re retired, and you’re actually watching it happen?

CATMULL: I’d put my successors in place, and it was time, and it felt like the fulfillment of an arc. I fully expect that the people at Pixar and Disney Animation will continue to do things that I can’t even foresee. That’s the nature of creativity: you’ve got a glimmer of something, but you can’t completely see it. And when you get surprised, that’s when it’s gratifying.

Pat, are you ready to retire?

HANRAHAN: Well, yes and no. When Salesforce acquired Tableau, I retired as Chief Scientist. My full-time job all along has been teaching and doing research at Stanford. If you want to maintain a good attitude in the world, just hang out with students. I teach systems programming to freshmen and sophomores these days, and they have such a positive attitude, are eager to learn, and work so hard. I love going into school and being surrounded by my students.