The Reality of Simulated Actors

Is there some elemental quality that is exclusively human? Must actors portray it for their characters to be believable? Or can they be replaced with digital simulations? Should they therefore fear for their jobs? What do actors and animators have in common? How will they interface with their on-screen avatars? Can an actor-animator collaboration win an Oscar?

Almost two years ago I made my first attempt at predicting when, if ever, we would be able to replace human actors with digital simulations [2]. Here I revisit and refine those predictions. The gist is that we must separate acting from (the appearance of) actors. I therefore expect that (1) we will not replace acting, nor therefore actors, in any known way in any known timeframe. But (2) we may well replace the appearance of actors in my lifetime. I have in mind a parallel to the goal my colleagues and I once had, starting about 1974, to realize the first completely digital feature film. That took 20 years, far longer than originally guessed, and the result, Toy Story [Pixar/Disney 1994], was a cartoon. I now predict it will take an additional 20 years to produce the first completely digital “live-action” motion picture, including by definition fully realized human beings. Explicitly, this will be demonstrated by the complete replacement of the appearance of a lead actor in a feature-length motion picture, including comparable amounts of screentime and spoken dialogue, and comparable numbers of close-up and medium shots. Moreover, I predict that the screen representation, including voice, will be “driven” by at least one accomplished human actor, a member of the Screen Actors Guild.

Computer graphics colleagues have questioned my time estimate as being too conservative, but I stick to it because solving the problem is more difficult than commonly understood; at the very minimum, it will require a Moore’s Law increase in computing power of four orders of magnitude, as promised by 20 years. Using the “10×in 5” formulation—anything good about computers gets 10 times better every five years—we will probably need 10,000 times more computing power within a typical movie budget than is currently available. Cameos of increasing length will appear in the meantime, as the techniques of human representation are mastered and Moore’s Law yields sufficiently cheap cycles.

Consciousness

Prediction 1 follows from the lack of any successful theory of consciousness. That is, there is no known way of describing how to make a machine, including ourselves, conscious. We only know that we, at least, are conscious, so it will probably be possible to understand consciousness someday, but that is a statement of my personal religion, not a scientific prediction. The best current theory of consciousness is that of neuroscientist Antonio Damasio [1], but even he does not attempt to explain a fundamental aspect of the problem, the “qualia problem”—a tiny example of which is how we derive “blueness” from light of certain frequencies exciting the electrochemical system of our retinas and the brain behind them. His theory posits that consciousness and emotion are inseparable, and that consciousness is in fact a feeling, based on his clinical observations that the removal of brain parts causing emotions to cease also causes the cessation of consciousness. It is impossible to simulate acting if we cannot understand consciousness and emotion. Therefore, acting requires actors; there is no known way around them. So the remainder of this essay concerns Prediction 2—replacing the screen appearance of actors.

A key point is that animators are actors, though silent ones. When I met animator Frank Thomas, one of the “grand old men” at Disney, several decades ago, he was acting into a mirror to inspire his animation of the character Sir Hiss in Robin Hood [Disney, 1973]. Today Pixar hires animators based on their acting ability. Animation has always separated acting from the appearance of the actor. We do not tend to think of animators as actors because, until now, their screen appearance has always been a cartoon. Their screen appearance—or avatar, to borrow a term from the Internet—has been an object or a comic drawing of a simple human or animal.

Actors are animators. Human actors can be thought of as animating their own bodies as their screen appearance, or avatar, and a voice, of course. The really good ones convince us that the same body, their own, is that of many different people. They seldom change gender and cannot be animals or objects. A favorite movie, Being John Malkovich [USA Films, 1999], explored the possibilities available when some other actor “drives” John Malkovich’s body. In one scene, a woman driving his body has sex with a woman. Is that hetero- or homosexual? By the way, it is revealing to see what people choose as their Internet avatars: sometimes themselves, but also animals, objects, and, surprisingly often, a human of the opposite sex.

Two major problems confront us: animators have to be given realistic human models to animate; and actors, freed from their bodies, have to be given effective methods of driving these models, or avatars. That is, representing the appearance of reality in a convincing way is a problem, the “model problem,” and interfacing to such a model is a further problem, the “control problem.” Solutions to both will require major increases in computational power.

Thousands of Processors

The computation of a single frame of a major digital motion picture is much more computer-intensive than is commonly understood. Each frame of Toy Story took an average of seven hours to compute, and each frame of Toy Story 2 [Pixar/Disney 1998] four years later took about five. The best digital movies today—clearly and designedly cartoons—require some of the largest computations on Earth, several thousand processors running around the clock for a couple of years.

My colleagues and I have long considered 80 million polygons (megapolys) per frame the threshold of “reality,” meaning a sufficiently rich approximation that audiences cease to be concerned about its authenticity. In another five to 10 years we will see 80 megapolys per frame as an average frame complexity. But that is only a measure of satisfactory stills.

A successful representation of a human actor must move accurately, too. This is where the control problem confronts us. Woody in Toy Story had about 100 controls for his face alone. Al (the scheming proprietor of Al’s Toy Barn) in Toy Story 2 had about 1,000. Each is clearly a cartoon. It is conceivable that a satisfactory human actor’s face might require tens of thousands of controls. Presented raw to an animator/actor, so many controls would be overwhelming. The control problem involves presenting artists with a sufficiently rich set of controls in an intuitive way. I suspect the problem is quite difficult. I hope to be surprised with solutions that bypass the old notions of building a model, driving it directly with animation variables; I have seen several prototype technologies offering such shortcuts, but they are not yet perfected.

Collaborating with Animators

How will traditional actors adapt to this new world? One obvious way is by collaborating with animators, the other kind of actor. They already do this in the voices for cartoon characters. To be clear, the actors of these characters are their animators, not the highly touted voice stars. Yet the voices—absolutely crucial to the believability of a character—affect the presentations by the animators, who are inspired by the gestures of the voice-actors when creating those of the corresponding avatars. Human actors will continue to do the voices. Actor-animator collaborations will surely play an increasingly important role in the future.

Another possibility is that some animators will cross the voice boundary and come into their own as they drive, or animate, increasingly realistic avatars. It has been suggested that awards be given to animator-avatar combinations as they are now given to actors driving their own bodies.

One thing is certain: Human actors will not go away in that future.