Adding Depth to Mobile Imaging

Smartphone cameras have improved to the point where they have become the only camera many people need, but whether still or video, the images they produce are still flat. Google’s Advanced Technology and Products Team (ATAP) hopes to change that, with a project that will take smartphone imaging into a new dimension—literally.

Enter Project Tango, aimed at enabling mobile devices to combine 3D sensing with motion detection to create 3D representations of interior spaces. Google has been working with software and hardware partners around the world, some of whom were ready to demonstrate their capabilities at the Google I/O Conference last month in San Francisco.

Project Tango is probably better thought of as a platform, rather than as a product. Various partners are contributing different pieces required to make the platform work, from image sensors to image processors to software tools that can make use of the captured images.

One partner is Paracosm, whose lead engineer on the project, Quinn Martin, explains that Project Tango started within Motorola Mobility, then part of Google. When Lenovo acquired the Mobility division, Google kept ATAP, and the project took a different shape. Rather than hardware, Martin says, “now they’re working more on a sort of enabling technology, or something similar to Android certification to say that a given device from a third party meets all the specifications to do Tango-type experiences.”

Gary Brown, vice president of marketing at Movidius, another partner, says, “For me, the Tango project puts the technology out there so that all the commercial phone makers and commercial software makers can have it. Google wanted to look at what you can do years from now and see if they could make a reference design.”

What Project Tango actually is, then, won’t be clear until other companies start building tools on top of it. To speed that along, Google offers a development kit in the form of a 7-inch tablet with 4 GB of RAM and 128 GB of storage, a motion tracking camera, and integrated depth sensing via an infrared laser in combination with an infrared-plus-RGB sensor (the Project Tango Tablet Development Kit is available for $1,024). The motion-tracking camera has a wide-angle lens that helps with area recognition; “with a very wide field of view,” explains Martin, “you have many more objects likely to be in the scene and more points to recognize.”

Movidius provides low-power processors and software that enable the 3D image processing to be done in a mobile device. “The new platform for computer vision is about transcending beyond taking good pictures,” says Farshid Sabet, Movidius’ senior vice president and chief business officer. “But computing vision applications are very compute-intensive, so if you want to do it in a portable, consumer application, it becomes extremely challenging. Movidius brings the ability to make these applications work.”

Paracosm’s contribution is on the application end. “We’ve never been a hardware company,” says Martin. “Our technology started out as purely a cloud-based service for taking in data, making 3D reconstructions, and allowing them to be accessed from many different devices. The 3D model enables a lot of interesting things, but when we combine our models with the Project Tango platform, it becomes a lot more powerful. You can have a 3D model that gives you an understanding of the structure of the scene, and then you have the area recognition that tells where you are in that 3D model at the time.”

Both Sabet and Martin talk about the potential for mapping the interior spaces of the world as thoroughly, and in as much detail, as Google has mapped the exterior spaces.

Martin describes two “forks” of development, one involving robot navigation, with industrial, military, and scientific applications. “The other big fork is augmented reality,” he says, “where you’re using the model to make it convincing that virtual objects are really in the world with you.” He suggests future first-person shooter games could be set in your own house, with zombies chasing you from room to room, since the software would know how the interior structure fit together.

Sabet points to Matterport, which is working on an application that would, for example, let a real estate agent create a 3D model of a house that clients could walk through, place virtual furniture in, and so on.

Both men also suggest shopping applications, whereby stores could create a map showing the location of objects within a store, and your phone could help you find the right shelf for the product you’re looking for, or display coupons on top of items at which you point the phone’s camera (or exactly where you’re looking, if you’re wearing a headset like Google Glass).

While Apple has not spoken about working on the same sort of technology, both Sabet and Martin point out that last year Apple acquired PrimeSense, the 3D sensor company behind Microsoft’s Kinect. (Paracosm’s first mobile effort relied on a PrimeSense camera strapped to a Microsoft Surface tablet.)

There have been no announcements of Project Tango-enabled devices for sale yet, but we may not have long to wait. According to Movidius’ Brown, “It’ll take a while for the phones to get made and the applications to get tuned, but I think within the next few years the camera on your phone is going to be way more than just a camera.”

Logan Kugler is a freelance technology writer based in Clearwater, FL. He has written for over 60 major publications.