fabric | rblg

Pixel perfect: Using aerial photos, image-processing software created this 3-D model of San Francisco, accurate to 15 centimeters. Credit: C3 Technologies

Technology originally developed to help missiles home in on targets has been adapted to create 3-D color models of cityscapes that capture the shapes of buildings to a resolution of 15 centimeters or less. Image-processing software distills the models from aerial photos captured by custom packages of multiple cameras.

The developer is C3 Technologies, a spinoff from Swedish aerospace company Saab. C3 is building a store of eye-popping 3-D models of major cities to license to others for mapping and other applications. The first customer to go public with an application is Nokia, which used the models for 20 U.S. and European cities for an upgraded version of its Ovi online and mobile mapping service released last week. "It's the start of the flying season in North America, and we're going to be very active this year," says Paul Smith, C3's chief strategy officer.

Although Google Earth shows photorealistic buildings in 3-D for many cities, many are assembled by hand, often by volunteers, using a combination of photos and other data in Google's SketchUp 3-D drawing program.

C3's models are generated with little human intervention. First, a plane equipped with a custom-designed package of professional-grade digital single-lens reflex cameras takes aerial photos. Four cameras look out along the main compass points, at oblique angles to the ground, to image buildings from the side as well as above. Additional cameras (the exact number is secret) capture overlapping images from their own carefully determined angles, producing a final set that contains all the information needed for a full 3-D rendering of a city's buildings. Machine-vision software developed by C3 compares pairs of overlapping images to gauge depth, just as our brains use stereo vision, to produce a richly detailed 3-D model.

"Unlike Google or Bing, all of our maps are 360° explorable," says Smith, "and everything, every building, every tree, every landmark, from the city center to the suburbs, is captured in 3-D—not just a few select buildings."

C3's approach has benefits relative to more established methods of modeling cityscapes in 3-D, says Avideh Zakhor, a UC Berkeley professor whose research group developed technology licensed by Google for its Google Earth and Street View projects. Conventionally, a city's 3-D geometry is captured first with an aerial laser scanner—a technique called LIDAR—and then software adds detail.

"The advantage of C3's image-only scheme is that aerial LIDAR is significantly more expensive than photography, because you need powerful laser scanners," says Zakhor. "In theory, you can cover more area for the same cost." However, the LIDAR approach still dominates because it is more accurate, she says. "Using photos alone, you always need to manually correct errors that it makes," says Zakhor. "The 64-million-dollar question is how much manual correction C3 needs to do."

Smith says that C3's technique is about "98 percent" automated, in terms of the time it takes to produce a model from a set of photos. "Our computer vision software is good enough that there is only some minor cleanup," he says. "When your goal is to map the entire world, automation is essential to getting this done quickly and with less cost." He claims that C3 can generate richer models than its competitors, faster.

Images of cities captured by C3 do appear richer than those in Google Earth, and Smith says the models will make mapping apps more functional as well as better-looking. "Behind every pixel is a depth map, so this is not just a dumb image of the city," says Smith. On a C3 map, it is possible to mark an object's exact location in space, whether it's a restaurant entrance or 45th-story window.

C3 has also developed a version of its camera package to gather ground-level 3-D imagery and data from a car, boat, or Segway. This could enable the models to compete with Google's Street View, which captures only images. C3 is working on taking the technology indoors to map buildings' interiors and connect them with its outdoor models.

Smith says that augmented-reality apps allowing a phone or tablet to blend the virtual and real worlds are another potential use. "We can help pin down real-world imagery very accurately to solve the positioning problem," he says. However, the accuracy of cell phones' positioning systems will first have to catch up with that of C3's maps. Cell phones using GPS can typically locate themselves to within tens of meters, not tens of centimeters.

Getting all the angles: Microsoft researcher Johannes Kopf demonstrates a cell phone app that can capture objects in 3-D. Credit: Microsoft

Capturing an object in three dimensions needn't require the budget of Avatar. A new cell phone app developed by Microsoft researchers can be sufficient. The software uses overlapping snapshots to build a photo-realistic 3-D model that can be spun around and viewed from any angle.

"We want everybody with a cell phone or regular digital camera to be able to capture 3-D objects," says Eric Stollnitz, one of the Microsoft researchers who worked on the project.

To capture a car in 3-D, for example, a person needs to take a handful of photos from different viewpoints around it. The photos can be instantly sent to a cloud server for processing. The app then downloads a photo-realistic model of the object that can be smoothly navigated by sliding a finger over the screen. A detailed 360 degree view of a car-sized object needs around 40 photos, a smaller object like a birthday cake would need 25 or fewer.

If captured with a conventional camera instead of a cell phone, the photos have to be uploaded onto a computer for processing in order to view the results. The researchers have also developed a Web browser plug-in that can be used to view the 3-D models, enabling them to be shared online. "You could be selling an item online, taking a picture of a friend for fun, or recording something for insurance purposes," says Stollnitz. "These 3-D scans take up less bandwidth than a video because they are based on only a few images, and are also interactive."

To make a model from the initial snapshots, the software first compares the photos to work out where in 3-D space they were taken from. The same technology was used in a previous Microsoft research project, PhotoSynth, that gave a sense of a 3-D scene by jumping between different views (see video). However, PhotoSynth doesn't directly capture the 3-D information inside photos.

"We also have to calculate the actual depth of objects from the stereo effect," says Stollnitz, "comparing how they appear in different photos." His software uses what it learns through that process to break each image apart and spread what it captures through virtual 3-D space (see video, below). The pieces from different photos are stitched together on the fly as a person navigates around the virtual space to generate his current viewpoint, creating the same view that would be seen if he were walking around the object in physical space.

Look at the video HERE.

"This is an interesting piece of software," says Jason Hurst, a product manager with 3DMedia, which makes software that combines pairs of photos to capture a single 3-D view of a scene. However, using still photos does have its limitations, he points out. "Their method, like ours, is effectively time-lapse, so it can't deal with objects that are moving," he says.

3DMedia's technology is targeted at displays like 3-D TVs or Nintendo's new glasses-free 3-D handheld gaming device. But the 3-D information built up by the Microsoft software could be modified to display on such devices, too, says Hurst, because the models it builds contain enough information to create the different viewpoints for a person's eyes.

Hurst says that as more 3-D-capable hardware appears, people will need more tools that let them make 3-D content. "The push of 3-D to consumers has come from TV and computer device makers, but the content is lagging," says Hurst. "Enabling people to make their own is a good complement."