Enhanced image: A prototype app for smart phones matches live images of people to stored profiles and shows icons for social networking sites around their heads.
Credit: The Astonishing Tribe
An application that lets users point a smart phone at a stranger and immediately learn about them premiered last Tuesday at the Mobile World Congress in Barcelona, Spain. Developed by The Astonishing Tribe (TAT), a Swedish mobile software and design firm, the prototype software combines computer vision, cloud computing, facial recognition, social networking, and augmented reality.

"It's taking social networking to the next level," says Dan Gärdenfors, head of user experience research at TAT. "We thought the idea of bridging the way people used to meet, in the real world, and the new Internet-based ways of congregating would be really interesting."

TAT built the augmented ID demo, called Recognizr, to work on a phone that has a five-megapixel camera and runs the Android operating system. A user opens the application and points the phone's camera at someone nearby. Software created by Swedish computer-vision firm Polar Rose then detects the subject's face and creates a unique signature by combining measurements of facial features and building a 3-D model. This signature is sent to a server where it's compared to others stored in a database. Providing the subject has opted in to the service and uploaded a photo and profile of themselves, the server then sends back that person's name along with links to her profile on several social networking sites, including Twitter or Facebook. The Polar Rose software also tracks the position of the subject's head--TAT uses this information to display the subject's name and icons for the Web links on the phone's screen without obscuring her face.

"It's a very robust approach" to facial recognition, says Andrew Till, vice president of marketing solutions at Teleca, a mobile software consulting company in the United Kingdom. "It's much, much better than what I've previously seen."

Till says that applying image and face recognition to the trend of posting photos on social networking sites opens up interesting new possibilities. "You start to move into very creative ways of pulling together lots of services in a very beneficial way for personal uses, business uses, and you start to get into things that you otherwise wouldn't be able to do," he says.

Polar Rose's algorithms can run on the iPhone and on newer Android phones, says the company's chief technical officer and founder, Jan Erik Solem. The augmented ID application uses a cloud server to do the facial recognition primarily because many subjects will be unknown to the user (so there won't be a matching photo on the phone), but also to speed up the process on devices with less processing power.

Academic and company research groups have developed augmented reality applications, which superimpose virtual objects and information on top of the real world, for more than a decade. But until the past year or so, all of these prototype applications required bulky headsets and laptop computers. With more powerful sensors, cameras, and microprocessors built into mobile phones, however, augmented reality applications have begun hitting the mainstream. Several apps take advantage of the GPS chips and compasses available in newer smart phones. For example, PresseLite's Metro Paris app and Acrossair's Nearest Tube provide iPhone users with directions to nearby subway stops.

But Gärdenfors calls such applications "relatively crude." They often obscure objects with labels, he notes, and are sometimes limited by the fact that location information may not be available. He thinks that many augmented reality services could benefit from including elements of computer vision to make information retrieval and label positioning more precise. "This could absolutely work for other kinds of objects, and I think we'll see that soon," he says.

However, Gärdenfors notes that using computer vision to identify buildings and other objects holds challenges that they didn't encounter in developing the augmented ID application. "With facial recognition, it's so obvious what you want to search for," says Gärdenfors. "With other objects, it may be harder to tell which item on the screen you want to identify."

Gärdenfors says that TAT has taken potential privacy concerns with the technology seriously from the beginning. "Facial recognition can be a kind of scary thing, and you could use it for a lot of different purposes." For that reason, the company designed Recognizr as a strictly opt-in service: people would have to upload a photo and profile of themselves, and associate that with different social networks before anyone could use the service to identify them. "You should only be able to look at people who have signed up for this," Gärdenfors says.

A concept video of the augmented ID application that TAT posted on YouTube last summer garnered a great deal of attention. Gärdenfors says the company often uses this strategy to determine which ideas justify further development. A live demonstration also received a lot of interest at the Mobile World Congress. "We're probably going to partner with some company over the next couple of months to take it to the next level and actually build [a product]," Gärdenfors says. While this will require partnerships with a device maker, a mobile service provider, and social networking services, the technology is developed enough that a commercial application could be ready in as little as a month or two, he says.

Copyright Technology Review 2010.