After a decade of work on augmented reality on mobile devices, Daniel Krause of Nokia's HERE Maps team thinks we're ready to get past the demos and start using phones that show us an X-ray vision of the world around us.
"We're moving from just a map for navigation that's a digital version of a paper map to a next-generation map that's much more contextual, that's much more active and dynamic, that fits in to the moment and really provides the information you need," Krause told TechRadar Pro.
The key to giving you that moment of information is making augmented reality fit seamlessly into the digital maps we already use, Krause says. Nokia added AR to its maps on Windows Phone last year and they quickly learned what does and doesn't work.
"People really like the idea of vital information in context and getting away from the equivalent of a paper map into the real world, but it's very important to keep the map context. If it's an augmented reality viewer that just shows stuff around you then people are confused about what it shows. Our technology, Live Sight, is integrated very tightly into the map; really it's just another map mode."
The breakthrough was building a 3D transition that morphs between the traditional overhead map view and the heads-up augmented reality view where you hold the phone up to use it like a lens on the world.
"We do a 3D transition between the orthographic map and the camera view and back," Krause explains. "We fly the camera down through the classic oblique angle and down to the ground and you see 3D buildings pop out and into the camera view." That lets you keep track of how what you can see around you fits in with the map, which Krause calls 'spatiality'. "One of the greatest uses of 3D is to provide the spatial connection between the heads-up view and the world around you."
So if you see an unusual building and you wonder what it is, you can pull out your phone. Instead of just looking on the map for a building, you can hold up the phone. "As you tilt the phone, the building would show in 3D. If you continue to push the phone up the building will tilt fully and fade into the building that's here in front of you," he explains. The sensors in your phone aren't accurate enough to do that on their own – "we've done a lot of registration work to get it right," Krause notes.
Where can I go now?
What people really find augmented reality tools useful for is exploring the world right around them. "We want to explore things within two or three hundred metres of where we're standing," says Krause. That's fun for tourists but even more useful for business travellers.
The labels you see in the camera view could tell you more than just the name of the building or business you can see, like ratings and offers. "Here are the restaurants you can see, and here are the ratings, and these two have a special on," he suggests.
If you have enough detail in your map, an augmented reality view could save you a lot of walking if you're heading into a large building like a shopping centre or an office complex. "I want to get a route to the door of the building – as a pedestrian, I could be half a mile off. Augmented reality could show me the points of interest. I want to take a quick glimpse; is it in here or how do I find it?"
HERE Maps is adding more features to make that easier. The first is showing the dynamic distance to a point of interest as you walk towards it, and the other is called Point Me There and puts arrows on-screen to point you in the direction of a point of interest.
"That solves the problem of doing a search and holding up the phone and all the Italian restaurants that are close are behind you," jokes Krause. Getting the direction is accurate up to about 20 or 30 feet he says.
Still issues to overcome
There are still some problems to solve concerning displaying the right information for where you are, such as when you're outside a large building and you want to see details of what's inside (like the different shops and services in a railway station) or you're inside the office and you want to see what's outside the building.
That's the occlusion problem: "There's something blocking what I'm looking at – should we efface things that are not directly visible? It works in the other direction too – if you're outside, you might not want to see the points of interest that are behind the building," Krause observes. Or if you're in a large venue, you might want to be guided to the exit without seeing the details of businesses outside. And what about handling different levels: how do you show if an exit is underground or the coffee shop you're looking for is on the second floor?
The Live Sight team is working on the best way to show all of these elements so they're meaningful. "It has to be so when the user pulls their phone out the information being given is really clear and accurate and in context – and it shows any deficiencies that may be there. The limitations [we face] are defined by the resolution of the location information that's available from hardware and how good we are at providing directions that are really constructive, and extremely robust."
Are we there yet?
Live Sight is the third-generation of Nokia's augmented reality technology. Krause thinks it will take off this time, partly because we have fast enough processors, big enough screens and good enough location tools in our phones, along with voice recognition that works reliably. It's also due to the fact that as smartphones have become more ubiquitous, we've started to demand ways of getting information that make sense on a phone screen – and AR is ideal for that.
"I think we're getting very close because people are interested in a simpler and more humanised experience. I don't want to stand out on the street and type in 'pizza'; I want to hold my phone up and say 'show me pizza joints'. There's a desire for simplification and speed. People want it to be simple and easy to get to. It's about searching, exploring, navigating – and making that entire workflow visual and effortless and preferably one-handed. There are lots of technologies starting to appear in phones and operating systems, like the increasing amount of voice recognition, that's starting to tie in with what we can do with Live Sight."
But he's also excited about what we get with the next generation of mobile devices. We asked him when will augmented reality become commonplace, and Krause replied: "I feel it will happen as phones get thinner and lighter, as they become transparent screens [on the world], things we hold up and look through – the glasses on your face or a tablet you hold up and unroll like a pirate map."