Augmented Reality offers digital view of the real world

Imagine visiting Paris and touring the city wearing a futuristic pair of sunglasses. As your gaze moves to different points of interest, messages pop up in your field of view giving a description of each location. Restaurants have virtual signs summarizing reviews of the food. If you need to get somewhere, arrows pop up pointing you in the right direction. Popular landmarks have historical descriptions floating in the air above them. You can even stop and watch a virtual reenactment of Marie Antoinette's beheading.

Such possibilities may not be too far off in the future. For Steven Feiner, a professor of computer science at Columbia University, augmented virtual reality has already become a real possibility. Feiner and his lab, the Computer Graphics and User Interfaces Laboratory, have designed a mobile system that projects interactive graphics objects onto the user's field of view. A student, who wears the computer and sensory equipment strapped on via a backpack, can walk around campus and experience a virtual tour of the history of the campus. Termed "situated documentaries," the technology allows a user to experience a historical event at the actual locations where it occurred. For example, the lab has developed a documentary depicting the 1968 student revolt at Columbia.

Last Thursday, Feiner spoke at CMU about his research in augmented reality (AR). The lecture, held in Newell Simon Hall, attracted an audience of 30 to 40 students and faculty in human-computer interaction and computer science. Feiner described his work in developing an algorithm for displaying text labels on top of the real world without interfering with other objects or text in the display.

The concept of augmented reality is akin to the heads-up displays used by fighter jet pilots. AR aims to augment the real world with 3-D representations of additional objects - data that could be piped in from the Internet or from personal databases.

It has applications in repair work, surgery, tourism, and entertainment, to name a few. A doctor performing surgery on a patient's leg could have X-ray images displayed on the leg, helping to guide the procedure.

Once AR becomes more sophisticated, it could even be used for virtual games set in the real world. Justin Weisz, president of the CMU Online Gaming Society, sees many gaming applications for augmented reality. A first-person shooter such as Quake could be set in the real world, with virtual monsters, items, and even landscaping.

"A game like this would be a great motivation to exercise and be physically active, rather than just vegetate in front of the computer," said Weisz, pointing out another possible benefit of augmented reality games.

Optical vs. video displays

Augmented reality systems can use either of two types of head-worn displays: optical or video see-through displays. An optical display uses a mirror beam splitter -- which is the same technology used in fighter jets - to reflect an image into the user's eye. In contrast, a video display uses a head-mounted camera and a screen to display video from the camera mixed with the virtual graphics objects.

Optical displays allow the wearer to see the world in full resolution and with a complete field of view. However, the rendered graphics objects can only be displayed as semi-transparent images. The video displays allow more powerful and realistic graphics effects, but limited video resolution makes the display look much less detailed than the human eye can see the world.

Feiner's AR system, which is named the "Touring Machine" (as a pun on the name of the famous Alan Turing), uses an optical see-through display. Feiner stressed that his AR system was designed to be quickly constructed with off-the-shelf components, and that AR systems in the future will be much smaller and more stylish. "I don't think any normal person would walk around with this in the future," said Feiner of his 25-pound backpack-based system.

"I think a lot of these issues will be adequately satisfied by commercial products by the end of the decade," he said. He believes that future AR systems will be much lighter, smaller, and a whole lot less weird-looking. They will be light and stylish, making them practical for typical consumers.

Determining Location

In order to have an accurate understanding of the world, an AR system needs to have detailed readings on its location and relation to external objects. Standard nonmilitary data from the global positioning system (GPS) is not accurate enough for the purposes of augmented virtual reality. Feiner's system needs to know the exact movement and location of the subject at all times. The machine uses differential GPS, a technique that provides accuracy on the level of centimeters.

Differential GPS uses an extra GPS receiver and a radio transmitter at a fixed location in another area. The transmitter broadcasts an error correction based on the difference between the GPS measurements it receives and the location at which it knows it is stationed. This allows the original receiver to correct GPS signals to less than a meter of error. Feiner's system improves on this even further by using real-time kinematic GPS - a technique which compares the signal phases of the two receivers.

The system also uses an inertial tracker with a magnetometer to help track a wearer's movements. The inertial tracker helps to detect quick motions, and the magnetometer detects the person's orientation with the ground by measuring the earth's magnetic field, like an electronic compass.

Virtualized reality at CMU

Takeo Kanade, a professor and researcher in the CMU Robotics Institute, has developed a technology for "virtualized reality." Using a system of 30 cameras at the 2001 Super Bowl, his system, which CBS called "Eye Vision," gave viewers a 3-D view of the game at selected points.

Kanade thinks his system could some day be useful in Augmented Reality applications. In order for AR systems to generate realistic images, they will need to have detailed data on the actual world.

Kanade's virtualized reality system takes images from multiple viewing angles in the real world and inputs them into a computer. Though his system uses multiple stationary cameras, it could potentially be adapted for a mobile AR system, or an AR environment could be set up in a fixed location.

The cameras take in an enormous amount of data, and if a very high number of cameras were to be used, the data could require processing power on the order of 20 petaflops (a petaflop is 1015 calculations per second), according to Kanade.

"Twenty petaflops actually is not an exaggeration," said Kanade. "I think [20 petaflop machines] will come, one way or another. I have never seen any field in the past whose limitation was only the computation."

Cray, the supercomputer manufacturer, has set a goal of delivering a petaflop supercomputer by 2010. With advances in both processing power and graphics algorithms, augmented reality systems will most likely come closer and closer to seeming like a part of reality.