Two Indian students develop ZUNAVISION

Stanford artificial intelligence researchers, computer science graduate students Ashutosh Saxena and Siddharth Batra and Assistant Professor Andrew Ng have developed software that makes it easy to reach inside an existing video and place a photo on the wall so realistically that it looks like it was there from the beginning.

The software can easily plunk an image on almost any planar surface in a video whether wall, floor or ceiling. And the embedded images don’t have to be still photos—you can insert a video inside a video. A photo disappears behind the person walking by and then reappears just as in the original video. It also involves camera motion that causes the portion of the wall containing the embedded object to move and change shape.

Siddharth Batra, graduate student in computer science, places the Stanford logo onto a wall of the Hewlett Teaching Center using software that can embed graphics into a video—images, photos or even another video.

If your favorite video of a party is marred by a small, troubling detail - in it is a photograph of someone whose picture you did not need to keep, ZunaVision will allow you to "take down" the photo - and replace it with your favorite person’s photo.

The researchers said the software can place an image on almost any plain surface in a video - it can even insert a video inside a video. This, for instance, opens up the possibility of your singing karaoke sideby- side with your favorite singer. Or livening up those dull vacation videos.

The researchers said that anyone with a video camera might now earn some spending money by agreeing to have unobtrusive corporate logos placed inside their videos before they are posted online. The person who shot the video and the company handling the business arrangements would be paid per view in a fashion analogous to Google AdSense, which pays websites to run small ads. The technology can cheaply do some of the tricks normally performed by expensive commercial editing systems.

The embedding technology is driven by an algorithm that first analyses the video with special attention paid to the section of the scene where the new image will be placed. The color, texture and lighting of the new image are subtly altered to blend in with the surroundings. Shadows seen in the original video will be seen in the added image as well.

The result is a photo or video that appears to be an integral part of the original scene, rather than a sticker pasted artificially on the video.

The algorithm ("3D Surface Tracker Technology") used to produce these realistic results is also capable of dealing with what researchers call "occluding objects" in the video - a guest walking in front of the newly hung photo. It achieves this by first building a model, pixel by pixel, of the area of interest in the video.

"If the lighting begins to change with the motion of the video or the sun or the shadows, we keep a belief of what it will look like in the next frame. This is how we track with very high sub-pixel accuracy," Batra said. "It’s as if the embedded image makes an educated guess of where the wall is going next, and hurries to keep up."

Other technologies can perform these tricks - witness the spectacular special effects in movies and the virtual first-down lines on televised football games but the Stanford researchers say the existing systems are expensive, time consuming and require considerable expertise.

Some of the recent Stanford work grew out of an earlier project, Make3D, a website that converts a single still photograph into a brief 3D video. It works by finding planes in the photo and computing their distance from the camera, relative to each other. "That means, given a single image, our algorithm can figure out which parts are in the front and which parts are in the background," said Saxena. "Now we have extended this technology to videos."

Siddharth Batra, graduate student in computer science, places the Stanford logo onto a wall of the Hewlett Teaching Center using software that can embed graphics into a video—images, photos or even another video.

BY RITU PANDEY