A camera-equipped cellphone can be used to control a computer as if it was a three-dimensional mouse, thanks to prototype software developed by UK researchers.

The software makes it possible to move and manipulate onscreen items simply by waving a handset around in front of a screen, a bit like the motion-sensitive Nintendo Wii controller.

"It feels like a much more natural way to interact and exchange data," says Nick Pears, of York University, UK, who made the system with colleagues Patrick Olivier and Dan Jackson at Newcastle University, also in the UK. "Most people who see it think it is really cool."

Pears says the current prototype, which can be used to control a desktop computer, is just the first step.

"The invention really comes into its own when you realise that modern large public displays are really just computers with big screens," he says. For example, the software could let people interact with video advertisements.

To control a screen, a user simply aims their cellphone's camera at it. The handset then connects, via Bluetooth, to the computer that operates that screen.

Extra dimensions

Once a connection is established, the computer knows exactly where the phone is pointing because it places a reference target on top of the normal video feed and compares this to the phone's picture (see image, top right).

The distance between the cellphone and the screen is based on the way the screen's size changes due to perspective.

The computer translates the phone's movement and rotation in three dimensions into the actions of an onscreen cursor. It possible to use the phone like a 3D mouse, interacting with objects by pressing the phone's buttons or rotating the phone.

In testing, volunteers were asked to resize an image on a screen. They selected the picture using a button and manipulated it by moving or rotating the phone. Moving the phone closer to the screen enlarged the photos, and drawing it away made them smaller.

Another trial involved sketching a house using the phone.

Natural interaction"

I like this because connecting phones and computers is just such a pain right now," says Mark Dunlop, who works on user interaction and mobile phones at the University of Strathclyde, Scotland. "You should be able to see something on screen and just get hold of it."

Mobile phones may be ubiquitous, but people are only now starting to use them for more than just calls and messages. "We're still looking for more natural ways of using them to interact with other devices," Dunlop says.

Better prototypes need to be tested first, he says, but it is important to enable people to download information from public displays to their cellphone.

"We need ways for people to get that public information onto their personal devices, for example from a train station display," he says. "This could be one way to do it."

A paper on the prototype set-up will be demonstrated at the International Conference on Computer Vision Theory and Applications in Madeira, Portugal later in January 2008.

Top of Page

BACK to STEP News