Researchers from Canada are developing a prototype technology that will enable users to execute operations on a computer with their hands – but not with a mouse.

The program is called Typealike and by utilising a laptop webcam with a mirror attached to it, actions on a computer are carried out when users perform hand gestures in the vicinity of the keyboard according to an article via The Engineer.

“The webcam is pointed at your face, but the most interaction happening on a computer is around your hands. So we thought, what could we do if the webcam could pick up hand gestures?” said Nalin Chhibber from the University of Waterloo’s Cheriton School of Computer Science in Ontario, Canada. 

The team took advantage of machine learning to ‘train’ Typealike after collecting a series of recorded hand gestures courtesy of volunteers.

“It’s a neural network, so you need to show the algorithm examples of what you’re trying to detect,” said Fabrice Matulic, who is both a senior researcher at Preferred Networks and a former postdoctoral researcher at University of Waterloo’s Cheriton School of Computer Science. 

“Some people will make gestures a little bit differently, and hands vary in size, so you have to collect a lot of data from different people with different lighting conditions.”

Going forward, the team believe that Typealike could be used to make other technologies more efficient, such as virtual reality.

“People look at something like Typealike, or other new tech in the field of human-computer interaction, and they say it just makes sense. That’s what we want,” said Daniel Vogel, who is an associate professor of computer science at Waterloo. 

“We want to make technology that’s intuitive and straightforward, but sometimes to do that takes a lot of complex research and sophisticated software.”