Why did that happen so quickly for touch gestures and not for 3D gestures?

Julius Hassemer: Yes, that’s surprising when you consider how affordable 3D gesture recognition has become. For example, Leap Motion does a device for about 80 Euros for the small space in front of a computer, and which is simply hooked up to the computer via a USB port. Microsoft games console Kinect can be used for larger spaces, and has been around since 2010. Yet there is no area of applications where gesture control is the standard either for smartphones or for Smart Homes. And that’s not for want of networks: For some time now various household devices have been able to communicate with one another, or can be remotely controlled using a smartphone – but only on the two-dimensional surfaces of cell phones or tablets. Why is 3D lagging behind? Hard to say. But one aspect is most likely that we are used to controlling devices by touching them with our hands, by pressing a button, pulling a cord for the blinds, or turning a dimmer switch. The only time we use our hands in a touch-free manner is when we are communicating with other people, when we are gesturing to accompany what we say. Maybe it only feels natural and intuitive to operate a machine using gestures when we actually have the sense we are not only giving orders but are “speaking” with the device.

And do you really think this situation will come about quickly, and that our interaction with machines will feel like a real conversation?

Julius Hassemer: No. But if gesture recognition is limited to my making a sliding motion in the air causing the sliding roof on my car to open, or the blinds on my window to open then I am going to favor a switch, because there at least you have tactile feedback through the switch itself.

What exactly is your work and what can it achieve?

Julius Hassemer: I do basic research, which cannot of course be transformed into a specific application. I perceive the following dilemma: Today, gesture control is still undergoing teething problems and the focus is very much on technical issues. However, in the midterm these technical problems will be solved and the hardware will be able to capture my gestures precisely, regardless of whether I’m standing right in front of the device or am walking through my apartment. Then you have a whole lot of motion data, in other words, the information about the location of every finger at all times. After which we will have to address the big questions. What does all the data actually mean? If we started with the basics now, then later, once the technology is advanced enough, we’ll be able to work on its practical realization, which will rely on basic gesture categories.

What do you mean by basic gesture categories?

Julius Hassemer: Gesture research already includes various suggestions for how to classify gestures. For example, different types of pointing gestures, rhythm stick gestures, depictive gestures and so on. One of the problems that prompted me to suggest a new classification is that existing categories are often difficult to express in spatial parameters. But what is special about using gestures is that the hand imitates certain shapes and motion patterns in three-dimensional space. And these shapes and motion patterns are the starting point for my gesture classification.

What does a developer need to know about the different types of gestures if devices are to be controlled in the home environment?

Julius Hassemer: Today gesture-operating concepts are either very restricted, say for opening a sliding roof, or they are limited to conventional menu control, in other words a list of menu items that you choose from. Naturally, you can also use gestures to control devices but the operation is neither quicker nor easier as a result, on the contrary. Well, different types of gestures have different functions and communicate different spatial information. We need to exploit that. Let’s take an apartment completely controlled by gestures: An occupant enters her apartment and is surprised it is so cold. She would like to do two things, put up the blinds and find out how long the heating has been on today. She could use gestures to operate a menu on a screen and select the desired functions – main menu, item 3, then sub-item 8 and then turn a switch and return to the main menu for the next command. But that would be tedious and would ignore the potential of gestural communication. Making an upward motion with the hand when it is palm upwards and towards the blinds and then simply pointing an index finger at the heating could execute the functions directly. The information the computer needs could then be shown on a screen or simply conveyed acoustically. In this case the gesture type – raising the hand versus a pointing gesture – substitutes for the selection in the menu. If a pointing gesture were made at the blinds and the hand moved up and towards the radiator this would both serve to turn up the heating and provide information on how long the blinds were down. In other words a clever distinction between gestures replaces the menu.

Am I then not influenced as a human being in my actions and range of gestures by devices? Can I still move around freely without something happening involuntarily?

Julius Hassemer: You have hit on a very important question, and one that has practical consequences: “When am I on?” Should gesture control always be on? Then it would constantly capture motion data, which is awkward from the point of data protection issues. And do you have something like an on/off gesture so you can activate gesture control? What should the on/off gesture look like so it is not executed by mistake? It would have to be a gesture that is quick, needs little effort and can be made in every situation. Hands are often busy for example, feet too, but the gaze must go where it wants and so you are always limiting yourself. Maybe we could use our eyebrows. Some people can move them separately, or use a short combination of gestures. Those are very practical problems, but they can be solved fairly easily

Where do you see these applications in our future homes?

Julius Hassemer: I can imagine technology will no longer be visible in future homes. And I won’t need a smartphone or an alternative input interface. After all I have my “gesture space”. I will control my environment via sensors hidden in the walls. This would mark a decisive change: We would no longer need to have physical contact with technology. Touch would only be used amongst people (laughs). In the movie “Her” such a virtual reality is embedded in a real setting and consequently perception of the space alters radically. Space is enhanced by technology. (Hassemer screws up his eyes slightly and uses his extended fingers to point to a vague spot ahead.) My apartment is empty. It is a room in which I can do everything. Phone my kids, operate my devices or do yoga in an empty space. You could pare back what you need on an everyday basis, because we would return to the space, the pre-technical space in which technology may be invisible, but is omnipresent. And that can be very positive. There is nothing comforting about technology and it does not provide me with any emotional extras. If my computer is a drywall installation, I never have anything to do with it, and never have to touch it.

Is gesture control a convenient technology for the luxury segment or a technological innovation that everyone can profit from?

Julius Hassemer: Gesture control is not forever fated to be a fancy toy for luxury homes but at best could lead to a shift in who we perceive the interaction between man and machines. Today, smartphones are affordable for people of all income levels even though they started out as luxury items. Or take “Nintendo Wii”: it is now an inexpensive option for old or ill people as they can do a minimum of exercise or whatever suits their physical condition. And that works out much cheaper than hiring a physiotherapist. Incidentally, a follow-up to this console would combine wonderfully with a comprehensive gesture control system in the smart home.