Media

The man with the electric lollipop

Adrian David Cheok is one of the pioneers of what is known as mixed reality. He examines ways in which the digital and physical worlds can better interact. The result is a series of incredible ideas that have the potential to make all our lives easier over the long term

My goal has always been to make new things. This was true growing up, I loved to play with gadgets and take apart things to figure out how they worked. I soon realized I really enjoyed “hacking” in the good sense of the word. My entry into the world of invention was actually fairly conventional – I studied electronic engineering and then completed my doctorate in power electronics. My first job was in the development lab at Mitsubishi Electronics, where I worked on a range of interesting projects. These included increasing the efficiency of electricity grids, developing high-speed trains and a range of other topics that engineers love. However, I quickly became aware of the fact that we were only working to improve existing items. This of course was important work in itself and a practical endeavor, but I felt that I wanted to do more to help bring innovations into the world. At the same time, the Internet age was just beginning. As a result, I started to focus closely on digital technology and arrived at the following question: Why did the virtual worlds and virtual reality of the 80s and 90s, which had started so promisingly, never really take hold and accelerate technical development? Why had these approaches only found niche uses, such as in automotive development?

Numerous renowned psychologists have looked at this question, and largely share the same conclusion: As real people, we are not comfortable in purely virtual worlds where we cannot touch, feel, taste or smell. We have developed our senses over millions of years and think in fact through our bodies. Increasingly, neuroscientific discoveries show that the separation between mind and body is an illusion. My conclusion from this was that we had to make virtual worlds sensory so that they could actually be experienced by people. Only by doing this will we ensure that virtual reality genuinely aids technical development and helps improve people’s lives around the world. Basically, this is what my work in the “mixed reality lab” is all about.

When I started working in mixed reality, I initially looked at developing mobile augmented reality. We developed computer games where the boundaries between the real and virtual worlds were blurred. This included projecting a virtual object, such as a virtual flower in a real vase. I wrote a series of scientific papers on how this could be achieved, which were very well received in the academic world. However, I didn’t really get a clear idea of the effect we might have on society until we presented “Human Pacman”. “Human Pacman” is a video game that is played at the point where the virtual and physical worlds meet: Players become the actual pacman character, play in the city streets by collecting virtual cookies, and run away from ghosts who are their friends.

This might just sound like fun, but the concept has a very serious background. Increasing numbers of children spend day after day glued to their computer. Of course, you may ask why they need a “Human Pacman” to get them outside. Why can’t they simply go out and kick a ball around? This may be a valid point, but it means nothing if children don’t do it. We can’t turn back the clock. Kids love computer entertainment, so we must ensure this has a physical, interactive edge if we want our children to do more exercise. This doesn’t just apply to kids either – it is equally valid for the elderly. In fact, it applies to both groups together. To address this issue, we invented “Age Invaders”, a mixed reality game based on “Space Invaders”. The twist here was that grandparents used laser beams with fast lasers, while the children’s were slower, and in addition, the children should follow fast dance steps. The children thus had to move quickly to dodge lasers from the grandparents, while the older players had much longer; the interaction between generations was fantastic to see. Through studies in Singapore, we also established that we could introduce older people to computer technology through games like this, and they enjoyed playing with the younger generations. First they play, then they surf the ‘Net, then they use computers to simplify their everyday lives. We saw this as a particularly positive side effect of our games.

The games brought my lab’s research international recognition, and for that I am of course very grateful. However, I wanted to explore further the potential of digital entertainment and communication. As I mentioned, for a few years I have been driven by the question of how we can turn Internet communications into encounters that feel genuine. This might sound a bit funny at first. We can already chat and Skype with video, and send endless e-mails; and that’s all great. Yet people still constantly fly around the world, putting a great strain on the environment, in order to meet personally. Why is this? The reason is the same as that behind the limited success of virtual worlds. Rational communication only accounts for 30 to 40 percent of all our communications, while the rest is conveyed via body language, gestures, touch, smell, situation, etc. Chatting to a friend over dinner has an entirely different quality to an MSN chat. In the mixed reality lab, we are looking for ways to make Internet communications feel more genuine through the use of non-verbal elements.

We have started with touch, or more precisely, hugging. A prototype has been developed for a system that we call the “Huggy Pajama”. Through this, parents who are away travelling can embrace their children at home in bed, provided they are wearing the special sleep suit with controllable air chambers. It feels incredibly realistic. Please don’t misunderstand the aim of our work. We are also aware that the world has more pressing needs than a remote hugging system. What we’re carrying out here, however, is research into improving remote communications. Our strategy is to come up with crazy things, in the hope that they will at some point lead to important practical applications.

A good example here is the electric lollipop. This allows flavors to be transferred remotely. You place an electronic plate in your mouth that activates the taste nerves, which enables, for example, a literally sweet message to be sent over the Internet. In the near future, we also hope to be able to transmit smells via the web. I understand that this might also sound a little crazy or unnecessary, but the project is being financed by the Japanese computer firm NEC and has a serious background. Studies show that 20 percent of all elderly people in Japan can go over a week without speaking to a single person. Work pulls families apart all around the world; traditional family meals are only held occasionally, if at all. My hope would be that a sensory Internet brings us closer together. Grandmothers or grandfathers can cook in their kitchen, while grandchildren sit in their own kitchen, smelling what’s being cooked and talking about the meal with them.

Once again, we don’t want people to only communicate through technology that artificially creates authenticity (however contradictory that may sound). Humans require genuine authenticity. Perhaps one day it will be possible to walk through a virtual Coliseum that looks and feels relatively real. People will still travel to Rome, however, to see the actual Coliseum. I feel a world in which people only embrace using the “Huggy Pajama” would be a nightmare. There are parents and children today who communicate with each other via MSN Messenger whilst being in the same house – which is of course fairly ridiculous, and it may even be damaging.

At the same time, we must seize the opportunities offered by new technology to improve communications. At present, we are leaving the information age and moving into the “age of experience”. The Internet has multiplied our communications, but at the same time it has separated us physically to a certain extent. Technically assisted, sensory communications offer an opportunity to bring us closer together again – during times of work, school or business meetings and trips. There are even other potential uses which initially I did not think of. I’ll give you an example: When we presented the “Huggy Pajama”, I received messages from all around the world. One e-mail came from a father in the UK whose daughter was in quarantine in a hospital. Nobody was allowed to approach her, and the “Huggy Pajama” would have been the only way that the little girl could have been embraced.

The YGL community has really helped clarify the possible social effects of inventions. I’m now 39, and in the next part of my life I will try to examine how creative technology can help underprivileged people. Geeks and “mad engineers” like me need to see this as a much more important part of their work. I am very excited as to where this path may take me.