Curve article on mixed reality, interview with Adrian David Cheok

The Curve: Media theorist Lev Manovich has talked about how in the 90s virtual reality, was seen as somewhere we went to, entirely separate form the physical world. The pioneers were in early cyberpunk fiction like William Gibson’s Neuromancer trilogy, how did these visionaries influence you?

AC: Communication is a fundamental need of most people and apart from a few geeks like me, maybe they don’t want to escape into a virtual world. Social networks like Twitter and Facebook prove that people have a great desire for communication. This is the main change.

Early on I was working on augmented reality as a useful means for children in their learning, exploring whether it could it be used for reading and seeing 3D objects in the books and if it is done collaboratively with parents or other children it’s more engaging. We’ve realised now that our brain is very much connected to the body and so learning for children and coming up with creative ideas we need to trigger all of the senses. Even related fields of neuroscience now can do magnetic imaging of the brain; can see that the brain is a network system, so we would not learn the way we do without our body. It’s now seen as important that we have as much physical and multisensory communication.

The Curve: How will this shape the future of the image?

AC: What will happen, especially now that it’s turning digital, is that images in the physical world will become very personal. Now we can track people, their location, their context, their interests, and we can personalise images with digital signage. That’s going to change, the public space will be communicating all the time with the individual. This will include being able to project smell and taste. There’s a student in Keio University where I am teaching who made a kissing screen. You can see an image of someone and then as you approach the screen you can kiss them. It becomes tangible.

This may sound weird but the theory is that the image can respond with taste and smell. One of the early stages will be in advertising. Already some hotels have signature smells, every hotel you stay in of that chain smells the same. Because the smell and taste sensing is directly connected to the limbic or emotional part of our brain they can directly affect emotions, I am here in Vienna, I could have some food here and share the taste with my family in Tokyo.

The Curve: There’s been a lot of excitement about Google glasses. What are your thoughts on how they might work, on the image of the world they will provide for us?

AC: It’s a very good idea but I don’t think the technology has actually been developed. An important point for augmented reality with these kinds of glasses is the head-mounted display is still very big and most people don’t want to wear these geeky glasses. The reason the goggles/displays are so big is the optics. You have to have certain lenses to get the image on to your eye. Their idea is to take the lens part and put it on to your eye as a lot of people wear contact lenses already. The electronics part is very small. So you can have sunglasses projecting the image on to the lens on your eye. Once we have this small head-mounted display then the concepts like Google glasses will become reality.

The Curve: Will this change how we process visual information?

AC: People already have an overload of information. Email for example is almost becoming impossible, if you have ten or twenty people you email with. When it’s like 200 emails a day then it’s impossible. You need to be very careful what information is provided when you have these kind of augmented reality displays. You can quickly go from something that is interesting to a nightmare of information. We have to think about the social side of technology and the impact. For example you can imagine meeting someone at a conference or party and being able to pull up their details, remembering their name, their likes that kind of thing. I think the early stages of any commercialisation are entertainment and advertising; it’s a way to make money so these technologies can be cheaper and widespread.

The Curve: This kind of Blended Reality imagery is more immersive, like a game, more intense, more directly connected to us?

AC: We are reaching hyper-connectedness, a quantum step in hyper-connectivity. For young people growing up it’s just normal. Time has become one of the world’s greatest resources. You can load up more songs on your iPod than you can ever listen to in your whole life, effectively infinite information. We have to realise that human time is not infinite – media is infinite and digital is infinite.

The Curve: What is the next level your work in mixed reality is moving towards?

AC: At this workshop I have been at, there was a very good discussion about the brain-computer interface. There’s been work on measuring from the brain and controlling a mouse and there’s been work on the opposite direction, where the computer influences the brain. I’ve been thinking of using magnetic stimulation of the brain that’s already used in the medical field where they are using Trans-Cranial magnetic stimulation for therapy. You can activate certain neurons in your brain using magnetic coils on your head so it’s not invasive. If you can measure people who are eating chocolate, what part of the brain is activated, it can then be recreated with a pulsed magnetic field so theoretically you could smell chocolate without chocolate actually being there.

The next stage of this is direct brain interface, where you could communicate using brain-to-brain communication. Already people are talking about simple things to measure if you are happy. Trying to activate your partner’s mood. So there was work done in Japan in advanced telecommunications research laboratory, where they could measure what people were looking at. Simple shapes you could measure just from the magnetic imaging of the brain. Already we are getting to the point where we can activate or read images or thoughts in the brain and that’s probably going to be the next stop to combine biology with the internet.

I set up the Mixed Reality Lab in early 2000, the idea was to make inventions which allow the real and digital world to merge. The latest works have been about digital taste stimulation. To stimulate your tongue with taste messages, so you could send someone a taste message. Going beyond the information communication we can already send someone any kind of information through the internet. So I am in Vienna now. You can Google, find out about the city, the weather, there are probably 100,000 photos of Vienna. But what does it tangibly feel like to be here? To taste the food and drink the wine? That’s the next stage of mixed reality and that’s what I want to invent in the lab. One of the reasons I wanted to go to Japan is that there’s a very strong field in robotics in Japan. Robotics is a strong digital media which is essentially physical, so I am now working with students and Professors who have a robotics background to make these new devices for taste and smell.

To be able to touch each other via the internet, to feel what it’s like to be in a place, to combine them in a Telepresence anywhere in the world and share the experience. That’s the important point of mixed reality, and it goes back to the important point of why is Facebook so popular? Human beings have a great desire to communicate, so these technologies need to address fundamental needs like communication, something fundamentally important for people’s happiness.