Cutting Edge
Exploration of accessibility and the future created by Sony's technology
Sep 26, 2019
Continuing from last year, Sony took part in SXSW (South by Southwest) 2019, the world's largest creative business festival in Austin, Texas, in March 2019. After two successive years of participation, Sony this year showcased its new initiative on inclusive design by presenting the experiential exhibit “CAVE without a LIGHT.”
We interviewed Yuri Takegami, Yasuyuki Koga, and Junya Suzuki, to share their story behind this project.
Profile
-
Yuri Takegami
Quality & Environmental Dept.,
Sony Corporation -
Yasuyuki Koga
Interaction Technology Development Dept.,
Application Technology Development Division 2,
R&D Center,
Sony Corporation -
Junya Suzuki
Audio Technology Development Dept.,
Fundamental Technology Research and Development Division 1,
R&D Center,
Sony Corporation
Background behind “CAVE without a LIGHT”
──Sony announced the experiential exhibit “CAVE without a LIGHT” at SXSW 2019. What’s that program like?
Yuri Takegami：In full darkness where vision is blocked out, participants explore a dark cave setting reproduced by Sony’s audio signal processing, reverberation creation, and haptic technologies, and enjoy a music session with other participants as a group at the end.
──So, the participants cannot see anything at all. Is that right? How did you reproduce a “cave” without visual information?
Yasuyuki Koga：The program goes like this. Participants first get on a lift in a lighted room and wear an open-ear headset. The room immediately gets dark, and vibrations and spatial sound surround them, which gives them a simulated experience as if they fell into a cave underground.
Once they arrive at the cave, a quiet surrounding is suddenly interrupted by the sound of flying bats, making them experience superior sound through “sound localization” and “reverberation effect.”
Junya Suzuki：Our system is designed so that if you clap your hands or let out your voice, that sound is picked up by the microphones, and reverberations are created and rendered to the open-ear headset in real time.
For CAVE without a LIGHT at SXSW this time, we use the reverberate data measured at an actual cave that causes a lot of reverb, so participants can experience the expansion of the sound as though they were in a real cave.
Koga：After that, the participants use the small bongo drums installed on each lift to join the music session. If one person beats a drum, the beating sound will be picked up by the microphones and converted to vibrations on the floors and handrails of all participants’ lifts based on the built-in haptic system. In this way, the participants can enjoy playing music all together only through the sense of sound and touch.
Takegami：With guitar, keyboard, and vocal sound additionally joining, the music gets heated up, and when the music session reaches a peak with the audience cheering in the background, the room all of a sudden lights up, instantly bringing the participants back to the real world from the dark cave. That’s how it works.
Koga：There were many people who started singing or dancing while banging on the bongo, which really surprised and impressed us. We felt it was true that music surpasses language.
Sony at SXSW 2019 "CAVE without a LIGTH"
What does Sony think accessibility is?
──CAVE without a LIGHT at SXSW 2019 was a totally new user experience created by Sony’s inclusive design (a method of design that includes or understands various people in creation to acquires new awareness), and it was the first case that announced to the public that Sony is taking initiative to emphasize accessibility. Well, let us ask at this point, “What is accessibility in the first place?” and “Why does Sony focus on accessibility?”
Takegami：We are often asked about the difference between accessibility and usability. Usability pursues the improvement of ease of use, whereas accessibility aims to increase the range of people who can use devices or services. In other words, accessibility pursues the ease of use, regardless of limitations such as age or disabilities. Those limitations include the ones temporarily caused by an injury or sickness or not being able to understand a language while in a foreign country.
Sony believes one of its social responsibilities is providing products and services that can be used by people even with such limitations or in such situations, and Sony is making great efforts to develop products and services that care about accessibility.
──As the concept similar to accessibility, we often hear the term “barrier-free” or “universal design.” What’s the difference?
Takegami：“Barrier-free” is used to describe for buildings and other facilities. “Universal design” is a design accessible to all people. “Accessibility” is a measure of how products/services are easy to use, independently of a user’s age or disability status. Its variability and customizability are also defined.
Inclusive design, on the other hand, is a methodology. It incorporates various viewpoints, enabling higher quality and wider range of technology. In the case of CAVE without a LIGHT, collaborating with Suzuki-san, who lives without sight, is a very important process to us. Working with him has made us realize new discoveries and potential needs that we could not be otherwise aware of, allowing us to create a new experience that can be enjoyed by everyone whether with or without sight. Delivering new user experience based on inclusive design is, after all, creating products that consider accessibility to be used by a wide variety of people.
Suzuki：Historically speaking, voice interaction is a good example. We have voice recognition or voice synthesis systems these days, but such technologies were originally developed for people with vision loss. Considering the fact that such dedicated technologies are now an interface available to anyone, it is important to think that limitations or constraints are good opportunities for us to overcome challenges, ensuring that not only a large majority of general users but also people with disabilities can properly use our products and services. This whole effort should entail further improvement of quality in our products and services.
Takegami：Sony believers that inclusive design will widen the possibilities of new technology and creativity. Based on that policy, various sections across the company, such as Quality & Environmental Department, R&D Center, and Creative Center, gathered and worked together on this new challenge. Actually, it just started as an internal activity, but our outcome was highly evaluated and given the chance to participate in SXSW.
Significance of presenting information through sound
──Let us ask Suzuki-san and Koga-san. Before joining the project of CAVE without a LIGHT, what kinds of work were you doing?
Suzuki：I’d been holding an idea of journeying to many locations through “sound,” and to achieve this, I tried to reproduce the reverberation characteristics of many places in real time. Theoretically, in order to increase reality, it is necessary to reproduce long reverberation as accurately as possible. To generate real-time reverberation elements, on the other hand, the amount of calculation should be reduced. Since reproduction of long reverb requires computational complexity, I devised various ways in calculation. This applied technology is used in CAVE without a LIGHT.
Koga：I had developed a neckband-style wearable device for several years. This technology localizes the sound and enables users to enjoy music or voice guidance coming from the neck speaker as if the sounds were coming right close to their ears.
Last year, when I developed a Sound AR prototype using this sound localization technology and presented it in an internal event within Sony, I received a suggestion that I should combine it with the reverb technology developed by Suzuki-san to create a new challenge, which gave me an opportunity in the end to exhibit CAVE without a LIGHT at SXSW.
Suzuki：Actually, many studies and researches have been performed for about 40 years to provide sound information to the visually impaired people. But most of them did not turn into products, or even if they did, the market did not accept them. The crucial reason is that it is highly dangerous for the visually impaired people to cover their ears, especially when walking, regardless of how good information is provided with sound or speech. People with sight loss like me always rely on the ambient sound to understand the surroundings. In that respect, the neckband-style wearable speaker developed by Koga-san is an epoch-making invention because it does not block the surrounding sounds, while at the same time it can give us useful information through sound.
Koga：I’m glad to hear that. This product was presented in San Francisco as a demonstration experiment a few years ago, and one blind man, after experiencing it, instantly jumped at me asking, “When can I get it? I’m waiting for a long time to have such a device!” Another person who tried the device commented to me: “This device has let me realize how many people are constrained by information and devices. The device and technology like this might change the people’s lifestyle in the future.” Through that experience, I felt great potential in this technology, and I’ve come to think that this technology can be used to grasp the space or reproduce three-dimensional sound, and if some form of feedback is constantly given to the information during that process, it should enable a user to get personalized information that is relevant to that person’s needs.
Physical sound and psychological sound
──So, you worked with Suzuki-san in the form of inclusive design. What kind of awareness have you got, Koga-san?
Koga：What impressed me most when I first met him was his explanation that the sound we hear is something like white noise floating in the air and we just hear it in real time after reflected from the wall. Usually, people only hear what they want to hear, so the sound like noise will not come into their ears, but people with visual impairment like Suzuki-san can distinguish the strength of noise reflection from the wall. I was totally astonished when he told me that he could know whether he is approaching to a wall by hearing the strength of reflection.
Suzuki：A wall reflects the ambient sound, and this reflected sound and the pure sound I directly hear from my surroundings will bump in my ears, adding one sound waveform to another, which changes the sound. Of course, the sound quality varies depending on the distance between me and the wall, as time delay in waveforms will be different. This phenomenon is called “coloration,” and I told Koga-san that if this is realized by headphones or earphones, we can give a higher reality to users like “Hey, something is coming this way!”
Koga：As soon as I heard that explanation, I tried walking closer to the wall, but I could not tell the difference at all. I just understood that there is such a fact. One day, I happened to see Suzuki-san walking and I watched him for a while. Although he was not walking along the wall, he simply turned on the corner without effort, which surprised me again and made me confirm what he had talked about was true.
Suzuki：Everyone gets surprised with that, but I’m just normally walking because the sound is obviously different. However, what is most interesting to me is that I cannot feel coloration, the sense of something coming closer to me, in the binaural recording, which can reproduce human hearing. I think I’m recognizing the space based on other factors, in addition to hearing the sound with my ears. I need to explore this mystery more deeply, and I’m very excited to think there might be a key around this mechanism that leads to reality.
──There may be a possibility that sound is sensed by something other than ears, such as skin or organs, because it is a wave.
Koga：Yes, there is such a possibility. I often meet with a group of audio engineers lately, and we are discussing about human recognition and reality. Even if a waveform can be reproduced, how humans recognize it is different according to individuals. When it comes to how humans grasp the space or distance, it is all the more difficult to find the answer. I think we need to persistently repeat various experiments to find what should be reproduced with higher priority to create more reality.
Suzuki：There are two approaches for sound technology. One is an approach that tries to accurately reproduce an actual thing, for example, reproducing an actual wave front. The other one is a psychological approach, which focuses on how the human mind interprets sound and recognizes what the sound is. The former is called “physical sound” and the latter is “psychological sound.”
There are many measurement tools and calculation methods for physical sound, but as for psychological sound, there is so much room to dig deeper.
Research on hearing is research on humans
──What kind of values do you think will be created if physical sound and psychological sound are delved into deeper from those both approaches?
Suzuki：An aging society is coming soon in Japan, and various types of assistance will be in demand. One easy-to-imagine example is hearing extension. It is not simply the matter of making sound louder. It is the technology that selects and narrows down information that is really necessary right here, right now within the society flooded with information. As you might know, there is a cocktail-party effect in the sense of hearing, which is a human’s ability to focus on a single voice from a preferred direction when many others are talking. But according to some research, an aged person will be weaker in this singling-out ability. I think we can complement such an area by delivering many assistive things on a hearing-sense basis.
──So, research and development in the auditory field is to develop technology that gives the chance to know more about humans.
Suzuki：That’s true. I have been dealing only with the auditory field, working on voice synthesis and voice recognition. Now I’m involved in the overall audio domain. And even Just looking at this field, I’m very impressed, feeling that human activities are so admirable. It might be hard work to seek a way to get closer to humans, but I feel very excited at the same time to think about that.
──It would be interesting if Sony announces in the future that it has found humans have eight senses, or rather fifteen senses, instead of five—a new discovery of human recognition mechanism...
Koga：I hope such a thing would suddenly happen in the future. In that respect, our research field is the one that studies humans. However, understanding humans tend to have bias, and to get rid of that bias, we need to talk with as diverse people as possible. Inclusive design will be all the more important.
Takegami：Sony’s unique accessibility will be one of our major themes. Having Koga-san and Suzuki-san work together this time made us realize a lot of things that were not conventionally realized by engineers. We want to deliver fun to everyone with or without vision. We want to draw such a world. We will continue to strive for it.
Koga：Technology must be designed to care for people after all, because those who create and those who experience are all humans. If AI evolves simply from the aspect of technology, it might become wiser, but it steadily becomes boring or will not be wanted by people. It is important that those who create must be more conscious of it. Specifically, a diversity of people should gather and collaborate in creation, which should automatically widen the range of people who enjoy the products and services output from it. There are many technologies and systems that I want to work on, and I want to keep on developing them by understanding inclusive design as an approach to attain our goal.
