Co-creation led by a lead user
with disabilities

In XR Catch, two players face each other and pass a virtual ball created by extended reality (XR) technology back and forth. To throw the ball, all a player needs to do is shake the smartphone in their hand. Four speakers are placed between the players to notify them of the ball's movement using sound. Relying on the moving sound, the other player can catch the ball by pressing the button on their smartphone. The ball’s speed varies depending on how the thrower swings their hand, and the sound created when the ball is caught changes according to three levels, depending on the accuracy of the catcher's timing.
In addition to a face-to-face version where players face each other in the same space, the team has also developed a remote version that can be enjoyed by players in remote locations using MUSVI Corporation's telepresence system "MADO." So far, trial sessions have been held at elderly care facilities and for children with visual impairments. In July 2023, Sony also held an exhibition event for the general public at Sony Park Mini and Sony Store Ginza. Many people have tried the game, and its circle of co-creation has expanded.

The initial incentive for XR Catch came from Teruhiro Nakagawa, a board member at PLAYERS Foundation, who participated in the project as the lead user with visual impairments. Mr. Nakagawa had mentioned that he wished he could play catch with his son. This single comment was the starting point for a new challenge: creating an inclusive game of catch that could be enjoyed by anyone, regardless of disability or age.
What began as an encounter with Mr. Nakagawa and a process of trial and error eventually expanded into an unexpected outpouring of support and sympathy, then into a vision leading to the future. We held a roundtable with the four members involved in the project.

XR Catch was developed as part of an Inclusive Design initiative. Can you tell us about the background to this?

NishiharaSony Group is working to improve accessibility so that all users can be themselves and share their excitement through our products and services. At the Creative Center, our part of this is working together with lead users, including people with disabilities and the elderly, to promote inclusive design initiatives that create new experiences and value.
The starting point for this particular project was a workshop conducted with lead users in July 2021. About five people, each with visual and hearing impairments, participated alongside designers, R&D engineers, and researchers from Sony. The workshop was quite productive, even though it was held online. We interviewed the participants about their daily routines and problems, and asked them to talk about "things they would like to do, but have given up on doing because of their disabilities."

TambataWhat left the greatest impression on me personally was simulating the experience of having a disability. The lead users gave us online assignments such as, "Go to the refrigerator and get a drink while wearing a sleep mask over your eyes." It was a very new experience for me.

NakagawaI myself still had my sight as a teenager. While listening to the various opinions expressed during the workshop, I remembered that I used to play baseball and watch soccer, and mentioned that I wanted to play catch with my son. At that point, I never dreamed that my idea would be the one they adopted.

Design that questions
the essence of catch

Amidst the many ideas and realizations that came up, what made you decide on turning the game of catch that Mr. Nakagawa mentioned into a project?

TambataSimply put, because it seemed the most difficult. When we began to think about how we could achieve this, the story kept expanding to include placing sensors around the space, developing new devices, and so on. That is why I was, conversely, intrigued by the challenge.
On the other hand, the question that emerged was, "What is the point of playing catch in the first place?" It's not a game of win or lose, but of the thrower and catcher keeping tempo, and of the satisfying feeling of repetition. We decided to extract these elements and redesign the act of playing catch itself.

NishiharaAs the project manager, I was looking at the entire process. Yet, as a team member, I still wanted to incorporate new technology into the project. But getting too caught up on the technology would raise the hurdles we faced all the more. It was good that we were able to discuss this point thoroughly and narrow it down to something simple.

TambataWe had a team of three engineers and two designers, but our focus was on coming up with something we could execute right away with our existing technology. We created a video of two people playing catch using CG and narrowed down our list of the game's essential elements. Smartphones were easy to use because they have many usable technologies such as sound, vibration, and acceleration sensors.
Another thing we emphasized was that there should be no time lag when throwing the ball back and forth. If there's even a slight lag in the sensation of catching the ball, this will inevitably increase the player's discomfort. We explored ways to make the process as burden-free as possible, in terms of both hardware and software.

And the first prototype was completed in November 2021. Since then, Mr. Nakagawa, you have been testing the game each time there was an update.

NakagawaYes. At first, I was surprised that my idea was adopted. At the same time, I could feel in the air that everyone was watching my reaction with great interest. (Laughs.) My first impression was, "Wow, this is just like playing catch." The fact that the ball has been reduced to sounds makes it easy for anyone to enjoy, regardless of whether or not they have visual impairments. I was impressed.

NishiharaI was relieved to hear Mr. Nakagawa say that, but at the same time, I saw new challenges. For example, it's difficult to press a small button while holding the smartphone and making a throwing motion at the same time. We had created a simple phone holder with Velcro to prevent the smartphone from flying out of the player's hand, but when we showed the prototype to the team in Sony Lifecare's nursing care business, we received feedback that it needed to be easier to attach and detach.^*1

KarasawaI learned about XR Catch through another project, where I was involved in designing a retirement home. While I felt that this would be a fun game for the elderly, I said to Nishihara, "If the elderly are going to use it, why don't you make the holder a shape that is easier for them to grip with less force?" That was the moment it was decided that I would participate in the project. (Laughs.)

*1 Sony Financial Group, “Testing the Use of Accessibility Technology in the Lives of the Elderly: XR Catch Trial Event at Sonare Atelier Kugayama (article in Japanese)

Updating the game
by incorporating feedback
from the elderly and children

So more and more collaborators joined the project, both inside and outside the Creative Center.

NakagawaThis came as a surprise to me as well. From the light-heartedness with which we spoke, more and more people began to cooperate with us. I was surprised every time I touched the updated prototype. I never thought it would turn out like this.

NishiharaI feel the same. Even if you develop something new, whether or not it will generate sympathy and lead to the next step depends on the project itself. In this sense, the trial sessions with the care facility residents and children with visual impairments were events that led to great discoveries.

TambataFor example, initially, when a catch was determined to be unsuccessful, it made a disappointing sound. However, after seeing the response at the care facility, we changed it so that everyone can always catch the ball, and when they catch it well, it makes a triumphant "great job!" sound. This change made it so that even the people watching them play could join the experience of their joy. The feedback we received from supporters and the audience revealed new challenges for us. I felt this was an interesting way to proceed with the project.

KarasawaThe children in particular were delighted, and they would move in unconventional ways, such as waving their hands in the air, as if to say, “This is how you throw a ball!” It was truly one surprise and discovery after another. At the same time, these helped us to make continuous improvements.

NishiharaPersonally, the event that was most rewarding to me was the one organized by Hiyoko no Kai (Japanese only), an organization that supports children with visual impairments and their families. We held this event remotely between Sony City Minatomirai and the Senzoku Gakuen College of Music campus in Mizonokuchi, Kanagawa Prefecture. What stood out to me was that some children with visual impairments did not know what playing catch was like in the first place. But when I demonstrated the game by saying, "I'm going to throw the ball to you in four beats from now," they immediately understood how to play it and were incredibly excited to play.

Response to the remote version that
connected strangers to each other

Can you tell us more about the remote version of the game that was introduced at Hiyoko no Kai, in which people in remote locations play with each other via MUSVI Corporation's telepresence system "MADO" (Japanese only)?

TambataOpportunities for face-to-face interaction were limited because of the COVID-19 pandemic, and we realized that the important thing about playing catch is who you play it with. So we started working on a remote version, but this brought its own host of issues. When using typical online conferencing tools, the video and sound lagged, and players couldn't hear the sound effects because environmental sounds were cancelled out to prioritize conversation. That's where the "MADO" R&D team came in. We talked to them, and they were able to make some adjustments to the existing system that solved our issues beautifully.

NishiharaIn the remote version, the lag was handled by the server and adjusted so that no discrepancy was felt by either side. This made it possible to create the sensation that both players were sharing the same space, even when playing far away from each other.

KarasawaThe question for the remote version was how to achieve the same back-and-forth tempo. The ease of pressing the button became even more critical. The button on the second-generation holder was operated by gripping and releasing the holder with all the fingers of the hand, similar to the action of throwing a ball. However, many people commented that they were afraid that the smartphone would fly out of their hands if they released their grip while throwing. So in the third generation, we changed the attachment method from Velcro to straps, and enlarged the button to a size slightly larger than the fingertips. We made sure to design the shape and materials of the holder so that any player could use it, regardless of their hand size, grip strength, or allergies.

After going through all this trial and error, you held a two-week public event in July 2023, connecting Sony Park Mini in Ginza with Sony Store Ginza, located about 200 meters away.

NakagawaI had no idea that the team was also producing a remote version, and it surprised me once again. I had never imagined that I would be able to play catch with strangers in remote locations, and with international customers who happened to be visiting from overseas, no less. (Laughs). I would love to do this with my son as well.

KarasawaWe had a great number of participants during the event period, and through their responses, I felt that we had transcended various challenges. First was the one we faced with Mr. Nakagawa's original wish to play catch with his son. And then the second was how to get young children who did not even know how to play catch to enjoy it. Further down the line, we found out that language barriers and physical differences, such as adjusting the game for elderly people who remain seated while playing, could be overcome. I felt that we had bridged many differences and created something that everyone could enjoy playing.

New prospects for communication,
connected by shared feelings

It seems like the more people respond to the game, the more ideas can be developed further to expand the project’s future prospects.

Tambata I feel that there is still more that can be done, even when it comes to the individual sounds. We have received constructive ideas from people who tried the game, such as, "I want to throw a curve ball," or "I want to feel the impact when I catch a ball." I believe that not being able to see the ball stretches players' ideas and imagination.

NishiharaI was really encouraged to see strangers having fun playing catch with each other in their own styles. A person with visual impairments who tried it said, "Playing catch can also connect the players' minds." I think XR Catch still has more potential, such as expanding it further as a communication tool.

KarasawaWatching people play catch without talking, I saw it as a form of communication that requires players to get a feel of each other that goes beyond words. If it eventually becomes possible to play this game using only a smartphone, without floor-mounted speakers or "MADO," it will be much easier to deploy in a variety of situations.

NakagawaEach person, whether they are playing or watching, feels something and expands their imagination through XR Catch. I think creating new value through the experience is exactly what the game is about. People from all walks of life have tons of fun playing it and can’t help but want to talk to someone about the experience. I think it is because we aimed to make the game playable by all kinds of people, not only by people with visual impairments, that it has become such an expansive communication tool.

TambataExactly. Once people experience it, they become a part of our project. (Laughs.) Starting with Mr. Nakagawa, to the people at the facilities who cooperated with us, to the children and their parents, to the people who came to our events.

NishiharaIt really was like that. (Laughs.) We'd love to keep working on the project with Mr. Nakagawa.

NakagawaWhat sort of developments will we see next? I'm really looking forward to it, too!

Discussion held on August 4, 2023, at the Creative Center