At Sony Computer Science Laboratories (Sony CSL), established in February 1988, researchers endeavor to create new research areas and paradigms, as well as new technologies and businesses for the common social good. One such pursuit is the Music Excellence Project, which holistically provides physical education and art education for young pianists.

In this project, Sony CSL senior researcher Shinichi Furuya asks the question "Is culture sustainable?" while aiming to enable musicians, who are the leaders of culture, to learn how to make the best use of their bodies efficiently based on findings from physiological and neuroscientific research. The goal is to free musicians from physical discomfort and psychological stress so that they can concentrate on the pursuit of sophisticating musical expression. This blog will introduce Furuya's thoughts on how we can help musicians break through their perceived “virtual” limitations and realize a society in which the culture of music can evolve to new heights.

Outline of the Academy Program in the "Music Excellence Project"

A comprehensive art and physical music education program for teen pianists

The Academy Program is a comprehensive program in physical and art education for teenage pianists. This program, which supports the growth of artists involved in musical activities, aims to free pianists from mental and physical challenges as early as possible and enable them to focus on musical expression. This is the world's first comprehensive piano education program that pursues sophisticating musical expression and leads to continuous improvement in both skills and mind.

From my own experience, I wish a future society without any physical troubles among musicians

― Why did you start this research?

― Why do you think physical education going without scientific basis?

One possible reason is the belief that as long as you have the patience, you can do whatever you want. However, although I myself am patient enough, I proved that it was not enough to continue practicing patiently so as to play as imagined.
Another reason is that we tend to mix art and science in an inappropriate manner. Researchers have tended to say like "These are the proper ways to move your body" without fully considering its connection to music, or, “You can create stronger emotion with this kind of musical expression.” However, these words let musicians think, "Well, what does a scientist know about music and musical performance?” It is the artist's job to think of and choose musical expression, and it is our job as researchers to figure out how to create that expression using the brain and body. I think that physical education for performers is still in the process of being scientifically developed because we don't have a comprehensive understanding of these two processes.

― It seems that the students accept like “if you, a former pianist, say so, then it must be true”.

Maybe so. However, I don't think it's a good idea to rely on that. For example, if an adult suffering from sleep reads a sleep-aid book that says, "This book was written by a famous professor at XX University," they may immediately feel that they can get a good night's sleep.
On the other hand, our students participating in my Academy Program are very honest and have experienced dramatic changes in their performances in my coaching, and therefore, they feel that they can trust what I mention. Children probably don't even know my job title or any details of my research. I like that because it's fair without any bias in front of the students as one teacher. It's my job to make results and support what the students want to create, and I always enjoy feeling fear of having risks of losing their trust if I don't keep doing so.

― How did you become a researcher at Sony CSL?

My father was a heavy user of Sony products, so I have been a big fan of Sony ever since childhood. In 2009, I had the opportunity to work with a world-renowned pianist Lang Lang at an event called "Let's enjoy Science and Music" at Sony Explorer Science in Odaiba, Tokyo (now closed), during which I came to know about Sony CSL and was encouraged to check it out by a number of Sony people. Eventually, in 2017, I applied for the open position and became a researcher at Sony CSL.

A new music education that combines art education and physical education

― Please give us an overview of your project to create a new form of music education.

The Music Excellence Project has been running its Academy Program since July 2020 to provide music education for teenage pianists . There are many academies designed for pianists around the world, but only ours seriously incorporates physical education into the program. However, I believe that a core of music education is art education, so we could fortunately invite a world-known pianist and educator, Prof. Dina Yoffe, as a musical director of the academy. To make a synergy with physical education, our assistant teachers, who are all professional pianists themselves, give monthly lessons on musical expression and others.

When learning expression from a pianist teacher at conventional lessons, many of the students have to overcome the challenge as to how to create that musical expression using their body. They understand “what” they want to accomplish, but don't know “how” to realize it. At that stage, I step in to coaching and say like, "your teacher told you to express in this way. Do you know the appropriate way of moving the hand to create that?” While communicating with the student, I sometimes take video of the student's hand movements and show them back in slow motion.

After learning how to use the body in this way, we return to the first point and ask, "Have you been able to express what you wanted to express?" If they haven't achieved their musical goal, then we try again using different ways of moving the body.

― What technology do you use in the Academy Program?

We use a "multimodal sensing system of skills" that uses various sensors and cameras to capture and analyze the movements of the pianist's body. This system allows the pianist to "overcome the limits of their abilities" in two ways: by being able to visually magnify parts of their body that they cannot see while playing, like a magnifying glass, and by creating five senses outside of their own body.
Not only can this system be used in real time during the lesson, but it can also be used for post-lesson review. When I was taking lessons, I could only take the audio of recording lessons with a Walkman® or DAT and review by listening back while looking at the sheet music after returning back to home, which is a sound-only review, but with this system, we can save both the sound and the physical information as soon as they are created, so that the opportunities for optimal learning do not decline over time against memory decay.

― I heard some students have been immediately able to play music that they previously couldn't play even after practicing for hours.

Yes, that is true. There are limits of what humans can perceive, especially when it comes to fast movements like playing the piano. However, if we can completely capture data using the power of technology such as touch sensors, we will be able to exceed the limits of our innate perception, and I believe that this cannot be achieved by traditional repetitive practice alone.

― How do you share your expertise with your assistant teachers?

This is a very important topic. One purpose of the academy is to educate the students, while the other is to provide educational opportunities for the assistant teachers. Even if they haven't learned much about teaching methods at music conservatories while they were students, many pianists often have to suddenly turn to the teaching side immediately after graduation.
At the academy, assistant teachers are able to learn physical education in a way that connects to their own experiences by witnessing various physical and musical changes in their students. The assistant teachers have mentioned that this process has improved their teaching repertoire.

On the other hand, one has to be careful not to concentrate too much on physical aspects such as body movements. For example, it is important to be able to move unconsciously so that you do not have to think like "right foot first, then left foot," when going down the stairs. If you are too much aware of it, you may not move at all. Therefore, I don't say 90% of what I want to say and try to say only 10% through intensively summarizing, and I also ask the assistant teachers to keep it in mind.
Another point is to correct the "cause" instead of the "phenomenon". For example, it looks strange when you play the piano with your shoulders raised, but it might be okay if you don't have any physical or musical problems along with it. As long as music sounds fine, I think saying “your shoulders are raised” is just as meaningless as saying “your hair is disheveled.” However, if this awkward shoulder posture causes musical issues, then we have to solve that problem.

To contribute to evolution of musical culture

― Will even piano teachers be able to provide physical education if they understand your system?

We have made our best effort so that it comes true. In general, people who teach piano are experts on music, but they are not experts of the human body. There are two advantages of having such teachers use our system.
First, teachers can learn the relationship between the body and sound. Second, teachers don't have to waste lesson time for things irrelevant to music expression. While it is necessary to teach the essence of the culture of music, such as "Why should I make this sort of sound here?", I do not think it necessary to devote piano lesson time to teach like, “Why isn't the sound not rich enough?” – “Because the palm is collapsed.”

― If small kids learning to play the piano participate in the Academy Program, could they achieve a performance level that no one has ever reached?

That is our ideal. Nowadays, it takes a lot of practice time to learn one way of expression, so the choices of expression tend to be limited. However, if you take advantage of educational program we provide from an early age, you would be able to produce a wide variety of expressions and apply them appropriately.

Our system cannot directly have musical culture evolve, but I think we can push up the bottom line of the culture by promoting new education using this system. It would be interesting to see pianists who choose musical expressions that had been previously unimaginable, and if that doesn't happen, it would be a pity to see a lack of diversity in our culture.

― People say that everyone who participated in the program has improved considerably, but how do you see it from your perspective?

I found a lot of changes in ways of practicing for all students, which is a real growth. Previously, they would try to achieve their goals through repetition of practicing, but now they can hypothesize, "If I do it this way, it will work," and can test the hypothesis by themselves to some extent.

Some people continue to make the best efforts, but don't experience any improvement that they expect. Then they start to consider it as their limit of the talent. I call this “the virtual limit.” As a teacher, I often notice that students can improve more if they practice in a different way from what they do. Fortunately, students learn to discover ways to break through the problems by themselves and continue to break through their virtual limits. I can't teach next to them all the time, so I would be happy if they can break through those limits on their own, and if there will be an increasing number of people who will inherit these methods.

Toward breaking through the limits of musicians' creativity

― Do you have any plans to expand the Academy Program to the public by putting it into practical use?

That is the challenge for the future. In order to put it to practical use, it is necessary to reduce the cost of the entire system and to increase its stability, and above all, it must be possible to have physical education provided without me. In that sense as well, the assistant teachers play vital roles.

Expert pianists who work hard and still struggle with going higher than a certain level is a relatively small number of people, so breaking through that limit is a challenge from a commercial perspective. Also, there are only a few people who take such issues seriously.
However, as there is a well-developed system to support top athletes, I would like everyone to understand the significance of supporting top artists as well without leaving them alone. I believe that through such support for artists, culture will be passed on to the next generation sustainably and will develop even further.

― Would you also consider collaborating with outside partners?

Yes. It's important to have more outside partners, and I'm very happy to have so many people be a part of a project that I started from scratch. We also hope that our music education will be a showcase of how technology can contribute to art within the Sony Group that is working on developing cutting-edge technology. I think there are many fields where we can collaborate, and also would like to make more connections available within the Sony group.