People
Aiming for an “Era of Realistic Reproduction of Remote Spaces before Our Eyes”
3D High Image Quality and Low-Latency Transmission Technology
Sep 29, 2023
The COVID-19 pandemic has radically transformed our lifestyles, and since then, “remote” interaction has become commonplace. In such times, our reality is supported by technologies such as video and telecommunications. Researchers Noriaki Takahashi and Yuto Kobayashi are developing technologies to realize a higher level of reality even between “remote” locations through “3D high image quality and low latency transmission technology.” What value will be created in a future where things can be communicated and shared across physical and psychological distances?
Profile
-
Noriaki Takahashi
Intelligent Video Technology Development Dept.
Content Technology Research & Development Div.
Technology Development Laboratories
Sony Corporation -
Yuto Kobayashi
Intelligent Video Technology Development Dept.
Content Technology Research & Development Div.
Technology Development Laboratories
Sony Corporation
Closing the distance to create new experiences
──Please tell us regarding the research that you are currently working on.
Takahashi：We are currently working on developing technology that allows users to feel as if distant spaces, people, and objects are in front of their eyes or “right there.”
The goal of our research is to create new experiential value through spatial reproduction technologies such as “3D high image quality” and low latency transmission technologies. This is not only advancing video technology but also delivering unprecedented experiences and creating new value. This is the motivation for our R&D.
──What kind of technology enables people to feel as if distant spaces, people, and objects are “right there”?
Takahashi：For example, telepresence system "MADO" connects different spaces via a large screen display and provides a communication platform where users can experience immersive interaction and can even feel a presence. The system has already been integrated into society, and Sony Bank has launched remote consultation services using MADO. I understand that medical institutions and local governments are also adopting MADO.
Telepresence system “MADO”
──Therefore, it is truly a social integration of technology that enables people to feel as if distant spaces, people, and objects are “right there.”
Takahashi：In addition, Sony has commercialized a “Spatial Reproduction Display” that enables the naked eye to view 3D images with a lifelike sense of reality. This method is called “Eye-sensing Light Field Display” and utilizes our proprietary vision sensor and eye-sensing technology, as well as our 3D high-image-quality technology. This technology is capable of continuously detecting the position of the viewer’s eyes to deliver optimal 3D images to the left and right eyes. The result is a reality that seems to be “right there,” something that cannot be experienced with conventional displays.
Right now, we are working on further advancing this Eye-sensing Light Field Display technology by developing a 55-inch vertical 8K-resolution prototype capable of projecting life-size human figures. This prototype made its first public appearance last year at Sony Technology Exchange Fair (STEF) 2022.
Demonstration at STEF 2022
──The very technology that is changing the definition of "reality".
Takahashi：We developed this technology with the aim of delivering images that allow users to experience a “reality” that is comparable to reality using real-scale, high-definition spatial reproduction technology and low-latency transmission technology. At STEF 2022, a demonstration was held using the 3D telepresence system that we developed, where visitors could experience a “handshake” with an idol who was in a remote location. Event staff who experienced this system commented that it was an instrumental technology, especially for interactions with overseas artists where physical distance is a common barrier.
Transcending distance to address social challenges
──What other potential applications do you have in mind?
Kobayashi：In addition to online communication, we are considering using the system for operational support in remote areas. The system makes it possible to provide services via video in places where it is difficult for specialists or experts to visit immediately, such as remote islands or deep mountainous areas.
At STEF 2022, we also presented demonstrations envisioning applications such as the remote control of heavy machinery at construction sites or the operation of robots and equipment in factories from remote control rooms. In recent years, the shrinking workforce has led to labor shortages in various industries, resulting in an increased demand for solutions such as remote control and remote skill transfer.
──What kind of impact will the developed technology have on remote operations?
Kobayashi：With remote operation using 2D video, it is sometimes difficult to determine depth. For example, if depth is unclear, even simple tasks such as operating a machine, picking up an object, and moving it can be time-consuming. By contrast, the use of 3D video transmission technology to view depth in three dimensions makes it easier to determine depth and enables more intuitive remote operation.
Another advantage is that the user can determine the size of an object. When an object is shown on a 2D display, the further back it is, the smaller it appears, and the closer it is to the front, the larger it appears, making it difficult to get a sense of its size. However, our system can provide information such as “a 10cm object is 1m away,” which is helpful for more accurate operation.
Demonstration of remote operation
Takahashi：Another area that we are considering is using the system for remote medical care. Currently, there is a growing shortage of doctors in remote islands and rural areas, posing a serious social issue. We believe that in cases where remote medical care is provided as initial treatment in these areas, more accurate patient information can be provided to doctors than with conventional video-conferencing systems.
Toward a future where images can express reality
──What motivated you both to enter your current field of research?
Kobayashi：I first became interested in 3D image technology during the so-called “3D craze” when I was in junior high school and high school. When I first saw 3D TVs and 3D game consoles, I was shocked by experiencing the sensation that I could almost touch the images with my own hands.
Although my current research is related to the physical distance of remote operations, I think that my original experience was the “emotional distance” between myself and the content, which was transformed by 3D imaging technology. For example, even if you are watching the same movie, when it is viewed in 3D, you sometimes feel as if the characters in the movie are closer to you than when you are watching it in 2D. It is interesting to be able to provide that kind of excitement through image technology. This thought inspired me to get involved in the actual development of such devices and create my own technology after entering university, and I joined Sony.
Takahashi： I joined Sony just around the time when high-definition televisions were introduced, and since joining the company, I have been working to improve the image quality of televisions, cameras, and transmission systems.
Currently, I am involved in the development of 3D image technology, and I believe that the potential for new technological development lies not only in the beauty of the images, which makes use of the expertise cultivated in the development of high-quality 2D images, but also in how high a degree of “realism” and “presence” can be achieved.
Kobayashi：At Sony, I can commercialize the results of my research and can propose them to society. I find it very rewarding to be involved in everything from research to output. I would like to continue to develop 3D imaging technology and make proposals that will be useful to future society.
Prospects are born from encounters
──In what ways do you think these technologies will change the society of the future?
Kobayashi：We engage in R&D while imagining the applications and users, but it is difficult to speculate at the R&D stage when it comes to the number of future users and whether it is likely to bring significant benefits to the company. For this reason, getting various feedback from people inside and outside the company is very useful at forums like STEF 2022. Another result of STEF 2022 was that we were also approached by people outside the company who were interested in using the technology in their business.
Takahashi：At STEF 2022, in addition to the “handshake event,” we prepared a “signage” demonstration by presenting a recorded video of an idol introducing her/his CD and placing the actual CD in front of the display. Those who experienced it commented that they would be inclined to buy the CD if it was presented in front of them with this 3D video. We have even heard of a person who has become a true fan of the idol and has gone to see a live concert. I believe that our technology not only will “save people from having to go to remote areas” but also will increase people’s desire to interact more with the “real” and appeal to the value of the “real,” just as live baseball broadcasts increased the number of fans who visited baseball stadiums in the early days of television.
