Cutting Edge
Eye-sensing Light Field Display
Delivering 3D Creators' Visions to Customers the Way They Intended
Sep 30, 2020
3D displays have great potential for various applications, such as showroom displays and entertainment. Conventional 3D displays, however, suffer from issues such as low content resolution, the hassle of wearing 3D glasses, and 3D motion sickness. The Eye-sensing Light Field Display (ELFD) developed by Sony is a new type of 3D display that has the potential to solve these issues. By working together with creators to develop not only the content but also the production environment, we aim to popularize this system in a wide range of fields.
Profile
-
Kazuki Yokoyama
Tokyo Laboratory 09
R&D Center
Sony Corporation -
Takayuki Kurihara
Tokyo Laboratory 01
R&D Center
Sony Corporation -
Takaomi Kimura
Tokyo Laboratory 10
R&D Center
Sony Corporation
-
Ban Kawamura
Product Design Div.,
TV Business Group
Sony Home Entertainment
& Sound Products -
Thaisa Yamamura
Business Development
Sony Electronics Inc.
Delivering a "Spatial-Reality Experience"
with Sony's Unique Technologies
──What kind of display is an ELFD?
Ban Kawamura：ELFD is a display that allows you to see objects in 3D volumetrically, as if they were right in front of you. The realism remains the same even from different angles. It allows you to experience high-definition 3DCG (3D Computer Graphics) images with the naked eye, and we expect this technology to be applied in a wide range of applications such as monitors for 3DCG creators, showroom displays, and entertainment. For example, before creating a mockup of a small product like a remote controller, you could check the volume, texture, and appearance with ELFD.
A 15-inch diagonal prototype was exhibited at CES2020. As long as the content fits within these dimensions, it can be displayed as a high-definition, three-dimensional object.
──What kind of technologies are used?
Kazuki Yokoyama：ELFD uses three of Sony's proprietary technologies: (1) High-speed, high-accuracy, real-time sensing technology, (2) Real-time light field rendering technology, and (3) High-precision 3D display technology.
First, the high-speed, high-accuracy, real-time sensing technology. In order to make the viewer feel as if the object is real, it is necessary to continuously display the correct image and viewpoint, based on the position of the user's eyes. ELFD uses Sony's unique high-speed vision sensors and face recognition algorithms to achieve accurate eye detection with minimal delay. The positions of both eyes are calculated in real time, not only for the horizontal and vertical planes but also for depth.
Next, the real-time light field rendering technology. Based on the user's position, ELFD calculates how the object would look through the user's eyes if it were placed inside the display and converts this information into a sort of deceptive picture on the display's surface. In combination with micro-optical lens technology that constantly delivers images for both eyes, a light source image is emitted from the display panel. Since this image is created in real-time with high speed and high accuracy, the correct viewpoint always reaches both of the viewer's eyes, making them feel as if the object is actually there.
High-speed, high-accuracy, real-time sensing technology
Realtime Light Field Rendering
Takayuki Kurihara：Finally, there is high-precision 3D display technology. Although there have been other 3D displays that do not require glasses, they could not accurately present 3D images due to low resolutions and crosstalk between the eyes. The ELFD display makes full use of Sony's highspeed vision sensor with a unique micro-optical lens that utilizes eye tracking, significantly reducing crosstalk compared to conventional 3D displays.
High-precision 3D display technology
──Please tell us about Sony's previous 3D display development prior to the development of ELFD.
Yokoyama：There was a time about ten years ago when 3D was all the rage. At the time, Sony was developing a 3D TV with 3D glasses. However, while the 3D TVs initially attracted a lot of interest, they did not take hold in the market because some people found it difficult to see sterically, and some even felt sick while watching them.
A few years later, we began considering a new type of 3D display that incorporated binocular parallax adjustments. At the time, Kurihara-san's team was developing an extremely high-precision naked-eye 3D display for medical use, and when I saw it, I felt that it had potential. By applying this display technology and examining the optimal placement of the display panel and current viewpoint image, we were able to develop the prototype of ELFD. We introduced this technology both at our internal company-wide tech exhibition and widely at CES, and unlike with the previous 3D TVs, most people experienced the benefits of this technology. The factors that allow viewers to perceive object depth are the presentation of a constant, distortion-free binocular parallax, the installation of side walls that provide actual comparison cues to the casing to promote fusion, and the presentation of a low-latency motion parallax.
Faithfully Reproducing the "Creator's Intent"
──What was important to you during the development?
Takaomi Kimura：The main concept behind the development of ELFD was facilitating and helping to bring the "Creator's Intent" to life. This technology is not only impressive on a technical level, but it's also a vehicle through which creators can express their ideas. Therefore, during the development process, great emphasis was placed on dialogue with creators. We wanted to allow creators to produce ELFD content with tools they are familiar with, so we created a software development kit (SDK) that supports Unity and Unreal Engine, two of the most widely used platforms in the 3D content production field. We began distributing the α and β versions of the SDK to creators both inside and outside the company early in the development process so that we could get feedback and make improvements. This allowed us to understand their requirements in terms of accuracy and performance and gave us a chance to create a better product together.
Content production process using tools
──What was the reaction of people who experienced ELFD?
Thaisa Yamamura：At CES 2020, we had a great turnout, with people queuing up to try ELFD. We received a lot of positive feedback, as well as excellent suggestions that we had not thought of.
And at InfoComm 2020, the largest professional audiovisual trade show in North America, we won the AV Technology Best of Show Special Edition. Collaboration with internal and external partners helped expand applications of our product. That's the beauty of bringing new technology to the world and seeing people use it in various functional ways to improve their lives.
Exploring New Possibilities for 3D Content with Creators
──What are the prospects for ELFD in terms of product design and R&D?
Kawamura：To maximize the potential of ELFD as hardware, we need content that highlights that potential. To that end, we need to improve the content production process for creators. First, we need to develop optimal software and tools for content production, such as the SDK, and then provide ELFD displays for checking content. By doing this, we can ensure a minimum ecosystem for creating, viewing and distributing the content.
Yokoyama：Currently, ELFD only works by limiting the display to a small rectangular parallelepipedal area and limiting the number of viewers to a single person. Therefore, you cannot see accurate three-dimensional images if the content protrudes from the screen, or if multiple people are viewing the content at the same time. We are working on these problems. One of the difficulties in making a larger system is that it reduces visibility as the image depth increases. Here, we aim to make a larger system while maintaining a sense of freshness and sharpness. In terms of the number of viewers, our first goal is to achieve the same viewing experience in a two-viewer situation. There are many use cases where two people watch the same content together, so achieving this will allow for a wider range of applications. In addition, we are working on improving real-time sensing, support for displays with higher frame rates, and faster rendering to ensure that the subject remains in place even when the viewer moves quickly.
──What do you want to achieve in the future?
Kimura：XR (AR/VR) has great growth potential in entertainment applications and many B2B fields, even in the "With/Post COVID-19" era. We believe that creators will play a key role in coming up with new use cases, so we will continue developing technologies that allow creators to faithfully reproduce and showcase their ideas.
Yokoyama：I have long dreamed of creating a visual experience realistic enough to be mistaken for the real thing. We would like creators in a wide variety of fields to produce high-quality content so that we can spread this technology and make it a permanent part of people's daily lives.
Kawamura：In the future, we are looking to apply it to the education sector and VR display devices, as well as expanding it into fields like B2B solutions and content distribution. Since this is a new field with a wide range of potential business possibilities, we would like to get the whole Sony Group involved with the world of ELFD.
