The Eye-sensing Light Field
Display delivers
realistic
3D images to the naked eye
Click here for the description of the video content.
Display for enjoying realistic 3D
space representations
The Eye-sensing Light Field Display reproduces a realistic 3D image in space and allows the user to view the 3D space from any desired angle. Besides high spatial resolution, the elements of depth representation and spatial fusion are added, making the user feel as if the displayed object is actually in front of him or her. This feature sets this display far apart from conventional 3D display units.
Click here for the description of the video content.
Click here for the description of the video content.
A 15.6-inch 4K commercial model of the spatial reality display created using the Eye-sensing Light Field Display technology, the ELF-SR1, is now marketed in North America and China after being launched in Japan in October 2020.
We are also developing a 32-inch 8K model, which is twice as wide vertically and horizontally while maintaining the same pixel density as the ELF-SR1. Its prototype was on special exhibition at “Sony presents Dino Science, The Dinosaurs of Laramidia 2021@YOKOHAMA” held in the summer of 2021.
A system that selects and
reproduces only necessary rays
and three core technologies to
realize this system
One of the key elements of the Eye-sensing Light Field Display is the “light field.” The space in which humans exist is made up of various light sources. When we think of this space as the world that humans see, it can be represented as a light field consisting of numerous rays of light.
However, there are so many light rays in the space that it is impractical to develop a display that reproduces all those rays. For the Eye-sensing Light Field Display, therefore, we have developed a system that selects the light rays necessary for the viewer from a massive amount of light ray information and reproduces only the selected light rays. This allows a high-precision light field to be viewed from any desired angle.
Fundamental technological
principle of the Eye-sensing Light
Field Display
The system first decides on the space to be displayed and then selects only the light rays that reach the viewer’s eyes from there. This requires a sensor that detects the positions of the viewer’s eyes quickly with high accuracy and tracks the movement of the eye positions. Next, the system projects the selected rays to a naked-eye 3D display plane and renders the space by calculating the value of each pixel of the display directed at the left and right eyes in real time. By selecting the rays that reach the viewer’s eyes correctly in real time, the system reproduces a 3D space realistic enough to make the viewer feel as if it is actually there.
We are developing the following
three core technologies to deliver
a natural,
comfortable viewing
experience.
Eye-sensing
In order to make the viewer feel as if an object is really there, it is necessary to continuously display a viewpoint image that is correct when seen from the viewer’s eyes. To meet this challenge, we use Sony’s high-speed vision sensor, as well as the Eye-sensing system employing high-accuracy face recognition algorithms, for continuously detecting the positions of the viewer’s eyes with accuracy and tracking their movement in real time. The high-speed vision sensor captures an image for every few milliseconds to detect the position of the face and identify facial feature points such as eyes and facial contour. By estimating the 3D shape of the face from this feature point information, the system calculates not just the distance to the face but the rotational direction of the face as well, tracking the various movements of the viewer. The camera system including the imaging lens is optimized, enabling viewing from any desired viewpoint within a range spanning from 50 degrees horizontally to 60 degrees vertically while achieving a natural, smooth motion parallax. The whole process from imaging to rendering is executed with low latency, which minimizes crosstalk between viewpoint images as well as the image delay that can cause motion sickness.
We have developed face detection algorithms using deep neural network (DNN) technology and optimized them to suit high-speed vision sensors. This ensures a comfortable and stable viewing experience insusceptible to noise during imaging and blurs due to the lighting condition. Moreover, the viewer’s face can be detected normally even when he or she is wearing a face mask.
Real-time Light Field Rendering
A light source image emitted from the display panel is reproduced in real time based on the viewer’s eye position information. Reproducing the light rays limited to the viewer’s viewpoint position as a perspective projection image according to the above-mentioned fundamental technological principle allows the 3D space to be viewed from any desired angle as with the real space. While this perspective projection image output on the display plane is distorted, it is presented so that it looks undistorted from the viewer’s position, creating some sort of illusion. This is one of the visual effects that makes the viewer feel as if what he or she sees is really there without being conscious of the display.
In recent years, there have been significant improvements in the rendering quality of game engines. Their highly realistic expressions have come to be appreciated not just in the game industry but also in other fields where realistic image representation is required, such as video production and architectural design. These engines enable real-time rendering of realistic 3D images. In addition, since creators and designers can work with familiar tools in familiar environments, the game engines are also expected to promote the use of existing contents and spread to new applications.
Micro-optical lens
A unique micro-optical lens is used to deliver images generated in real time to both eyes, allowing for a natural, high-definition naked-eye 3D viewing experience. Conventional naked-eye 3D displays using a lenticular lens or parallax barrier need to present many viewpoints to cover a wide viewing range, resulting in degradation in resolution and decrease in image quality due to crosstalk between viewpoint images. By contrast, our micro-optical lens assumes viewpoint position sensing and real-time light ray reproduction algorithms, which enables optical design dedicated to image generation for the two left and right viewpoints. Also, degradation in resolution and crosstalk between viewpoint images can be minimized by providing sufficient pseudo viewpoints and exerting control to selectively display images for the two left and right viewpoints.
This display control does not work if the micro-optical lens and the flat panel display become misaligned when they are stuck together. We have therefore developed a manufacturing technique for sticking these devices together to an accuracy of several tens of micrometers, as well as an adjustment system for detecting slight misalignments after manufacturing. Misalignments are corrected during the real-time image processing based on the adjustment information, which further improves the quality of the 3D image.
An unprecedented 3D
viewing experience
We have explained our three core technologies so far. Other technologies we have introduced include cameras, displays, a calibration system capable of highly accurate adjustment based on the viewer position and correction signal processing to further reduce crosstalk. Our software and algorithms use technologies developed for VR head mounted displays as well. We also consider ergonomic design important. The display is positioned obliquely to make the viewer unconscious of it, and the structure is laid out in such a way as to make it easier to visually recognize the 3D space.
By collaborating with various R&D projects within the Sony Group and integrating the technologies mentioned above, we have achieved an unprecedented 3D viewing experience.
Applicable not only in
entertainment but also in
education,
business and
healthcare
The Eye-sensing Light Field Display is limited to personal use (use by a single person). Integrating a broad range of unique technologies, this display enables realistic spatial representation superior to conventional 3D display technologies. It allows the user to enjoy realistic three-dimensional representation of 3D contents casually, which enable new value of entertainment. Currently, Volumetric Capture technology to capture live-action 3D space is being increasingly used for video production, viewing with AR/VR headsets, etc. The Eye-sensing Light Field Display is anticipated to come into wider use as a display that allows you to view 3D videos in three dimensions as is.
The Eye-sensing Light Field Display is also expected to be applied not only in the field of entertainment, such as games and films, but also for many other professional purposes including education, business and healthcare. Since it enables an intuitive grasp of the spatial layout, this technology is projected to be useful in business settings as well as for presentations. Furthermore, we think that, when combined with real-time 3D video capturing and transmitting technologies, Eye-sensing Light Field Display will make it possible to communicate and work with people at remote locations as if speaking with them in person.
From a futuristic perspective, we are also paying attention to holography as a technology to represent depth in a broader range. Using a bench-top prototype of a phase modulation SLM, we have confirmed that imaging with multiple focal plane works effectively in color holograms. We have also successfully conducted a demonstration experiment of a deep neural network-based algorithm that enables high image quality and real-time calculation. The image quality, amount of calculation, viewing range and viewing field are the challenges we need to address to achieve more natural 3D representations. We will keep working on this technology, paying attention to advances in devices and signal processing.
After a bumpy road comes
great joy
Our eye-sensing light field display is a good example of how R&D efforts become products and services. We got many positive feedbacks on our innovations from the customers. These supporting voices further motivate us to carry on. In R&D, a series of trial and error is often required for making a product. The developing process also requires people with different expertise. By working with our colleagues, collectively we tackle the technical issues from many different aspects to refine our solution. This is a process with a lot of fun. I hope that we can share with you this exciting experience.
Researchers
Koji Aoyama
The Eye-sensing Light Field Display is a mix of various cutting-edge technologies. We’ve polished our solution with the help of experts in many different fields. Creating a new high-performance technology is not an easy task. We bump into one problem or another almost every day. To succeed, it is vital to be brave enough to dream on something impossible.
Shihhao Wen
I am very excited to have a chance to create things that I thought existed only in science fictions. There are two things about R&D that motivate me the most. One is to contribute to society through innovation, and the other is to always learn something new through the work. I also believe that a job needs to be fun. I am sure that you can find something fun in Sony’s R&D, otherwise you can always make one by yourself here.
Masamoto Horikawa
I have an ambition on creating a new culture of 3D video experience. I feel obligated to always make excellent products in order to finally achieve this goal. Sony does business in diverse areas, and each department seems to have its own organizational atmosphere or culture. I want you to know this, because personnel exchange and transfer between departments are not rare.
Ruri Oya
How much can we wow by offering new types of experience? That’s what motivates me. Sony’s R&D Center is committed to progress technologies that can change people’s life in 5 or 10 years from now. We have a work environment that’s ideal for people who want to shape the future of our generation.
