Cutting Edge
Volumetric Capture for Movie Quality
Virtual Production
Nov 18, 2019
Using spatial reality technology for movie production
Volumetric Capture technology refers to creating 3D CG from images taken by multiple cameras, and using this data to produce video that looks like it has been shot at a real location. Currently, the Sony’s R&D Center, Sony Pictures Entertainment, and Sony Innovation Studios, which specializes in virtual production, are working together to produce high-quality video that can be used for movie production.
Profile
-
Junichi Tanaka
Tokyo Laboratory 09
R&D Center
Sony Corporation -
Yosuke Kaji
Sony Innovation Studios
Sony Pictures Entertainment
Improving each other’s technology and aiming for a completely new visual expression
──How are Sony’s R&D Center, Sony Pictures Entertainment, and Sony Innovation Studios connected?
Junichi Tanaka：Sony Innovation Studios was established on the Sony Pictures Entertainment premises in Culver City, the U.S. in 2018, and is a special studio that engages in virtual production using various technologies including the volumetric capture technology. The visual production team at Sony Pictures Entertainment expressed their interest in Sony’s R&D Center’s volumetric capture technology before Sony Innovation Studios was established. At the opening ceremony for Sony Innovation Studios in June 2018, we revealed the real-time volumetric capture system that we developed together with Sony Pictures Entertainment.
──What did you demonstrate at the opening ceremony?
Tanaka：We put together a prerecorded 3D image of a presenter and mixed it with the live presenter, giving a real-time demonstration that made it look like there were two of them. The guests were astonished by the photorealistic image quality.
Real-time volumetric capture demo revealed at opening ceremony for Sony Innovation Studios, in June 2018 (no sound)
──Since then, it seems that Sony and Sony Innovation Studios have continued working together.
Tanaka：After the Sony Innovation Studios opening ceremony, we at the R&D Center put forward the idea of introducing volumetric capture technology to the Japanese music industry, while Sony Innovation Studios pursued even higher image quality, and we have continued aiming for new heights with this technology. Recently, we have begun attempting to place a character filmed using volumetric capture technology into a virtual space created with the unique technology of Sony Innovation Studios. The spaces that they create have such high image quality that they would seem like as if it was shot at real locations to anyone who did not know, so it was expected that the image quality of video we have shot couldn’t meet their quality. In order to realize better image quality, we decided to have some of our best engineers visit the Sony Innovation Studios site and experience actual production, aiming to engage in technical cooperation alongside creators.
More real and simple. Aiming for applications in movie production
──What kinds of strengths has Sony brought to the table during this cooperation?
Yosuke Kaji：The main strength of our volumetric capture technology is the rendering technology that uses 3D computer vision to create video even from viewpoints where cameras are not installed. If you try to create 3D data from live-action video, no matter how refined the process is, shapes will differ from reality. Adding color to this only makes it look even more unnatural and CG-like. As a result, we introduced Sony’s unique photo-realistic rendering technology that allows full use of captured video to create natural images that look like they were shot with a conventional camera. We have not yet been able to produce results that truly look like they have been shot with a camera, but we will continue developing our technology and improving the quality to the point where it can be used for movies.
──We heard that you have also been planning something in video editing work.
Kaji：We are currently working on technology to automatically turn previously captured images into 3D models. Highly accurate calibration and alignments are required to create high-fidelity 3D models. The R&D Center has now succeeded at automating this process and placing emphasis on real-time element at the same time. This allows large reductions in the time and effort it takes to edit. In movie production at Sony Innovation Studios, there are many occasions where the artistic nature of a work requires manual processing which takes a considerable amount of time. On the other hand, virtual production requires the handling of a huge amount of data than usual, which takes a long time to process and edit, and Sony’s automation technology can be useful in improving production efficiency.
(Left) 3D CG model generated by Sony’s rendering technology / (Right) Recreation of image that appears to have been taken on location
Making new video experiences more familiar
──What kinds of developments and possibilities can we expect?
Tanaka：Virtual production allows the remote creation of images that appear to have been taken on location. We believe that by applying this technology, we will be able to create an experience that makes people feel close to each other at any time, even when they are far apart from each other. Our ultimate goals are to make technology more familiar, and deliver this experience directly to customers. Especially in light of the COVID-19 pandemic, we think that the importance of virtual production is steadily increasing. By continuing to fully invest R&D technologies, and establishing a new video production environment along with the Sony Innovation Studios, we hope to contribute to the movie industry.
Kaji：We believe that Sony Innovation Studios provides the optimal environment to pursue the highest picture-quality volumetric capturing. As we continue striving to realize better image quality while collaborating with partners in various fields, we believe these activities will lead to the creation of new experiences.
- *For another article on volumetric capture technology, please visit the link below where we introduce "Volumetric Capture Technology That Goes Beyond Omnidirectional Visualization."
- *For another article about Sony Innovation Studios, please see "Utilizing the latest technologies and getting closer to our creators at Sony Pictures—With a vision of One Sony" at the link below.
