Cutting Edge
Voice 02
The Possibilities of Virtual Production Will Revolutionize
the Film Production Site
Jul 4, 2022
Profile
-
Haruyasu Yamazaki
Advanced Production Dept. 1, Technology Div., Sony PCL Inc.
Sony’s virtual production excels at real-time performance
The KILIAN’S GAME Project was created as an effort by Sony Group’s cross-organizational Content Technology Strategy Committee to verify the effectiveness of new technologies for content production. Two production teams, one in Japan and one in the US, were responsible for shooting the film. Sony PCL, which handled shooting on the Japanese side, installed a virtual production studio at their headquarters in 2020 and began testing the technology. In 2021, the studio was relocated to TOHO STUDIOS (Tokyo), where verification tests and business activities continued. With the results and expertise obtained from the tests, in February 2022 a permanent production studio was installed within Sony PCL’s new creative base, Kiyosumi Shirakawa BASE, marking the start of a new business.
The shooting for this project was performed at the ”Kiyosumi Shirakawa BASE”. The effectiveness of virtual content production was verified. A total of 30 staff members, 15 from Sony PCL and the rest from partner companies, were responsible for the shooting. I worked as a producer for the Japan production unit.
The key virtual production technologies are the large-scale LED display and the function that links the camera’s position with the 3DCG backgrounds on the screen to display them in real time. The LED display used in this project is the Crystal LED B-series, developed by Sony specifically for virtual production. It features a 1.58mm pixel pitch size, high resolution , high brightness, high contrast, and a wide color gamut. Low reflection processing has minimized the projection of illuminating lights, making it the optimal display for backgrounds. In-Camera VFX adds a tracking system capable of precisely capturing position information of the cameras used for filming.
The In-Camera VFX mechanism utilizing the Crystal LED
First, steric background data is created using 3DCG, and a background image is displayed on the Crystal LED. The image and the actors in front of it are filmed together by the camera. The camera’s position and angle are measured by the camera tracking system’s infrared sensor, and based upon the measurements, the Unreal Engine (a game engine made by Epic Games, Inc.) renders CG in real time. This allows the background’s appearance and depth to change according to the camerawork, making it look as though the movie is being shot at an actual location and allowing for more realistic expressions. It also lets filmmakers confirm the film footage composed of the background and the actors in front of it on the spot in real time.
This real-time functionality is the defining feature of Sony’s virtual production. With ordinary systems, there is a delay between the camera and the background’s movements, which causes misalignment of spatial recognition between the background and the real objects.
However, the Sony’s virtual production provided by Sony PCL has reduced latency (response time) to just a few frames to limit delays. Thus, backgrounds matching the camera operator’s chosen angle are displayed immediately, enabling them to shoot footage closer to their original intention.
Direction must not be hampered by technological limitations
KILIAN’S GAME, a collaboration between Japan and the US, was shot by teams working in two separate locations.
In that context, prior planning and information sharing between the teams were extremely important when it came to utilizing virtual production. We had to ensure there would be no gaps between the footage shot by the two teams by sharing the effective angle information particular to virtual production with the camera operators located in the US beforehand. To achieve this, the Japan team carefully reviewed the shooting plans for each scene shot with virtual production. They shared diagrams showing where cameras would be placed and how they would shoot. Because the coordination at this stage would greatly affect the progress of the production, the two teams held weekly meetings to exchange information.
The actual filming plan
For the actual production, one of the important themes we kept in mind was that “Direction must not be hampered by technological limitations.” We always strived to utilize virtual production in a way that would ensure the director was able to shoot the film as intended.
To give an example, one of the director’s production plans had an early morning room scene. With virtual production, it can actually be difficult to illuminate dark scenes. Despite the challenging situation, we did not want to change the direction due to this, so we devised various methods to resolve the problem, including carefully designing the lighting for the filming and doing post-processing work.
Another hurdle was the production of flames. Naturally, using actual fire inside the studio had to be avoided . For this project, we tried depicting flames within special background CG unique to virtual production. Not only was it difficult to create realistic flame CG, but we also needed the Japanese CG designers to cooperate closely with the American team responsible for producing backgrounds and sets in order to portray the flames in a natural fashion. By remotely sharing the testing situation with the staff from FuseFX, the American company which produced the background CG and making various adjustments, we were able to realize realistic flame effects. The experience with special effects we gained through this endeavor is a very valuable asset for us.
The biggest merit is enabling creators to share their vision in real time
Regarding the merits to filmmaking brought about by utilizing virtual production, first, it enables a remote reproduction of the filming environment, so it is expected to effectively remove time constraints and reduce staff and equipment moving costs. Second, it provides a safe filming environment with background CG, such as the aforementioned flame production, and for scenes taking place in dangerous situations like the edge of a cliff. Additionally, for the settings of Western-style houses, like the scene for our current project, it allows for advanced realism by closely linking the camera angle’s movements with the background, which is extremely effective for recreating sets located far away.
I believe the biggest merit is the ability for creators such as directors, camera operators, and actors to share and confirm their vision on the spot. For instance, in the regular post-production process (the work taking place after the filming), the background CG is added after the shooting, so filming takes place in front of a green screen, and the actors must perform while imagining a background they cannot see. The same is true for the camera operators. The final vision can only be confirmed after the processing is finished. Because virtual production allows the crew to review the finalized footage in real time, it excels in making sure there are no perception gaps between creators. Thus, I believe it is a method that contributes to the production of better content.
For the actual shooting, detailed pre-production (preparations for the shooting) allowed us to accurately recreate the environment of an actual Western-style house in Log Angeles, so the director, camera operators, and actors were all capable of sharing a common vision. In particular, the theme the leader of the Japan unit had for his vision was based on Japanese culture, which could only be recreated by a Japanese crew filming in Japan, and these tastes were well reflected in the footage by the remote production between Japan and the US.
Revolutionizing existing film production styles
From the creators involved in the production, we received comments such as “The camera interlocking latency was lower than other systems, and the backgrounds moved as I expected, so it was easy to adjust the angle.” Also, the Crystal LED’s fine pitch allowed a camera to get close to it and film from various angles, which was also highly evaluated.
Virtual production tends to be viewed as 3DCG technology, but I believe that in essence, it is a filming technology. Constructing it from scratch through trial and error, including controlling the studio lighting by linking it to the equipment and backgrounds, has been a major challenge, and the studio’s design continues to evolve to this day.
I have been seeing the gradual adoption of virtual production methods into the filmmaking industry, but I believe there are still some challenges that must be overcome before it is fully accepted. The biggest one is the change in production style. Traditionally, filmmaking has taken place via an established process that includes budget allocation, scheduling, etc. With virtual production, pre-production is extremely important, so finding a way to convince creators to spend the necessary time and money for implementing it is vital to transforming traditional production styles.
By doing pre-production with special care, we can shorten the entire schedule and reduce associated costs, and deploy more resources fot pursuing greater creativity and higher quality of the finished product.
The Content Technology Strategy Committee will next explore the possibilities of various technical proposals received from within Sony Group as a result of KILIAN’S GAME, and I would like us to focus on simplifying the production flow for 3DCG backgrounds. Taking on this challenge will also be extremely important for helping virtual production become more popular in contents production industry.
Virtual production, a cutting-edge technology, will evolve dramatically from here on out. Sony Group will explore its opportunities in various forms.
Message: Haruyasu Yamazaki
I would like to evolve virtual production into a unique content creation solution by combining it with cutting-edge technologies such as AR (Augmented Reality) and AI while further utilizing the expertise Sony PCL has accumulated with its virtual production technologies and content creation. Sony is capable of providing special opportunities like this. I feel very rewarded to be able to bring new technologies like virtual production to the field of creative work and enrich creativity.
With this advanced filmmaking technology, I would like to transform various content creation scenes, including movies, music videos, commercials, and hybrid event
