In the short film, ballet dancers mimic the movements of swans in graceful synchronization on the shores of a serene, misty lake surrounded by deep forest. Viewers may wonder how a traditional ballet could have been performed in open nature in this way. The truth is that the entire film was shot in the Virtual Production Studio at KIYOSUMI-SHIRAKAWA BASE.

Virtual Production (VP) is a film shooting method that synthesizes the real and the virtual in real time by projecting a virtual environment on LED wall in the background while filming a live subject in front of it. Sugano, an engineer who worked on the project, explains, "Many of Sony's latest technologies are used in the pursuit of bridging the boundary between the real and the virtual to make the final product look realistic."

Applying Sony's cutting-edge technology to the classic ballet "Swan Lake"

Eisuke Ohga, who directed this production, explains that the inspiration for making a film based on "Swan Lake" came from seeing examples of cutting-edge technology being used on Broadway and in other stage productions.
"We took on this challenge as a means of showcasing the technical capabilities of KIYOSUMI-SHIRAKAWA BASE in video production, and also to see just how a traditional stage production that everyone knows could be transformed into something brand-new through the use of technology such as VP and Volumetric Capture."

The production was specifically planned in a way to incorporate imaging technology into every element of the performance. For example, in one magical shot (shown above), the camera zooms in on a dancer moving through the fog. Ohga explains, "We aimed to achieve a natural look by switching from a 3D CG dancer filmed with Volumetric Capture to a real dancer dancing in front of LED wall, at the exact moment the camera passes through the morning fog that fills the screen."

Volumetric Capture is a technology that allows high-quality reproduction of 3D CG images from a variety of angles by capturing a subject with more than 100 cameras installed on the ceiling and walls of a studio and converting the images into 3D digital data. What sets this technology apart is that it can freely visualize even viewpoints that have not been directly captured by the cameras.

Ohga explains, "In shooting this scene, I wanted to make the camera movements as grand and sweeping as possible to showcase a level of depth that cannot be achieved on stage." On the other hand, if the camera movements became too large, the LED wall required for VP would not be able to cover the entire background image. Therefore, Volumetric Capture was used to insert CG images of the dancers and achieve a realistic visual expression.

The VP studio (left) and Volumetric Capture studio (right) at KIYOSUMI-SHIRAKAWA BASE. The Volumetric Capture studio was installed next to the VP studio in 2022, and in 2023 the VP studio was enhanced with improvements such as an expanded LED wall.

"KIYOSUMI-SHIRAKAWA BASE is one of the few studios where a VP studio and a Volumetric Capture studio are located next to each other on the same floor," says Sugano. This advantage was also utilized in the filming of "Swan Lake.”

Live Keying technology development inspired by an artist’s feedback

In addition to VP and Volumetric Capture, the short film also incorporates Live Keying, a new Sony technology used in live concert staging that that isolates an object from a live image in real time and instantly displays it on an LED wall installed in the background of the studio or stage.

With conventional video processing technology, it was considered difficult to accurately extract the outline of a person in a situation where the background image changed drastically or the subject moved a lot, but Sony's Live Keying technology has made this possible through the use of Sony's special sensors.

Sugano, one of the developers of this technology, emphasizes, “Live Keying technology is particularly suitable for VP with vividly changing CG background images.”

The impetus for developing this technology came from a request Sugano received from an artist. As part of her work Sugano accompanies music artists to venues to provide technical support for concerts, and she often receives feedback and requests from artists and staff on the spot. "I heard a casual comment from an artist during rehearsal asking if such a thing [as is enabled by Live Keying technology] could be possible, and this led to me starting the Live Keying project."

Sugano explains that the decision to incorporate this technology into the video production of "Swan Lake" was backed up by her desire to "have more live music performances use Live Keying technology, and also to increase the number of cases where this technology is used together with VP."

Technology leading to new types of on-set communication

In video production, it is necessary for all staff involved in the shooting to share the visual image envisioned by the director.

Ohga says that using VP on this production also allowed him to easily share his vision of the production with the dancers in real-time on the shooting sets. "Normally, when we plan to use CG images in the background, we have to shoot with a green screen. However, with LED wall showing background images (in real-time), it is easier for the actors to visually grasp what the scene is about, and this also makes it easier for them to get into character for the role they are playing."

According to Ohga, many of the dancers commented that the use of this technology helped them to deliver a better performance.

One of the most frequently cited difficulties in filming with CG is lighting adjustment. In the video editing process, it is necessary to match the angle and brightness of the light between the realistically shot image and the CG image to create a consistent image. However, since the people in charge of filming are not necessarily familiar with CG, a lot of time is spent on adjusting the light between the real and CG images.

Ohga explains, “Even if we change the way the dancers are lit, Live Keying allows us to project the dancers' images onto the LED wall in the background in real-time, on the spot. This decreases the need for CG post-processing and greatly improves efficiency.”

Getting closer to creators and making their visions a reality at KIYOSUMI-SHIRAKAWA BASE

While these cutting-edge technologies reduced the hurdles of filming, Ohga recalls that he also felt a sense of being overwhelmed by the high degree of freedom. "I was able to discuss the various challenges of filming with the technology developers right there on location, which allowed us to use technology smartly to create the images I had envisioned." He adds, "This is a project that could only have been undertaken by Sony."

As for Sugano, she expresses her enthusiasm from an engineer's perspective, stating, "I would like to continue to work on providing technology that contributes to creation by visiting sites and directly listening to the voices of creators."

Looking toward the future, Ohga says, "I would like to use the knowledge gained from working on a regular basis with creators outside the company on video production to develop better solutions at KIYOSUMI-SHIRAKAWA BASE together with Sony's engineers."