Where the element of place
can take creativity

The streaming event was part of the Ginza Sony Park program lineup. But why would the Ginza Sony Park Project—the renovation project for the Sony Building—go to the trouble of holding events, anyway? Daisuke Nagano, the project leader, walked us through the reasoning.

NaganoLet’s go back to the start: the Sony Building, which opened in 1966. The building was kind of the brainchild of Akio Morita, one of Sony’s founders, who wanted "to create not only a showroom for Sony, in the Sony identity, but also a meaningful piece of architecture unto itself." From there, the structure took shape around the concept of an "open facility in the city." The Sony Building site had a plot of space, about 30 square meters in size, at a corner of the Sukiyabashi Crossing in Ginza. Morita called it the "Garden of Ginza," and it was a spot for things ranging from Sony products to seasonal installations, like an aquarium and a flower garden, and a host of other events that gave the area some unique color. You see private companies creating and providing public spaces in urban environments all the time now—and Sony was already doing that exact same thing more than half a century ago. Over the decades since, the space has been a vital place, an active platform for Sony’s brand-communication efforts.

And then in 2013, with Sony diversifying out of electronics—which had been at the company’s core when the Sony Building originally opened—and into music, pictures, finance, gaming, and semiconductors, the changes prompted discussions about a new approach to brand communication. That’s how the "Ginza Sony Park Project" got its start, as an initiative to renovate the Sony Building and overhaul the company’s brand communication along with it. We wanted the effort to be about more than just rebuilding Sony’s home in Ginza. We wanted to make the process of rebuilding reflect the Sony identity of being unique too.

Nagano So as all these big new buildings were going up in Tokyo with 2020 on the horizon, we decided to not build anything. We didn’t want to just bring the old building down and then get right to work on putting another one up. Holding true to the concept of an "open facility in the city," which had helped shape the Sony Building at its inception, our idea was to reinterpret the "Garden of Ginza" identity in the contemporary context as a "Park of Ginza" and turn the Sony Building reconstruction site into an "experimental Park of Ginza in the middle of the city." That set us on course for the 2018 opening of Ginza Sony Park, which, over its limited three-year run, proceeded to draw 8.54 million visitors to a wide array of events, live performances, and other programs geared toward stretching the boundaries of creativity through the element of physical place.

About the Ginza Sony Park Project 〉

Capturing the possibilities of
"virtual" places

Ginza Sony Park has always put the focus on using a real place to spark creativity. If that physicality is so central to what the facility does, why would it want to host a virtual live-music streaming event?

NaganoOne of Ginza Sony Park’s most popular programs was the "Park Live" series, which brought a total of over 200 artists to the facility to help create a place for chance musical encounters that aren’t like what you get at clubs or other live-music venues. When COVID-19 hit, though, we had to move everything online. Before long, it was 2021—the last year that Ginza Sony Park would be in existence. There just wasn’t any way to keep the Park Live series going on-site with all the construction work. But we saw so much potential in live music; we didn’t want to just give up, and there were things that we wanted to tackle on the live-streaming front too.

What kinds of things?

NaganoThere’s something so memorable about real, in-person music events—it’s probably the fact that you can feel the sound and the overall vibe of the environment. It’s stock content, an experience you take in with all your senses, which makes it easier to remember. When you stream a live-music event online, though, pretty much all you’re really using are your eyes and your ears. You don’t get much in the way of visual stimulation, either, since musicians tend to stream from studios and places that you’ve already seen before. That puts the experience into the flow-content category, less memorable on the whole. If we couldn’t do something in person and didn’t want to go the standard online route, we figured there could be a different, third option for how we could do live music.

Then came the launch of Sony PCL’s "KIYOSUMI-SHIRAKAWA BASE", a creativity hub designed to help forge new forms of expression through cutting-edge technologies. When we looked at everything the “KIYOSUMI-SHIRAKAWA BASE” had in its technological arsenal, which included virtual production, we started to see how incorporating those assets into the Park Live series could let us unlock new experience value through the interplay between the real, physical world and the virtual realm. Kato and Ohya were on the Park Live planning team, and they were both really enthusiastic about trying the idea out. That’s how the Creepy Nuts performance got its start.

Reinterpreting
the live-music experience

For the streaming event, the first orders of business were finding an artist to collaborate with, putting together a team, and laying out a basic plan for the performance.

KatoWhen we started thinking about what artist we should work with, the first name that came up was a hip-hop duo called Creepy Nuts. Both members had done Park Live performances in the past, so we already had a good rapport. Most importantly, though, they were curious—just like we were excited about all the possibilities in play, they couldn’t wait to see where the ideas could go. Since Creepy Nuts also appears on Nippon Broadcasting’s popular All Night Nippon radio show, we thought it’d be cool if we could make the event a new take on live entertainment, a combination of a live performance and a kind of talk-radio segment.

KatoTo make the event a success, we knew we’d have to create a setting where Creepy Nuts could really be themselves. Basically, we wanted them to feel at home on set, so we decided to bring some of the people who’d worked on their All Night Nippon crew to draw up an overall plan for the performance. When you’re around the people you’re familiar with, it’s a lot easier to have fun. Cultivating that kind of vibe was important to us. We put the artist first, doing whatever we could to make sure they’d be able to tap into everything that makes Creepy Nuts special.

OhyaThe biggest key to turning the live-music streaming event into stock content that people remember was giving the viewers something new to experience, an emotional impact. Centering our focus on that core goal, we built our basic plan around making the most of virtual production’s unique benefits: delivering stunningly realistic visual reproductions and leveraging the virtual dimension to realize the unreal.

From there, we developed a story with three components. One part would deliver stunningly realistic visual reproduction to make viewers think that they’re watching a standard live stream in a real, physical location. Then we’d pull back the curtain to reveal that it’s all virtual and let Creepy Nuts "play" with other background imagery in a talk-radio setting. To cap off the performance, the last part would "realize the unreal" by putting the duo in environments that’d never be possible—a live performance you’d normally only be able to imagine.

(1) Delivering stunningly realistic visual reproductions

KatoFor the first part of the live stream, we needed a place to recreate via virtual production. There were lots of ideas going around; we thought about replicating spots that were important to Creepy Nuts’ career and the members’ lives, for example. In the end, though, we decided to go with the Nippon Broadcasting parking lot. It used to be a place where people would watch performances, wait to catch glimpses of celebrities coming out of the building, and do interviews, but there hasn’t been much activity there for a while because of all the difficulties in putting together events. All Night Nippon listeners have probably seen that parking lot somehow or somewhere before, so we knew it’d tap into people’s memory bias and evoke a reaction.

If we could use virtual production to recreate that parking lot to look just like the real thing, people would squint their eyes at the screen and be, like, “Wait a second—what are they doing in the Nippon Broadcasting parking lot?” That would be the first surprise, which we’d then follow up with an even bigger one: that everything, in all that amazingly realistic detail, was completely virtual.

(2) Playing with background imagery
in a talk-radio setting

OhyaThe performance in the Nippon Broadcasting parking lot gave way to the next portion of the program, where Creepy Nuts revealed that the set was actually virtual. We modeled the second part around the All Night Nippon motif and had Creepy Nuts take their seats in a makeshift, on-stage radio booth. There, we just let the duo chat and play around with virtual production. It was all about showcasing the technology’s range in a fun, engaging way, so we put a switch on the table that the Creepy Nuts members could push to switch the background imagery from an empty grid to the surface of the moon, a savannah, a sleek room in a New York office building, and rendering of what the B2 floor at Ginza Sony Park—now under construction—used to look like.

(3) Realizing the unreal

KatoFor the last part of the program, we didn’t want to create a faithful reproduction of something. We wanted to highlight what virtual production can do—to "realize the unreal." The last song on the set list, "Nobishiro," gave us a great opportunity to do that. The lyrics kind of exaggerate a sense of space, and we thought we’d try to make it possible virtually. One is Tokyo’s Kachidoki Bridge, which obviously isn’t the kind of place you can just shut down for a live performance, so we went ahead and made that the stage. That was the first "unreal" thing. Then there’s another spot in the song where R-Shitei sings, "Hidarite ni Sukaitsurī, migite ni Tōkyō Tawā" [SKYTREE to my left, Tokyo Tower to my right]. Again, those are two Tokyo landmarks that’d never be right across from each other, flanking you on both sides—so, again, we did the “unreal” and brought them close together on the background visual in time with the lyrics. We finished by setting Kachidoki Bridge in motion through the cityscape, which topped off a fusion of the real and the virtual.

OhyaWe also wanted to capture Creepy Nuts’ identity. They’re serious about their craft, honing their rap and DJ skills to let their music do the talking, so we knew we couldn’t let the playful atmosphere get too over the top or the visuals get so decorative that they got in the way of the songs. Balance was a big focus. Since we didn’t want the "unreal" elements to be unrealistic in how they looked, we took steps to make sure that viewers would perceive the overall picture as being realistic. The floor isn’t an LED display, which means that there’s always a dividing line between the floor surface and the screen in the background. That’s why we placed real, physical objects—like roadblock signs, for example—on the ground as a way of connecting the two worlds aesthetically and making the break less conspicuous.

Using technology to
give artists
extraordinary places to play

The team also paired up with engineers to develop new technologies for the live stream.

KatoThe biggest challenge was figuring out how to expand the backgrounds past the size of the actual LED display. The large-scale, high-resolution Crystal LED display at the "KIYOSUMI-SHIRAKAWA BASE" has the capacity for background staging in a standard studio setting, but it doesn’t have the size specs for shooting live footage. Capturing the power and energy of a Creepy Nuts set would mean getting lots of dynamic shots from lots of different angles, so we needed a bigger screen to cover all the area those shots would get in frame. We sat down with some "KIYOSUMI-SHIRAKAWA BASE" engineers about what we were looking for, and we eventually got to work on developing a technology for putting CG renderings of additional background images around the periphery of the LED display in real time. That way, there’d be ample background coverage to fill whatever shot we needed for the live stream.

* The display at the “KIYOSUMI-SHIRAKAWA BASE” is from the Crystal LED B-series, a curved 8K screen measuring 15.2 meters wide by 5.4 meters tall and boasting a resolution of 9,600 x 3,456 pixels.

OhyaWe put a lot of time and effort into making all the individual backgrounds too, working with engineers to get everything looking as realistic as we could. Take the Nippon Broadcasting parking lot, for instance: we spent five hours scanning the whole space with a 3D laser scanner and taking about 2,500 pictures, after which we combined what we needed into 3DCG data. The imagery in the background for the last song in the live set, from Kachidoki Bridge to Tokyo Tower and SKYTREE, is all 3DCG data we created by adding textures and animations to existing model data.

NaganoJust rendering CG images of backgrounds in real time is hard enough, but trying to do it all within a single live stream—with no way to go back and redo anything—adds quite a bit of difficulty and risk. The team even talked about going a safer, less demanding route and just streaming a prerecorded live set against prerendered CG. But our time doing Park Live events had shown us what actually experiencing something in real time can do, so we wanted to do everything live. Recognizing what the project meant to us, the engineers responded by tweaking things until they’d fine-tuned every last detail—and that paved the way for the live stream.

Designing blank spaces,
canvases for creativity

What did the team focus on when it came to actually streaming the performance?

KatoAfter we get the place all set, we just leave everything else to the artist. I like to think of co-creation with an artist as a tree. There’s the trunk—the core focus on creating a new experience, or our main objective—and then there’s everything else that stems from the trunk, like the branches and the leaves, all the other expressive elements that give the tree its shape and color. For me, that’s an important idea to keep in mind. When we were working on the Creepy Nuts collaboration, we met with the performers and the radio crew over the course of several different meetings to explain the overall concept for the event, the unique facets of virtual production, and other elements to help give them a solid trunk that they’d be able to grow their vision for the performance from. That process of solidifying the trunk went on right up to the start of the performance as we ran through the whole production scheme one last time.

NaganoGinza Sony Park has been an amazing platform for co-creation projects with artists from all across the spectrum—and while we’ve always worked to get on a common conceptual ground, not once have we ever told a performer to "do it this way." That’s because we know that the artist deserves first billing; what we want Ginza Sony Park to be is a place conducive to creativity. Every painter needs a canvas, a blank space to make their mark on. Ginza Sony Park is a canvas for artists, a place where they can give their imaginations a freedom that allows new experiences to take shape.

Opening up
creative frontiers

The members look back on their experience in virtual live-music streaming—and forward to what the future might hold.

OhyaOne of the most rewarding parts of the event for me was during the talk-radio segment, when R-Shitei talked about how he “disappeared” in the space between the LED display and the surrounding CG visuals. He must’ve noticed in rehearsal that there was a spot where you couldn’t see the person there, and then he just went ahead and tried it out himself. For me, in that moment, I felt like the idea we’d been going for really got through to the artist. You could tell that Creepy Nuts was having fun on set, and that made me happier than anything.

KatoA few minutes into the live stream, one of the people from the All Night Nippon crew actually came over to me with a question: “At this rate, they’re probably going to go over time—how much longer do you think you can give them?” We gave them more time, and I’m sure glad we did—things were so awesome on set that they used all the extra time up. It was also really gratifying to see comments from both Creepy Nuts fans and All Night Nippon listeners pop up in the stream chat. Watching those reactions come in made me feel like those fan communities were embracing the event, which was obviously great, but it also gave me the sense that there was still plenty of untapped potential for creativity across traditional boundaries.

NaganoThe event was really a team effort, with everybody working so hard to tap the full potential of the virtual-production technology. In the end, we took live music to a place it’d never really been before: a performance that straddled the line between the real, physical world and the virtual world. After the event had wrapped up, Creepy Nuts talked about how the approach is “going to give artists and creators so much room to let their imaginations run wild” and said that they felt like they’d been part of “a completely new format for streaming and in-studio performances . . . something that could go way beyond music.” Hearing that feedback just brought it home to me that the seeds of a brand-new creative experience were starting to sprout. We’re going to keep using this experience going forward, too, when the new Ginza Sony Park opens in 2024.