5G Real-Time Transmission

Jan 20, 2021

5G has several strengths including ‘enhanced Mobile Broadband (eMBB)’, ‘Ultra Reliable Low Latency Communications (URLLC)’, and ‘massive Machine Type Communications (mMTC)’. While a number of usages are being discovered in the field of sports, one of the areas of technology that Sony is currently developing is a real-time live streaming solution using a 5G network, where encoded video can be sent with low latency to a production facilities, or streamed to cloud services. With two successful experiments at sporting events in 2019, the movement toward practical application is accelerating.

Profile

Shigeru Yoshino

Media Solution Business Div.
Consumer & Professional Business Sector
Sony Imaging Products & Solutions Inc.
Hiroki Sato

System Solutions Technology Dept.
System Software Technology Center
Sony Imaging Products & Solutions Inc.
Yojiro Shimizu

Software Div.
SW3
Sony Mobile Communications Inc.

Takashi Gondoh

Mobile Communications &
Solutions
Sony Electronics Inc.
SungHyuk Yoon

Mobile PBU
Advanced Technology
Sony Europe B.V.

The possibilities 5G brings to live sports broadcasting

──What are the benefits of using 5G in sports broadcasting?

SungHyuk Yoon：During the broadcasts of major marathon races like the one in Berlin the images shot on site are being sent to the editing site such as an OB Van through reflecting antennas attached to an aircraft circling above the venue, but by sending these images via 5G instead, this whole set up can be vastly simplified reducing costs and environmental burden.

Shigeru Yoshino：If we can send data directly to the cloud using 5G, the stadium cameras can be wireless, allowing for more flexible shooting angles and positions. Another advantage is the ability to remotely edit and broadcast high quality video on cloud services, regardless of the venue.

Success in experiments at the Berlin Marathon and American Football matches

──You have already conducted two successful experiments in Berlin and Texas. Could you please tell us about the first experiment with 5G real-time broadcasting that took place in September 2019 at the Berlin Marathon?

Yoon：For the experiment at the Berlin Marathon, we partnered with the mobile network operator Deutsche Telekom and the video production company Infront Productions.Deutsche Telekom is one of the important partners of Sony Mobile Communications and they had frequently visited Japan to hold talks with our senior management. Sony also had a strong relationship with Infront Productions, as a longtime loyal customer of Sony Imaging Products & Solutions. In March 2019, we were approached by Deutsche Telekom if we would like to participate in a 5G live broadcast PoC (Proof-of-Concept) at the Berlin Marathon with Infront Productions. The timing couldn’t have been better as Sony Imaging Products & Solutions and Sony Mobile Communications were just starting to formalize such project and looking for a right opportunity to test so we agreed and the project launched soon after.

──What exactly did the experiment consist of?

Yoshino：The first use case was broadcasting workflow. Video streams from professional camera placed around the course were sent through a transmission box using Xperia 5G devices to the Outside Broadcasting (OB) Track where the video production took place. The second use case was live streaming on social media. Live streaming video shot by three Xperia 5G devices were sent to a cloud service, Virtual Production developed by Sony Imaging Products & Solutions, for video switching before being streamed on social media.

5G real-time transmission system diagram

──How did you modify the 5G devices?

Yojiro Shimizu：5G allows for high-speed, high-capacity communication, but it also generates more heat, so from the beginning of its development, we have been working on how to improve heat dissipation. In addition, the radio circuitry and antennas had to be expanded to 5G frequencies without losing the 4G frequencies, so we leveraged Sony Mobile Communications’ expertise in creating smaller, highly efficient devices and made repeated adjustments.

──What kind of processing takes place in the transmission box?

Hiroki Sato：Transmission boxes compress the video data sent from cameras and transmit it over IP. During live broadcasts, high image quality, low latency, and stability with no signal drops are absolutely essential. Achieving all three is not easy, and in addition, during this test, it was possible that the network would change during the relay. Therefore, compression technology and IP transmission algorithms were designed to guarantee high-quality video transmission with the lowest possible delay based on the network conditions, and a system which adjusted network capacity and loss compensation was also implemented.

──What kind of technology is virtual production?

Yoshino：Virtual Production is the cloud based live production solution. In a traditional live production, video switching from multiple camera sources, is done by on-site video switcher and on-site video operator. Sony’s Virtual Production move these video switching process and audio mixing process to the cloud, and stream the video to the social media. Our customers will be able to produce live video content remotely without attending on site, which will reduce production costs.

──Were there any particular difficulties in carrying out the experiment?

Shimizu：During this experiment, we used relatively low frequency 5G bands close to 4G, known as Sub-6, but the straightness of the radio waves were unexpectedly high. So we had a hard time to find out the good shooting position while maintaining fast uplink speeds for a stable video stream.

Yoon：The challenge started when the construction of 5G base stations in Berlin did not complete until two weeks before the event, so we weren’t able to do any local pre-testing until the very last minute. On the day before the event, the team spent a lot of time testing to find the perfect positions of the cameras with stable 5G connection, but on the morning of the marathon a large balloon arch appeared at the halfway point and blocked the whole camera angle, so we were forced to juggle around last minute readjusting the camera positions. The effort all paid off the moment camera footage over 5G at the halfway point was used on the live TV broadcast. The staff room was filled with both the cheer of excitement and sighs of relief, the moment I will not forget for a very long time.

──After Berlin, the second experiment was conducted two months later at an American football game in Texas, the U.S. Please tell us about the experiment in Texas.

Takashi Gondoh：We executed a joint trial for PoC with local mobile network operator having support from local content provider and football team. It requires many discussions in the venue owner, local government, local stakeholders, legislation and so forth to be able to install and roll out the new technology such as 5G at the particular venue, therefore it was definitely a challenge to make a plan as we at Sony desired initially. Eventually we were able to proceed, by communicating with internal and external stakeholders taking into considerations of restrictions and conditions with timely manner, and adjust and manage the planning milestone. There were times when things didn't work out as planned, but we collaborated with our partners and other stakeholders to adjust milestone settings, taking into account the constraints and preconditions, and communicated with the design and development team to move forward with the plan.

──What exactly did the experiment consist of?

Sato：We did two major tests here as well, the first one being transmitting from the cameraman’s booth. We used the 5G network as an alternative to wired transmission throughout the match. The second was transmitting from the field. Taking advantage of the benefits of wireless transmission, we made a small roundabout relay with lots of media personnel in the field.

Filming with a 5G-enabled camera and transmitting the footage wirelessly

──What were some of the innovations and difficulties you encountered in addition to the experiments at Berlin?

Shimizu：5G mmWave technology, which has been commercialized in the U.S., uses a much higher frequency band than Sub-6 and is expected to provide a higher capacity and speed, allowing for high-definition video transmission. However, it also comes with a number of technical challenges, such as the waves being easily blocked by buildings or even by human body and not traveling well over large distances. Sony Mobile Communications developed smaller antenna modules and optimized its placement, so that they could pick up mmWave signal from any angle. While monitoring the received signal quality, by switching the antenna modules efficiently, contributing to reduced transmission loss and suppressed heat generation. We also changed the device materials to ensure there would be lower negative effect on the mmWave transmission properties.

Sato：On the transmitting side, we also took the time to adjust for mmWaves. Camera operators and other people moving in the vicinity can have a large effect on radio wave conditions, and there was a high risk that the transmission speed would suddenly drop, so in order to improve stability, engineers from Sony Imaging Products & Solutions and Sony Mobile Communications got together to find a better radio environment. They also improved the video compression and transmission technology and worked very hard to preserve image quality and provide a stable transmission.

──I heard there were some limitations in the pre-experiment inspections before the event.

Gondoh：Sony team needed to assess physical 5G radio node locations and those performance prior to the PoC game due to the camera location setup and some configuration. The assumption was to execute those activities at least four times at the venue but we had to make series of adjustments in order to do that. Strict restriction was set on venue entry and dry run was required to see how the PoC works without bothering the audience experience at the venue. We tried to discuss/negotiate the plans with timely manner and was able to secure 3 days prior to the game for preparation.

Camera system set up at NRG Stadium

Delivering the value of 5G to more people

──Tell us about future developments and possibilities.

Shimizu：5G brings with it a huge number of possibilities. By using our experiences and integrating 5G communication technology with Sony’s wide variety of content and services, we would like to pursue our goals of “3R Technology” - Reality, Real-time, and Remote and contribute to new kando experiences and development of new businesses for Sony.

Sato：There got support from a lot of people, and I myself learnt a lot from this experience. With no barriers to fields or usages, I will do everything I can to further improve technology to create new value through 5G.

Sony's Technology 2020