People
AR HMD—Created from the ground up
AR pioneer Hiroshi Mukawa talks about 20 years of development in optical display
Jul 25, 2024
Hiroshi Mukawa is a Distinguished Engineer at Sony Group who has worked on the development of AR HMDs (AR head-mounted displays) from the ground up. He has continued to take on this challenge in order to make people’s lives more enjoyable and convenient. We spoke with him about his career thus far and his passion for technology.
-
Hiroshi Mukawa
Distinguished Engineer
Sony Semiconductor Solutions Corporation
The path from mechatronics to optics
── What is your current role at Sony, and your area of expertise?
As a Distinguished Engineer, I give technical guidance to R&D teams working on optical technologies related to AR (augmented reality) and MR (Mixed reality) at nanophotonics development sites.
My area of expertise is optical display technologies for AR HMDs.
AR is a technology that expands the real world by overlaying digital content onto the real world in front of us. One device for realizing this is the AR HMD, which displays images overlaid on glasses or goggles worn by the user. AR HMDs have the advantage of allowing users to view stereoscopic images hands-free while in motion. At present, however, we still have challenges to address. It is difficult to simultaneously achieve a wide field of view, high brightness, and compact form factor. Because AR HMDs have an extensive range of possible applications, various companies are exploring how this technology can evolve into the next-generation computing platform.
──What technologies have you leveraged for AR HMDs?
To realize AR experiences by HMDs the vision for AR HMDs, several technologies must be integrated together. There are sensing and recognition technologies that estimate user’s head pose based on images from the camera and sensor data. We also use image CG rendering technology to display virtual objects based on user’s head pose, as well as see-through display technology, which overlays virtual objects on the real world.
I have primarily been involved in the development of the diffractive holographic waveguide-based displays that deliver light from the AR HMD’s display panels to the user’s eyes. This method has high potential as it can display images and text with a wide field of view and compact form factor on transparent glass or resin substrates with a thickness of less than 1 mm and can be manufactured at low cost. A diffractive waveguide approach requires more light than other methods, but I believe that it is the optical technology that is currently best suited for realizing a glasses-like AR HMD for consumers.
──What was your area of study prior to joining the company? After joining the company, what products were you involved with?
I majored in mechanical engineering in university and studied applied physics in graduate school. I joined Sony because I wanted to deliver products that did not yet exist in the world and because I admired Sony’s audio products. I joined the company in 1987 and, as I hoped, began working in the department in charge of audio design and was involved in product design for the first generation MiniDisc. During the first half of my career in the Audio product field, I was in charge of design as a mechanical engineer. However, because of my interest in optical pickups, I studied abroad for one year to learn optics and ended up shifting to next-generation optical pickup development in the latter half of my career in the field.
One of the theories I learned while studying abroad was hologram technology. I didn’t have any specific ideas for how to use it then, but I wanted to incorporate this interesting technology into Sony’s products. Sony’s new effort for developing leaders began at that time, so I participated in a program for thinking about Sony’s business strategy for the future. It was there that I proposed my idea for Sony’s future business using AR HMDs leveraging hologram technologies. Luckily, I was awarded for this idea, which led me consistently being involved in product development and commercialization for AR HMDs from 2004 onward.
Following this, we unveiled the prototype for the world’s first full-color AR HMD using a holographic waveguide-based display in 2008. In 2012, we commercialized captioning glasses for movie theaters, and in 2015, we commercialized SmartEyeglass for developers. In 2019, we unveiled a prototype with a spatial tracking function, and worked to develop the next-generation AR optical module from 2021.
(Left) Prototype of full-color holographic waveguide-based AR glasses (2008)
(Right) Entertainment Access Glasses (STW-C140GI), captioning glasses for movie theaters (released in 2012)
──How did not studying optics during your student days affect your development?
Although it may have been a handicap in some R&D activities, in retrospect, it was rather an advantage. If I had majored in optics only, I wouldn’t have come up with the idea of taking on the challenge of diffractive waveguide-based displays using holograms, which was an approach that no one had taken. Of course, I had to start with basic research, but I believed that I would be able to catch up even though I started later as I assumed that it would take 10 years to commercialize the technology.
Moreover, knowledge regarding mechanical engineering, such as weight balance, heat dissipation, and lens barrel precision, is extremely important for AR glasses. I am focusing on making even better products by leveraging my knowledge in both areas.
Possibilities for innovation beyond smartphones
──You recently received a SID Special Recognition Award for your research in diffractive waveguide-based augmented reality (AR) displays. This is the first time this award has been received at Sony since 2016. How do you feel about this?
Honestly, the award was a complete surprise. We began the development 20 years ago, so I was thinking, “why now?!” However, it came at a time when the AR community is more exciting than ever, so I’m thrilled that my older work was well received. This award is usually given for technologies that have impacted the industry, but this time it was probably evaluated in light of possibilities for the market before really impacting the industry.
──What is your vision for the future of AR HMDs?
When we began development, I was expecting consumer applications, such as real-time translation, navigation, games, web searches, etc., but I currently think that the major market will be in commercial fields for some time. AR HMDs are actually being used for navigation of order picking in warehouses, remote instructions to less-experienced workers, training, and other applications. Going forward, I believe a time will come when AI will be able to supplement and enhance human capabilities, especially as its recognition and prediction accuracy improves, and will also be able to automatically create content tailored to the user’s environment. I also think that lightweight, refined hardware that can be used on a daily basis may come out in the next three to four years. This evolution of content and hardware will pump the consumer market.
When smartphones were released to the world, they integrated features that were in cameras, voice recorders, PDAs (personal digital assistants), and more. Similarly, by making AR HMDs lightweight and more stylish and improving their user interfaces, I believe that they will be a new option in addition to computers, TVs, and smartphones. Many researchers and engineers involved in AR are convinced that it has the potential to trigger innovation equal to or greater than the advent of the smartphone.
Fostering the next generation of talent at Sony
──What makes Sony’s presence in the field of AR unique?
AR is extensive and is based on the integration of various technologies, from creation and distribution of content to sensing, semiconductors, display, audio, and interaction for hardware.
In this context, it is important to consider in which areas and technologies Sony excels. I believe that we should strive to maximize value through a synergistic business approach that multiplies Sony's strengths in content IP, content creation, and device technologies.
──You are involved with a diverse range of industries and professionals through academic and industry conferences. What are some of the things that you have noticed in your interactions with others in your field?
The people in the AR community are my competitors, but we are also compatriots working together to help the industry thrive. Sharing information and showing research results at academic conferences motivates me. Recently, the number of young researchers and engineers specializing in AR has been increasing, so I’m glad to see lively discussions taking place.
What I have noticed while interacting with these external parties is Sony’s attractiveness as a place to work. Recently, outstanding engineers have joined Sony, and their reason was that Sony develops a wide range of businesses, such as games and entertainment, which will potentially be applications of AR technologies. I have recently launched a group-wide working group studying and proposing strategies for AR use cases and technologies. Communication with colleagues in the group who have different expertises will be the source of new ideas. I think that diversity will create opportunities.
──What are your goals for the future?
I believe that working on AR for such a long time while simultaneously working on other technologies in parallel has led me to where I am today. I appreciate that Sony is this type of corporate culture.
Going forward, I, of course, want to continue leading AR HMD development, but I am also thinking about how I can help empower rising technical talent with the work that we’ve done so far. Within the group, there are many experts for each technology, but when bringing everything together for AR, we need people who can design the entire process by thinking about how to combine these technologies. My key mission going forward will be to support talented people who can integrate Sony’s assets within our field. Furthermore, I want to continue to contribute to the AR industry as a whole, not just within Sony, in order to make people’s lives even more convenient and enjoyable with AR technology.
